In the first part of this paper (A Sensational Crisis. Part I: The Evidence) we have seen a number of empirical studies that show three ways in which the classical perceptual framework, what I called the "dogma", loses ground. The traditional distinction between some sort of "primary" stage, that was called sensation, and a cognitive one that was called perception can no longer be maintained without problems. Neurobiological research suggests that the processes involved in perception are so intermingled that there is little value in trying to divide them up neatly into sensation and perception. First, we have seen that perception is influenced by a top‑down flow of information, which intervenes in many different functional directions; in other words, there is a strong feed back of information between final and initial stages of the perceptual process. Secondly, we have seen how crossmodal sensory integrations is not only a fact, but it may be even a requirement. Finally, there may be some cases in which higher cognitive activities, such as categorization, are performed in the very same perceptual processing.

However, despite the fact that there is a wealth of data that undermines the solidity of the "dogma", the conceptual apparatus that is used to go about perception still counts on the very same framework defined at the beginning: a unidirectional, domain-specific perception divided in different stages, sensation, perception and cognition. In my opinion, the situation is untenable: we are in what we could call a «Sensational Crisis» which calls for a new conceptual framework. Unless we begin to take into account the empirical evidence, any epistemological-driven discussion will be at a loss. Epistemology cannot rely any more on the distinction between sensation, perception and cognition.

I want to introduce now a sketch of the elements that must be taken into account in designing possible ways out for the crisis. The proposed ideas are a construal derived from certain theories of cognitive development (see especially Nelson 1985, 1996), evolution of cognition (see especially Donald 1991) and perceptual based theories of knowledge (see especially Barsalou 1999).

Getting rid of sensations

In my opinion, the first task that lays ahead is eliminating spurious distinctions in perceptual theorizing. In particular, I suggest getting rid of the distinction between perceptions and sensations. It is true that there are some versions of each notion that could be maintained. For example, there are authors (Raftopoulos 2001), for which the term "sensation" refers to all processes that lead to the first stage in the transformation of environmental energy into neural coding (e.g., the formation of the retinal image). In this stance, the set of measurements, which initially is cognitively useless, is gradually transformed along the visual pathways in increasingly structured representations that are more convenient for subsequent processing. Moreover, the processes that transform sensation to a representation that can be processed by cognition are called perception. Perception would therefore include both low-level and intermediate-level vision, and is bottom-up, in that it extracts information retrievable directly from the scene only.

However, this position is useless because, apart from being very difficult to discern what is a retinal image and what processes include, the processes that lead to energy transformation are absolutely uninteresting if we do not take into account the first transformations that take place at the level of the sense organs.

We must not forget that, in fact, the reason for using the term "sensation" has been epistemological. Philosophy has traditionally used sensations to explain the nature of sensory experience and its relation to the physical world. Sensations have been typically formulated by philosophers as the immediate objects of awareness in sensory experience or, in a variant but essentially equivalent terminology, of what is given in such experience. The standard characterization is that sensations are a kind of sense-data, that is, private, non-physical entities actually having the experienced sensory qualities, that are the immediate objects of awareness or that are given. Conversely, perceptions are of another nature, they are judgments of the "sense-data" that are obtained by a process of inference and conceptualization.

But this, as we have seen, has no place in contemporary neuroscience. There is no sense in distinguishing certain processes that take place at the very beginning of the perceptual analysis from other processes that occur later; both need each other to provide a perceptual synchronous output. The characterization of the perceptual analysis should, in my opinion, consider all the perceptual processing as a single process. This does not mean that all the different stages of the perceptual analysis are the same or equivalent; perception is a complex process, with different components that assume specialized processes. Rather, what I suggest is that perceptual analysis has to be taken as a global process that has to be understood taking into account the different components and functional specializations and the interactions between all of them.

The bottom line is that we need not confuse ourselves with an inflation of terms and notions. Perception is a complex process and it doesn’t require more complication. Let the notion of sensation rest in the history of philosophy and focus on the intrinsic richness of perceptual processing.

Multi-modality of perception

It has been common for theories of cognition to adopt representational schemes that are amodal. According to these views, the conceptual system is independent from perceptual systems and functions according to different principles. As perceptual systems process an entity in the environment, the conceptual system translates it into an amodal representation. After the visual system represents the perceived features of a bird, for example, a separate conceptual system re-represents these features amodally, using a feature list, an instantiated schema, or an activated pattern in a network, depending on the theory that describes the conceptual system. Once established, this amodal representation supports all of the higher cognitive functions. From memory to language to thought, this amodal data structure is manipulated independently of perceptual systems to represent the world, draw inferences, make decisions, and solve problems.

The alternative endorsed here stems from the tradition of perceptual-based cognitive theories (see Barsalou 1999 for a review of the literature). According to the present proposal, perceptual processes should not be encapsulated into a specific sensory modality and should be characterized taken into account the different influences or interactions coming from other modalities. Therefore, the first step is to "free" the account of perceptual processes from the requirement of being "modality"-encapsulated.

We should not stop here. There is a further step which, in my opinion, is also necessary, even if it is much more bold. Perceptual processes should be characterized, at least to some extent, not only as the processing of sensory-motor information, but also as the processing of cognitive and emotional information. Moreover, according to this view, a common system underlies perceptual, cognitive and emotional processing. In fact, multimodality means that perceptual processing integrates all modes of perceived experience, and be distributed widely throughout the modality-specific areas of the brain.

The role of memory

The role of memory must be very important for a newer view of perception. As we have seen in the research review, memory should help the organism to guide perception and action in the world. The evidence suggests that perceptual systems are diachronically, meaning in the long run, open to some rewiring of the patterns of their neural connectivity, as a result of learning. In other words, these systems are to some extent plastic.

But how memory can be integrated into perception? This is a very difficult question because we lack the necessary models and evidence. We can, nevertheless, establish some possible hints. Basically, memory must register the interconnected web of the traces that the perceptual processing -placed in a temporal context- left in the cognitive system. Accessing such registers, memory can participate in the guiding of perception and action for an ongoing episode. Specifically, the activation of past episodes act as a sort of priming. Once accessed memory acts as a "context", as a background, where the activities or perceptions are placed. This is so because it provides information about the ongoing situation and anticipating future states.

An important point to take into account is that the relevant memory traces need not be abstract representations of the objects, properties and relations that occur in a temporal context. They are the actual multimodal traces that objects, properties and relations leave, and as I said, by being multimodal, the traces are perceptual, cognitive (in its most basic sense) and emotional. Therefore, this proposal implies that every episode and every occurrence of multimodal activities are not taken as members of a type of perceptual activity, but simply as exemplars. For the moment then, every episode has to be seen as unique. This leaves unexplained how a conceptual competence can be achieved out of simple perceptual exemplars. But this is a question that cannot be addressed here.

Be it as it may, the sort of memory that might be relevant in perception should neither be confused with memories. As I have presented, multimodal activities can be described as the neural states that underlie perception. Therefore, they are not conscious experiences. An episode is a complex to be characterized within an objectivist perspective, that is, I assume that it would be possible to specify the percepts that comprise an episode, both conscious and unconscious. Memories, by contrast, are characterized as a conscious subjective properties experienced by an “I”. It is true, though, that experiencing an episode produces a conscious experience, and that activation will probably always yield a memory, but there is more to such an episode that meets the conscious eye. At most, a memory can in a certain sense be characterized as the conscious part of episodes.

Towards a characterization of percepts

A new view of perception will obviously need to establish what, if any thing, a perceptual unit is. This is the most difficult task. If perception is an active process, with continuous interactions between many levels and modalities, how can we characterize any stable state? Should they correspond to representational states? Are they some sort of mental images or any other form of conscious subjective experience? Neuroscience has no ready answer to this question. There have been many approaches, such as connectionism, interactionism, but we are far from knowing how exactly the brain codifies and registers perceptions. For one thing, it is far from clear the sort of representational content they have. Perceptual processing is dynamic and in some cases non-contentful. For example, the states that instantiate the activity of the dorsal visual pathway are a sort of mediator between an environmental disposition and an action to perform, i.e., their function is to modulate actions rather than picture realities. As such, it is difficult to imagine the "representational" content they could have. The fact is that during sensory activity systems of neurons in sensory-motor regions of the brain capture information about perceived events in the environment and in the body. After perceiving a certain object and partially storing the underlying neural states, these states are later reactivated to recreate the activity of the process.

I will now outline some properties of what a stable perceptual state, what we could call percept, should have, acknowledging nevertheless the difficulty to give a complete characterization. Firstly, a proposal coherent with contemporary studies of perception should back an objectivist approach to perception, and hence avoiding the untrustworthy route of experiential analysis. A percept should be fully characterized from a third-person stance, in the same way as emotions are already characterized objectively (Gordon 1987; DeLancey 1997; DeSousa 1987; Oatley 1992). Moreover, a percept should be a record of the neural activation that arises during perception, and would not simply correspond to the conscious part of such a perceptual processing. The conscious aspect of a percept would only be a subset of the entire brain state that underlies a perception; it would actually be a very small subset that represents a coherent aspect of the activity (see Barsalou 1999 for a related notion).

Secondly, a multimodal view of perception should posit perceptual entities, percepts, as brain activities that correspond to a stable and complex processing of perceptual information. In fact, rather than an amodal redescription of the perceptual event representing the object, a percept should be viewed as an activity of perceptual systems. This activity should embrace other properties that are not considered specifically perceptual, including certain basic cognitive and emotional features. Indeed, the important fact is that "perceptual" will have to go beyond classical sensory modalities, as it usually does, and refer to a much more widely to any aspect of perceived experience, including propioception, introspection, as well as certain properties that are normally considered to be cognitive.

Additionally, percepts would have a further characteristic that is crucial: Percepts should not be independent from the other brain activities situation in which they are active, and are recorded as part of this context. Accordingly, the second axis on which the new perceptual stance should be characterized is its temporal dimension, which I call "episode". In this sense, the cognitive system would parse its perceptual analysis in units in which percepts occur. Episodes would be temporal units with boundaries, with beginnings and endings, even though what constitutes a boundary is not yet well-established (at this point it seems that, normally, boundaries are defined by coherent sequences of events that are spatially, causally or thematically related, such as episodes related to activities of eating, playing, going to bed, going in a car, etc.). At this moment, there is considerable experimental evidence that the episode is the atomic context of the experience of the first stages of cognitive development in children, as well as the cognition of apes (see Nelson 1996 for a review of event representation in cognitive development and Donald 1991 for a review on ape cognition). There are nevertheless differences between the current use of event representations (i.e., by Nelson 1996) or scripts/schemas (see Arbib, Erdi and Szentágothai 1997) and the proposal sketched here. I count as an episode every temporal unit which the cognitive system parses, and therefore I do not establish any pre-conceptual difference between types of episodes. In this sense, the cognitive system parses all of its experiences, and it does so to any sort situation in which the system is active, including all sorts of perceptual, emotional and propioceptive experiences. Therefore, an episode should include the set of what I call multimodal percepts that occur in a given temporal context, and it could be seen as an interconnected web of the traces that objects, properties, and relations -placed in a temporal context- leave in the cognitive system. The episode functions as the context in which percepts, and the relations between them, are cast. It is in such a context that the child will develop adultlike conceptual capacities.

Episodes are the atomic units of the architecture proposed here, but they are complex units, provided that they are made up of percepts. The claim, nevertheless, is not that episodes are a string of discrete percepts. Rather, the claim is that episodes are holistic elements. The child recognizes a spoon as a part of the whole eating episode. Accordingly, percepts do not exist independently of the episodes in which appear, nor are episodes decomposable into the percepts that take part in a temporal context. In other words, the elements that take part in percepts have to be characterized in relation with the episode in which they take part, and the episodes are to be characterized by the percepts that take part in them. This would seem prima facie to be somehow circular. The problem of intercharacterizing episodes and percepts is only a theoretical illusion, as right triangles and hypotenuses are intercharacterized. Episodes are global units that can be decomposed in percepts by the theorist; likewise, percepts can only be extracted from episodes by the theorist. The whole complex of multi-modal percepts is "supported" joined integrated by the temporal dimension, and it will be the time what makes all the structures maintained together.

Conclusion

In this paper I have reviewed recent neuroscientific evidence that compels theorists to review the conceptual framework that has been used to characterize perception. Additionally, I have proposed a sketch of the way in which such a revision could be accomplished. The task ahead is, nevertheless, huge, since the conceptual apparatus necessary to face the challenges of contemporary neuroscience is far behind the level it should have by now. Be it as it may, the task must be accomplished because any future epistemological-derived argumentation should count on a new perspective about perception.

References

Alston, W. P. (1993). The Reliability of Sense Perception. Ithaca, NY: Cornell University Press.

Anderson A.K., Phelps E.A. (2001). Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature 411: 305 - 309.

Antonini, A., Strycker, M. P., & Chapman, B. (1995). Development and plasticity of cortical columns and their thalamic inputs. In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical plasticity. Reading, MA: Addison-Wesley.

Arbib, M.A., Erdi, P. and Szentágothai, J. 1997, Neural Organization: Structure, Function, and Dynamics, Cambridge, MA: The MIT Press

Armstrong, D. M. (1961). Perception and the Physical World. London: Routledge.

Austin, J. L., (1962). Sense and Sensibilia. London: Oxford University Press.

Ayer, A. J., (1946-7). Phenomenalism, Proceedings of the Aristotelian Society, 47: 163-96.

Barsalou, L.W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22: 577-609.

Blakemore, C. & Cooper, G. F. (1970) Nature 228: 477–478

Calvert, G.A., Bullmore, E.T., Brammer, M.J., Campbell, R., Williams, S.C.R., McGuire, P.K, Woodruff, P.W.R., Iversen, S.D.S, David, A.S. (1997) Activation of Auditory Cortex During Silent Lipreading. Science 276: 593-596.

Crammond, D.J. (1997). Motor imagery: Never in your wildest dreams. Trends in Neuroscience, 20, 54-57.

Dalton, P., Doolittle, N., Nagata, H. and Breslin, P.A.S. (2000). The merging of the senses: integration of subthreshold taste and smell. Nature neuroscience 3: 431-432.

Darian-Smith, C. & Gilbert, C. D (1994) Nature 368: 737–740.

DeLancey, C. (1997). Emotion and the computational theory of mind. In (S. O'Nuillain, P. McKevitt, & E. MacAogain, eds) Two Sciences of Mind. John Benjamins.

Deschaumes-Molinaro, C., Dittmar, A., & Vernet-Maury, E. (1992). Autonomic nervous system response patterns correlate with mental imagery. Physiology & Behavior, 51, 1021-1027.

De Sousa, R. (1987). The rationality of emotion. Cambridge, MA: MIT Press.

Donald, M. (1991). Origins of the Modern Mind: Three stages in the evolution of culture and cognition. Cambridge: Harvard University Press.

Douglas RJ, Martin KAC (1998). Neocortex. In: The synaptic organisation of the brain (Shepherd GM ed), pp 389-438. New York: Oxford University Press.

Driver J (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature 381:66-68.

DuBose, C.N., Cardello A.V., Maller, O. (1980). Effects of colorants and flavorants on identification, perceived flavor intensity, and hedonic quality of fruit-flavored beverages and cake. J Food Sci 45:1393-1399.

Felleman D.J., Van Essen D.C. (1991) Distributed hierarchical processing in the primate cerebral cortex. Cereb.Cortex. 1:1-47.

Freedman, D.J, Riesenhuber, M., Poggio, T. and Miller, E.K (2001). Categorical Representation of Visual Stimuli in the Primate Prefrontal Cortex. Science 291: 312-316.

Frith, C., & Dolan, R.J. (1997). Brain mechanisms associated with top-down processes in perception. Philosophical Transactions of the Royal Society of London, Series B: Biological sciences, 352, 1221-1230.

Gao, J.H., Parsons, L.M, Bower, J.M, Xiong, J., Li, J., and Fox, P.T. (1996). Science 272, 545-547.

Gielen, S. C., Schmidt, R. A. & Van Der Heuvel, P. J. (1983). Neural and behavioral response enhancements to combinations of sensory stimuli are found within and across many sensory, Percept. Psychophys. 34:161–168

Gilbert C.D. (1998) Adult cortical dynamics. Physiol.Rev. 78:467-485.

Gilbert, C. D. & Wiesel, T. N. (1992). Nature 356: 150–152.

Gilbert, C. D., Das, A., Ito, M., Kapadia, M. & Westheimer, G. (1996). Proc Natl Acad Sci USA 93: 615–622.

Gordon, R. M. (1987). The Structure of Emotions: Investigations in Cognitive Philosophy. Cambridge University Press.

Guadagni, D. G., Buttery, R. G., Okano, S. & Burr, H. K. (1963) Nature 200: 1288–1289.

Hespos, S.J., & Rochat, P. (1997). Dynamic mental representation in infancy. Cognition, 64, 153-188.

Jeannerod, M. (1995). Mental imagery in the motor context. Neuropsychologia, 33, 1419-1432.

Jeannerod, M. (1994). The representing brain: Neural correlates of motor intention and imagery. Behavioral and Brain Sciences, 17, 187-245.

Johnson, M. (1987). The Body in the Mind. Chicago: University of Chicago Press.

Jousmäki V, Hari R (1998). Parchment-skin illusion: sound-biased touch. Curr Biol 8:R190.

Karni, A., & Sagi, D. (1995). A memory system in the adult visual cortex. In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical plasticity. Reading, MA: Addison-Wesley.

König P, Luksch H (1998) Active sensing - closing multiple loops. Zeitschrift für Naturforschung in press

Kosslyn, S.M., Thompson, W.L., Kim, I.J., & Alpert, N.M. (1995). Topographical representations of mental images in primary visual cortex. Nature, 378, 496-498.

Macaluso E, Frith C, Driver J (2000). Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206-1208.

Magnuson, J.S., & Nusbaum, H.C. (1993). Talker differences and perceptual normalization. Journal of theAcoustical Society of America, 93, 2371.

Merzenich, M. M. et al. (1984). J. Comp. Neurol. 224: 591–605.

Merzenich, M. M., & Jenkins, W. M. (1995). Cortical plasticity, learning, and learning dysfunction. In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical plasticity. Reading, MA: Addison-Wesley.

Murphy, C., Cain, W. S. & Bartoshuk, L. M. (1977). Sens Process 1, 204–211.

Nelson, K. (1985). Making sense. The Acquisition of Shared Meaning. New York: Academic Press.

Nelson, K. (1996). Language in Cognitive Development. Emergence of the Mediated Mind. Cambridge: Cambridge University Press.

Oatley, K. (1992). Best laid schemes: The psychology of emotions. Cambridge: Cambridge University Press.

Pascual-Leone, A. and Walsh, V. (2001) Fast Backprojections from the Motion to the Primary Visual Area Necessary for Visual Awareness. Science 292: 510-515.

Pavani F, Spence C, Driver J (2000). Visual capture of touch; out-of-the-body experiences with rubber gloves. Psychol Sci in press.

Peterson, M.A., & Gibson, B.S. (1993). Shape recognition inputs to figure-ground organization in three-dimensional displays. Cognitive Psychology, 3, 383-429.

Radeau M (1994). Auditory-visual spatial interaction and modularity. Curr Psychol Cogn 13:3-51.

Raftopoulos, A. (2001). Is perception informationally encapsulated? The issue of the theory-ladenness of perception. Cognitive Science 25: 423–451.

Reed, C.L., & Vinson, N.G. (1996). Conceptual effects on representational momentum. Journal of Experimental Psychology: Human Perception and Performance, 22, 839-850.

Saldaña HM, Rosenblum LD (1993) Visual influences on auditory pluck and bow judgments. Percept Psychophys 54:406-416.

Sengpiel, F., Stawinski, P. & Bonhoeffer, T. (1999) Nat Neurosci 2: 727–732.

Shams, L., Kamitani, Y. and Shimojo, S. (2000). What you see is what you hear. Nature 408: 788.

Sparks, D. L. & Groh, J. M. (1995) in M. Gazzaniga (ed.) The Cognitive Neurosciences MIT Press, Cambridge, Massachusetts.

Spence C, Driver J (2000). Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism. NeuroReport 11: 2057-2061.

Stein A. von, Chiang C, König P (1997) The effect of experience and behavior on interareal interactions. Soc Neurosci Abstr 23: 405.4 (Abstract)

Stevens, J.C. (1997). Physiol Behav 62: 1137–1143.

Stiles, J. (1995). Plasticity and development. Evidence from children with early occurring focal brain injury. In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical plasticity. Reading, MA: Addison-Wesley.

Vroomen J, de Gelder B (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on visual perception. J Exp Psychol in press.

Wallace, M. T., Wilkinson, L. K. & Stein, B. E (1996). J. Neurophysiol. 76: 1246–1266

Weisstein, N., & Harris, C.S. (1980). Masking and unmasking of distributed representations in the visual system. In C.S. Harris (Ed.), Visual coding and adaptability (pp. 317-364). Hillsdale, NJ: Lawrence Erlbaum Associates.

Whitaker, D and McGraw, P.V. Long-term visual experience recalibrates human orientation perception. Nature Neuroscience 3:13.

Yeo. R. M., Yonebayashi, Y., & Allman, J. M. (1995). Perceptual memory of cognitively defined contours: A rapid, robust and long-lasting form of memory. In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical pasticity. Reading, MA: Addison-Wesley.

Zacks, J. et al (2001). How does the brain parse activities in the world into discrete perceptual events? Nature Neurosci 4, 651–655

Zatorre, R.J., Halpern, A.R., Perry, D.W., Meyer, E., & Evans, A.C. (1996). Hearing in the mind's ear: A PET investigation of musical imagery and perception. Journal of Cognitive Neuroscience 8, 29-46.