In the first part
of this paper (A Sensational Crisis. Part I: The Evidence) we have seen a
number of empirical studies that show three ways in which the classical
perceptual framework, what I called the "dogma", loses ground. The
traditional distinction between some sort of "primary" stage, that
was called sensation, and a cognitive one that was called perception can no
longer be maintained without problems. Neurobiological research suggests that
the processes involved in perception are so intermingled that there is little
value in trying to divide them up neatly into sensation and perception. First, we have seen that perception is
influenced by a top‑down flow of information, which intervenes in many
different functional directions; in other words, there is a strong feed back of
information between final and initial stages of the perceptual process.
Secondly, we have seen how crossmodal sensory integrations is not only a fact,
but it may be even a requirement. Finally, there may be some cases in which higher cognitive activities,
such as categorization, are performed in the very same perceptual processing.
the fact that there is a wealth of data that undermines the solidity of the
"dogma", the conceptual apparatus that is used to go about perception
still counts on the very same framework defined at the beginning: a
unidirectional, domain-specific perception divided in different stages,
sensation, perception and cognition. In
my opinion, the situation is untenable: we are in what we could call a
«Sensational Crisis» which calls for a new conceptual framework. Unless we
begin to take into account the empirical evidence, any epistemological-driven
discussion will be at a loss. Epistemology cannot rely any more on the distinction between sensation,
perception and cognition.
I want to
introduce now a sketch of the elements that must be taken into account in
designing possible ways out for the crisis. The proposed ideas are a construal
derived from certain theories of cognitive development (see especially Nelson
1985, 1996), evolution of cognition (see especially Donald 1991) and perceptual
based theories of knowledge (see especially Barsalou 1999).
Getting rid of
In my opinion, the
first task that lays ahead is eliminating spurious distinctions in perceptual
theorizing. In particular, I suggest getting rid of the distinction between
perceptions and sensations. It is true that there are some versions of each
notion that could be maintained. For example, there are authors (Raftopoulos
2001), for which the term "sensation" refers to all processes that
lead to the first stage in the transformation of environmental energy into
neural coding (e.g., the formation of the retinal image). In this stance, the
set of measurements, which initially is cognitively useless, is gradually
transformed along the visual pathways in increasingly structured
representations that are more convenient for subsequent processing. Moreover,
the processes that transform sensation to a representation that can be
processed by cognition are called perception. Perception would therefore
include both low-level and intermediate-level vision, and is bottom-up,
in that it extracts information retrievable directly from the scene only.
position is useless because, apart from being very difficult to discern what
is a retinal image and what processes include, the processes that lead
to energy transformation are absolutely uninteresting if we do not take into
account the first transformations that take place at the level of the sense
We must not forget
that, in fact, the reason for using the term "sensation" has been
epistemological. Philosophy has traditionally used sensations to explain the
nature of sensory experience and its relation to the physical world. Sensations
have been typically formulated by philosophers as the immediate objects of
awareness in sensory experience or, in a variant but essentially equivalent
terminology, of what is given in such experience. The standard characterization
is that sensations are a kind of sense-data, that is, private, non-physical
entities actually having the experienced sensory qualities, that are the
immediate objects of awareness or that are given. Conversely, perceptions are
of another nature, they are judgments of the "sense-data" that
are obtained by a process of inference and conceptualization.
But this, as we
have seen, has no place in contemporary neuroscience. There is no sense in
distinguishing certain processes that take place at the very beginning of the
perceptual analysis from other processes that occur later; both need each other
to provide a perceptual synchronous output. The characterization of the
perceptual analysis should, in my opinion, consider all the perceptual
processing as a single process. This does not mean that all the different
stages of the perceptual analysis are the same or equivalent; perception is a
complex process, with different components that assume specialized processes.
Rather, what I suggest is that perceptual analysis has to be taken as a global
process that has to be understood taking into account the different components
and functional specializations and the interactions between all of them.
The bottom line is
that we need not confuse ourselves with an inflation of terms and notions.
Perception is a complex process and it doesn’t require more complication. Let
the notion of sensation rest in the history of philosophy and focus on the
intrinsic richness of perceptual processing.
It has been common
for theories of cognition to adopt representational schemes that are amodal.
According to these views, the conceptual system is independent from perceptual
systems and functions according to different principles. As perceptual systems
process an entity in the environment, the conceptual system translates it into
an amodal representation. After the visual system represents the perceived
features of a bird, for example, a separate conceptual system re-represents
these features amodally, using a feature list, an instantiated schema, or an
activated pattern in a network, depending on the theory that describes the
conceptual system. Once established, this amodal representation supports all of
the higher cognitive functions. From memory to language to thought, this amodal
data structure is manipulated independently of perceptual systems to represent
the world, draw inferences, make decisions, and solve problems.
endorsed here stems from the tradition of perceptual-based cognitive
theories (see Barsalou 1999 for a review of the literature). According to the present proposal,
perceptual processes should not be encapsulated into a specific sensory
modality and should be characterized taken into account the different
influences or interactions coming from other modalities. Therefore, the first
step is to "free" the account of perceptual processes from the
requirement of being "modality"-encapsulated.
We should not stop
here. There is a further step which, in my opinion, is also necessary, even if it is much more bold.
Perceptual processes should be
characterized, at least to some extent, not only as the processing of sensory-motor
information, but also as the processing of cognitive and emotional information.
Moreover, according to this view, a common system underlies perceptual,
cognitive and emotional processing. In fact, multimodality means that
perceptual processing integrates all modes of perceived experience, and be
distributed widely throughout the modality-specific areas of the brain.
The role of
The role of memory
must be very important for a newer view of perception. As we have seen in the
research review, memory should help the organism to guide perception and action
in the world. The evidence suggests
that perceptual systems are diachronically, meaning in the long run, open to
some rewiring of the patterns of their neural connectivity, as a result of
learning. In other words, these systems are to some extent plastic.
But how memory can
be integrated into perception? This is a very difficult question because we
lack the necessary models and evidence. We can, nevertheless, establish some
possible hints. Basically, memory must register the interconnected web of the traces that the perceptual
processing -placed in a temporal context- left in the cognitive
system. Accessing such registers,
memory can participate in the guiding of perception and action for an ongoing
episode. Specifically, the activation of past episodes act as a sort of
priming. Once accessed memory acts as a "context", as a background,
where the activities or perceptions are placed. This is so because it provides
information about the ongoing situation and anticipating future states.
An important point
to take into account is that the relevant memory traces need not be abstract
representations of the objects, properties and relations that occur in a
temporal context. They are the actual multimodal traces that objects, properties
and relations leave, and as I said, by being multimodal, the traces are
perceptual, cognitive (in its most basic sense) and emotional. Therefore, this
proposal implies that every episode and every occurrence of multimodal
activities are not taken as members of a type of perceptual activity, but
simply as exemplars. For the moment then, every episode has to be seen as
unique. This leaves unexplained how a conceptual competence can be achieved out
of simple perceptual exemplars. But this is a question that cannot be addressed
Be it as it may,
the sort of memory that might be relevant in perception should neither be
confused with memories. As I have presented, multimodal activities can be
described as the neural states that underlie perception. Therefore, they are
not conscious experiences. An episode is a complex to be characterized within
an objectivist perspective, that is, I assume that it would be possible to
specify the percepts that comprise an episode, both conscious and unconscious.
Memories, by contrast, are characterized as a conscious subjective properties
experienced by an “I”. It is true, though, that experiencing an episode
produces a conscious experience, and that activation will probably always yield
a memory, but there is more to such an episode that meets the conscious eye. At
most, a memory can in a certain sense be characterized as the conscious part of
characterization of percepts
A new view of
perception will obviously need to establish what, if any thing, a perceptual
unit is. This is the most difficult task. If perception is an active process,
with continuous interactions between many levels and modalities, how can we
characterize any stable state? Should they correspond to representational
states? Are they some sort of mental
images or any other form of conscious subjective experience? Neuroscience has no ready answer to this
question. There have been many approaches, such as connectionism,
interactionism, but we are far from knowing how exactly the brain codifies and
registers perceptions. For one thing, it is far from clear the sort of
representational content they have. Perceptual processing is dynamic and in
some cases non-contentful. For example,
the states that instantiate the activity of the dorsal visual pathway are a
sort of mediator between an environmental disposition and an action to perform,
i.e., their function is to modulate actions rather than picture realities. As
such, it is difficult to imagine the "representational" content they
could have. The fact is that during sensory activity systems of neurons in
sensory-motor regions of the brain capture information about perceived
events in the environment and in the body. After perceiving a certain object
and partially storing the underlying neural states, these states are later
reactivated to recreate the activity of the process.
I will now outline
some properties of what a stable perceptual state, what we could call percept,
should have, acknowledging nevertheless the difficulty to give a complete
characterization. Firstly, a proposal coherent with contemporary studies of
perception should back an objectivist approach to perception, and hence
avoiding the untrustworthy route of experiential analysis. A percept should
be fully characterized from a third-person stance, in the same way as
emotions are already characterized objectively (Gordon 1987; DeLancey 1997; DeSousa 1987; Oatley 1992).
Moreover, a percept should be a record of the neural activation that arises
during perception, and would not simply correspond to the conscious part of
such a perceptual processing. The
conscious aspect of a percept would only be a subset of the entire brain state
that underlies a perception; it would actually be a very small subset that
represents a coherent aspect of the activity (see Barsalou 1999 for a related
multimodal view of perception should posit perceptual entities, percepts, as
brain activities that correspond to a stable and complex processing of
perceptual information. In fact, rather than an amodal redescription of the
perceptual event representing the object, a percept should be viewed as an
activity of perceptual systems. This activity should embrace other properties
that are not considered specifically perceptual, including certain basic
cognitive and emotional features. Indeed, the important fact is that
"perceptual" will have to go beyond classical sensory modalities, as
it usually does, and refer to a much more widely to any aspect of perceived
experience, including propioception, introspection, as well as certain
properties that are normally considered to be cognitive.
percepts would have a further characteristic that is crucial: Percepts should
not be independent from the other brain activities situation in which they are
active, and are recorded as part of this context. Accordingly, the second axis
on which the new perceptual stance should be characterized is its temporal
dimension, which I call "episode". In this sense, the cognitive system would parse its perceptual analysis
in units in which percepts occur. Episodes would be temporal units with
boundaries, with beginnings and endings, even though what constitutes a
boundary is not yet well-established (at this point it seems that,
normally, boundaries are defined by coherent sequences of events that are
spatially, causally or thematically related, such as episodes related to
activities of eating, playing, going to bed, going in a car, etc.). At this
moment, there is considerable experimental evidence that the episode is the
atomic context of the experience of the first stages of cognitive development
in children, as well as the cognition of apes (see Nelson 1996 for a review of
event representation in cognitive development and Donald 1991 for a review on ape
cognition). There are nevertheless differences between the current use of event
representations (i.e., by Nelson 1996) or scripts/schemas (see Arbib, Erdi and
Szentágothai 1997) and the proposal sketched here. I count as an episode every
temporal unit which the cognitive system parses, and therefore I do not
establish any pre-conceptual difference between types of episodes. In
this sense, the cognitive system parses all of its experiences, and it does so
to any sort situation in which the system is active, including all sorts of
perceptual, emotional and propioceptive experiences. Therefore, an episode
should include the set of what I call multimodal percepts that
occur in a given temporal context, and it could be seen as an interconnected
web of the traces that objects, properties, and relations -placed in a temporal
context- leave in the cognitive system. The episode functions as the context in
which percepts, and the relations between them, are cast. It is in such a
context that the child will develop adultlike conceptual capacities.
Episodes are the
atomic units of the architecture proposed here, but they are complex units,
provided that they are made up of percepts. The claim, nevertheless, is not
that episodes are a string of discrete percepts. Rather, the claim is that
episodes are holistic elements. The child recognizes a spoon as a part of the
whole eating episode. Accordingly, percepts do not exist independently of the
episodes in which appear, nor are episodes decomposable into the percepts that take
part in a temporal context. In other words, the elements that take part in
percepts have to be characterized in relation with the episode in which they
take part, and the episodes are to be characterized by the percepts that take
part in them. This would seem prima facie to be somehow circular. The problem
of intercharacterizing episodes and percepts is only a theoretical illusion, as
right triangles and hypotenuses are intercharacterized. Episodes are global
units that can be decomposed in percepts by the theorist; likewise, percepts
can only be extracted from episodes by the theorist. The whole complex of
multi-modal percepts is "supported" joined integrated by the temporal
dimension, and it will be the time what makes all the structures maintained together.
In this paper I
have reviewed recent neuroscientific evidence that compels theorists to review
the conceptual framework that has been used to characterize perception.
Additionally, I have proposed a sketch of the way in which such a revision
could be accomplished. The task ahead is, nevertheless, huge, since the
conceptual apparatus necessary to face the challenges of contemporary
neuroscience is far behind the level it should have by now. Be it as it may, the task must be
accomplished because any future epistemological-derived argumentation should
count on a new perspective about perception.
Alston, W. P. (1993). The Reliability of Sense Perception. Ithaca, NY:
Cornell University Press.
Phelps E.A. (2001). Lesions of the human amygdala impair enhanced perception of
emotionally salient events. Nature 411: 305 - 309.
Strycker, M. P., & Chapman, B. (1995). Development and plasticity of
cortical columns and their thalamic inputs. In B. Julesz and I. Kovacs (Eds.), Maturational
windows and adult cortical plasticity. Reading, MA: Addison-Wesley.
Arbib, M.A., Erdi,
P. and Szentágothai, J. 1997, Neural Organization: Structure, Function, and
Dynamics, Cambridge, MA: The MIT Press
Armstrong, D. M.
(1961). Perception and the Physical World. London: Routledge.
Austin, J. L.,
(1962). Sense and Sensibilia. London: Oxford University Press.
J., (1946-7). Phenomenalism,
Proceedings of the Aristotelian Society, 47: 163-96.
(1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22:
& Cooper, G. F. (1970) Nature 228: 477–478
G.A., Bullmore, E.T., Brammer, M.J.,
Campbell, R., Williams, S.C.R., McGuire, P.K, Woodruff, P.W.R., Iversen, S.D.S,
David, A.S. (1997) Activation of Auditory Cortex During Silent Lipreading. Science
(1997). Motor imagery: Never in your wildest dreams. Trends in Neuroscience,
Doolittle, N., Nagata, H. and Breslin, P.A.S. (2000). The merging of the
senses: integration of subthreshold taste and smell. Nature neuroscience
& Gilbert, C. D (1994) Nature 368: 737–740.
(1997). Emotion and the computational theory of mind. In (S. O'Nuillain, P.
McKevitt, & E. MacAogain, eds) Two Sciences of Mind. John Benjamins.
C., Dittmar, A., & Vernet-Maury, E. (1992). Autonomic nervous system response patterns correlate
with mental imagery. Physiology & Behavior, 51, 1021-1027.
Sousa, R. (1987). The
rationality of emotion. Cambridge, MA: MIT Press.
(1991). Origins of the Modern Mind:
Three stages in the evolution of culture and cognition. Cambridge: Harvard
Douglas RJ, Martin
KAC (1998). Neocortex. In: The synaptic organisation of the brain (Shepherd GM
ed), pp 389-438. New York: Oxford University Press.
Driver J (1996).
Enhancement of selective listening by illusory mislocation of speech sounds due
to lip-reading. Nature 381:66-68.
Cardello A.V., Maller, O. (1980). Effects of colorants and flavorants on
identification, perceived flavor intensity, and hedonic quality of
fruit-flavored beverages and cake. J Food Sci 45:1393-1399.
Felleman D.J., Van
Essen D.C. (1991) Distributed hierarchical processing in the primate cerebral
cortex. Cereb.Cortex. 1:1-47.
Riesenhuber, M., Poggio, T. and Miller, E.K (2001). Categorical Representation
of Visual Stimuli in the Primate Prefrontal Cortex. Science 291: 312-316.
Frith, C., &
Dolan, R.J. (1997). Brain mechanisms associated with top-down processes
in perception. Philosophical Transactions of the Royal Society of London,
Series B: Biological sciences, 352, 1221-1230.
Parsons, L.M, Bower, J.M, Xiong, J., Li, J., and Fox, P.T. (1996). Science
S. C., Schmidt, R. A. & Van Der Heuvel, P. J. (1983). Neural and behavioral response enhancements
to combinations of sensory stimuli are found within and across many sensory, Percept.
Gilbert C.D. (1998)
Adult cortical dynamics. Physiol.Rev. 78:467-485.
Gilbert, C. D.
& Wiesel, T. N. (1992). Nature 356: 150–152.
Gilbert, C. D.,
Das, A., Ito, M., Kapadia, M. & Westheimer, G. (1996). Proc Natl Acad
Sci USA 93: 615–622.
Gordon, R. M.
(1987). The Structure of Emotions: Investigations in Cognitive Philosophy.
Cambridge University Press.
Guadagni, D. G.,
Buttery, R. G., Okano, S. & Burr, H. K. (1963) Nature 200:
& Rochat, P. (1997). Dynamic mental representation in infancy. Cognition,
(1995). Mental imagery in the motor context. Neuropsychologia, 33, 1419-1432.
(1994). The representing brain: Neural correlates of motor intention and
imagery. Behavioral and Brain Sciences, 17, 187-245.
(1987). The Body in the Mind. Chicago: University of Chicago Press.
Jousmäki V, Hari R
(1998). Parchment-skin illusion: sound-biased touch. Curr Biol 8:R190.
Karni, A., &
Sagi, D. (1995). A memory system in the adult visual cortex. In B. Julesz and
I. Kovacs (Eds.), Maturational windows and adult cortical plasticity. Reading,
König P, Luksch H
(1998) Active sensing - closing multiple loops. Zeitschrift für
Naturforschung in press
Thompson, W.L., Kim, I.J., & Alpert, N.M. (1995). Topographical
representations of mental images in primary visual cortex. Nature, 378, 496-498.
Macaluso E, Frith
C, Driver J (2000). Modulation of human visual cortex by crossmodal spatial
attention. Science 289:1206-1208.
& Nusbaum, H.C. (1993). Talker differences and perceptual normalization.
Journal of theAcoustical Society of America, 93, 2371.
Merzenich, M. M. et
al. (1984). J. Comp. Neurol. 224: 591–605.
Merzenich, M. M.,
& Jenkins, W. M. (1995). Cortical plasticity, learning, and learning
dysfunction. In B. Julesz and I. Kovacs (Eds.), Maturational windows and
adult cortical plasticity. Reading, MA: Addison-Wesley.
Murphy, C., Cain,
W. S. & Bartoshuk, L. M. (1977). Sens Process 1, 204–211.
Nelson, K. (1985).
Making sense. The Acquisition of Shared Meaning. New York:
Nelson, K. (1996).
Language in Cognitive Development. Emergence of the Mediated Mind.
Cambridge: Cambridge University Press.
Oatley, K. (1992).
Best laid schemes: The psychology of emotions. Cambridge: Cambridge University
and Walsh, V. (2001) Fast Backprojections from the Motion to the Primary Visual
Area Necessary for Visual Awareness. Science 292: 510-515.
Pavani F, Spence
C, Driver J (2000). Visual capture of touch; out-of-the-body experiences with
rubber gloves. Psychol Sci in press.
& Gibson, B.S. (1993). Shape recognition inputs to figure-ground
organization in three-dimensional displays. Cognitive Psychology, 3, 383-429.
Radeau M (1994).
Auditory-visual spatial interaction and modularity. Curr Psychol Cogn 13:3-51.
(2001). Is perception informationally encapsulated? The issue of the
theory-ladenness of perception. Cognitive Science 25: 423–451.
Reed, C.L., &
Vinson, N.G. (1996). Conceptual effects on representational momentum. Journal
of Experimental Psychology: Human Perception and Performance, 22, 839-850.
Rosenblum LD (1993) Visual influences on auditory pluck and bow judgments. Percept
Stawinski, P. & Bonhoeffer, T. (1999) Nat Neurosci 2: 727–732.
Kamitani, Y. and Shimojo, S. (2000). What you see is what you hear. Nature
Sparks, D. L.
& Groh, J. M. (1995) in M. Gazzaniga (ed.) The Cognitive Neurosciences
MIT Press, Cambridge, Massachusetts.
Spence C, Driver J
(2000). Attracting attention to the illusory location of a sound: reflexive
crossmodal orienting and ventriloquism. NeuroReport 11: 2057-2061.
Stein A. von,
Chiang C, König P (1997) The effect of experience and behavior on interareal
interactions. Soc Neurosci Abstr 23: 405.4 (Abstract)
Stevens, J.C. (1997). Physiol Behav 62: 1137–1143.
Stiles, J. (1995).
Plasticity and development. Evidence from children with early occurring focal
brain injury. In B. Julesz and I. Kovacs (Eds.), Maturational windows and
adult cortical plasticity. Reading, MA: Addison-Wesley.
Vroomen J, de Gelder B (2000). Sound
enhances visual perception: Cross-modal effects of auditory organization on
visual perception. J Exp Psychol in press.
Wallace, M. T.,
Wilkinson, L. K. & Stein, B. E (1996). J. Neurophysiol. 76:
& Harris, C.S. (1980). Masking and unmasking of distributed representations
in the visual system. In C.S. Harris (Ed.), Visual coding and adaptability (pp.
317-364). Hillsdale, NJ: Lawrence Erlbaum Associates.
Whitaker, D and McGraw, P.V. Long-term visual experience
recalibrates human orientation perception. Nature Neuroscience 3:13.
Yeo. R. M.,
Yonebayashi, Y., & Allman, J. M. (1995). Perceptual memory of cognitively
defined contours: A rapid, robust and long-lasting form of memory. In B. Julesz
and I. Kovacs (Eds.), Maturational windows and adult cortical pasticity. Reading,
J. et al (2001). How
does the brain parse activities in the world into discrete perceptual
events? Nature Neurosci 4, 651–655
Halpern, A.R., Perry, D.W., Meyer, E., & Evans, A.C. (1996). Hearing in the
mind's ear: A PET investigation of musical imagery and perception. Journal
of Cognitive Neuroscience 8, 29-46.