Sensory Processing

The inverse problem: the input is not enough

Hold up your hand and look at it. On the back of each eye, a two-dimensional sheet of photoreceptors is registering a pattern of light. That is all the evidence your visual system has. Now ask what could have produced that pattern.

A hand, at that distance, of that size. Or a much larger hand, much further away. Or a flat photograph of a hand. Or a peculiarly shaped object that happens, from this one viewpoint, to project the same outline. Or a hand of a different colour under differently coloured light. Every one of these is consistent with the pattern on your retina, and there are infinitely many more. The image does not determine the world. It never did.

This is the inverse problem, and it is not a curiosity of vision. It is the general shape of every sensory task. The same air-pressure waveform at your eardrum could have been produced by one voice at one distance or a different voice at a different distance, by a speaker or by a person. The same pattern of pressure on your fingertips could be a corner or an edge or a curved surface touched at an angle. In each case the physics runs forwards, from world to sensation, without loss; run it backwards and information has been thrown away, so the backwards journey has many possible endpoints and no way to choose between them on the evidence alone.

So the brain cannot simply read the world off its receptors, and any account of perception that says it does is wrong at the first step. The brain must instead guess: it must select, from the infinite set of worlds compatible with the input, the one that is most probable. And to do that it needs assumptions, prior beliefs about how the world tends to be, which it did not get from the current input and could not have got from it.

Where do these assumptions come from? Two places. Some are built in, laid down over evolutionary time by the fact that our ancestors lived in a world with particular regularities: light comes from above, because for four billion years there has been one sun and it has been overhead; objects are usually rigid; surfaces are usually uniformly coloured; the world is usually not conspiring against you. Others are learned, tuned by a lifetime of experience with the particular faces, voices, rooms, and objects that populate your world.

These assumptions are almost always correct, and that is exactly why perception feels effortless and truthful. You are not aware of guessing, because the guess is nearly always right. But it remains a guess, and this has a testable consequence that turns out to be the most useful fact in the whole of perceptual science: if you can construct a stimulus that violates one of the assumptions, the brain will apply the assumption anyway and produce a confidently wrong answer. That is what an illusion is. Every good illusion is a well-aimed attack on a particular assumption, and it works because the assumption is a good one. We come back to this at the end, once the machinery is on the table.

One currency: everything becomes voltage

Before the brain can infer anything, the physical world has to be converted into something a neuron can handle. This is transduction, and it is the first step of every sense without exception.

The physics could hardly be more varied. A photoreceptor must capture a photon. A cochlear hair cell must respond to a mechanical deflection of a few nanometres, a distance smaller than most viruses. A skin mechanoreceptor must register indentation. A thermoreceptor must register a change of a fraction of a degree. A taste cell must register a dissolved molecule. Five completely different problems in five completely different branches of physics.

And all five produce the same output: a change in the voltage across a cell membrane. In every case the mechanism is an ion channel, a protein pore that opens or closes and lets charged particles cross. In the hair cell, the channel is opened mechanically, by a tiny filament pulled taut when the hair bundle bends. In the photoreceptor, a cascade triggered by light closes channels that were open in the dark. In the skin, stretching the membrane deforms a channel directly. The engineering differs; the output is identical in kind.

The price of that universality is the problem the next section solves. Once everything is voltage, everything looks the same. Something else must supply the difference.

Meaning lives in the destination

The signals that reach the brain are, considered as electrical events, interchangeable. An action potential in the optic nerve and one in the auditory nerve are the same size, the same duration, the same shape. So the meaning cannot be in the signal. It has to be somewhere else, and it is: it is in the wire.

This is the labelled line principle. A signal arriving at the visual cortex is experienced as vision because it arrived at the visual cortex, and for no other reason. The pathway is labelled by its destination, and the label is fixed by anatomy, laid down in development before any stimulus ever arrived.

The evidence is not subtle. Press gently on your closed eye and you see a soft glow, a phosphene, because mechanical pressure has forced retinal cells to fire and the visual pathway reports its traffic as light regardless of what caused it. Compress the ulnar nerve at the elbow, the funny bone, and you feel tingling in your little finger, not at the elbow where the pressure actually is, because those fibres are labelled for the skin of the hand. A cochlear implant produces hearing in a person with no working cochlea, by stimulating the auditory nerve electrically. In every case the sensation follows the line, not the cause.

The consequence for this page is the important one. If meaning is carried by which line is active, then the brain's job is never to decode a signal; it is always to infer a cause. The signal tells it only that this line fired at this rate. Everything else, that there is a hand out there, that it is at arm's length, that it is yours, is inference built on top. The full development of labelled lines is in the systems hub.

Why the nervous system refuses to report absolute values

Here is a fact that looks like a bug and is in fact the single most important design decision in sensory neuroscience. Sensory neurons do not, on the whole, tell you how much of something there is. They tell you how much things differ.

The mechanism is centre-surround organisation. A retinal ganglion cell, for instance, does not simply report the light falling on its patch of retina. It is excited by light in a small central region and inhibited by light in a surrounding ring. Illuminate the centre alone and it fires hard. Illuminate the whole area evenly, centre and surround together, and the excitation and the inhibition cancel: the cell barely responds at all, even though a great deal of light is falling on it. It is deaf to uniformity. It shouts only at boundaries.

The same organisation appears in touch, where a mechanoreceptor's afferent is suppressed by activity in its neighbours, and in hearing, where lateral inhibition sharpens the tuning of frequency channels. The underlying mechanism, lateral inhibition, is one of the most reused circuits in the entire nervous system.

Now derive why. Consider a white page in bright sunlight and the same page in a dim room. The absolute amount of light reaching your eye from the sunlit page can be thousands of times greater. Yet the page looks white in both, and the letters look black in both. If your visual system reported absolute intensity, the sunlit black ink would register as brighter than the indoor white paper, and you would be unable to read either. The information you need, the thing that stays constant and that actually corresponds to something in the world, is the ratio between the ink and the paper. Absolute intensity is a fact about the illumination. Contrast is a fact about the object.

There is a second reason, and it is about cost. Natural scenes are highly redundant: neighbouring points on a surface are almost always similar, so a system that transmitted every value independently would be sending the same number over and over. Edges and changes are where the information is, in the strict information-theoretic sense that they are the parts you could not have predicted from their neighbours. Centre-surround organisation is, in effect, a compression scheme. It discards the predictable and transmits the surprising, which is precisely what a channel with limited capacity should do.

Maps that are orderly and deliberately wrong

Sensory pathways are laid out topographically. Neighbouring points on the receptor sheet connect to neighbouring points in the thalamus, and neighbouring points there connect to neighbouring points in the cortex. Vision has retinotopy: the visual field is laid out across the occipital cortex. Hearing has tonotopy: sound frequency is laid out as a strip across the auditory cortex, low at one end and high at the other, inherited from the physical layout of the cochlea. Touch has somatotopy: the body surface is laid out across the postcentral gyrus.

The immediate reason for this is prosaic and worth saying, because it is often left out. Wire is expensive. Axons cost space, myelin, and energy to maintain, and long axons cost more than short ones. If neurons that need to talk to each other are placed next to each other, the total wire length drops enormously. Topographic maps are, in large part, a wiring-cost solution. Neighbouring bits of the world need to be compared, so put them next to each other.

But the maps are not faithful, and this is where they become interesting. The somatosensory map does not allot cortex in proportion to body area. Draw a figure whose parts are scaled by their cortical representation, the famous sensory homunculus, and you get a grotesque: colossal hands, an enormous mouth and lips, a face out of all proportion, and a trunk and legs shrunk almost to nothing. The lips and fingertips occupy more cortex than the entire back.

The distortion is not a flaw. It is cortical magnification, and it follows from what the maps are for. Cortex is a finite, expensive resource, and the amount of it devoted to a body part determines how finely that part can be resolved. Your fingertips need to resolve texture at the scale of a fraction of a millimetre, because you use them to identify objects by touch; your back does not, because you do not identify anything with your back. Two-point discrimination on the fingertip is a couple of millimetres; on the back it is several centimetres. The map is not distorted relative to the body. It is proportioned to the job, and the body is the wrong yardstick to judge it by.

Vision does the same thing. The fovea, the central pit of the retina where cone density is highest, subtends about two degrees of visual angle, roughly a thumbnail at arm's length, yet it commands a disproportionate share of primary visual cortex. This is why you have to look directly at text to read it, and it is also why your eyes are in constant motion: you are dragging that tiny high-resolution patch around the scene, several times a second, and stitching the results into an apparently uniform world that does not exist anywhere on your retina.

Finally, the maps are not fixed. Cortical territory is reallocated with use. In someone who has lost a hand, the cortical area that used to represent it does not go silent; neighbouring representations, often the face, expand into it, which is part of the story behind the phantom limb. Intensive training reshapes maps in the other direction, expanding the representation of a trained finger or a practised sound. See neuroplasticity.

Building complexity from simple parts

No single neuron can recognise a face from raw light, and it would be a bad idea to build one that tried. The cortex instead builds complexity in stages, and the classic demonstration of how is one of the most consequential experiments in twentieth-century biology.

In 1962, David Hubel and Torsten Wiesel published a study in the Journal of Physiology in which they recorded from single neurons in the primary visual cortex of the cat while showing it patterns of light. What they found was that these neurons were largely indifferent to spots of light, which drive retinal cells nicely, and instead responded vigorously to oriented bars and edges. A given cell would fire hard for a bar at, say, forty-five degrees and be nearly silent for the same bar rotated ninety degrees. They called these simple cells. Other cells, which they called complex cells, also preferred a particular orientation but no longer cared exactly where in the receptive field the bar fell, and many preferred it moving in a particular direction.

The interpretation was the important part. A simple cell's elongated, orientation-selective receptive field could be constructed by summing the input from a row of centre-surround cells lined up in space. A complex cell's position-tolerant response could then be constructed by summing the input from several simple cells with the same preferred orientation at slightly different positions. Complexity is built by convergence: at each stage, cells pool the outputs of the stage below in a specific pattern, and the pooling creates a new, more abstract property that no individual input possessed.

1. Stage one: points

   Retinal ganglion cells and the thalamic relay report centre-surround contrast: a spot, a local difference. Nothing here knows anything about shape.

2. Stage two: oriented edges

   Simple cells in primary visual cortex pool aligned centre-surround inputs and become selective for the orientation of a bar or edge. An orientation is not present in any single input; it emerges from the arrangement of them.

3. Stage three: tolerant edges and motion

   Complex cells pool simple cells of the same orientation across nearby positions, yielding a cell that signals "an edge at this orientation, somewhere around here," and often "moving that way." Position tolerance has been bought at the cost of position precision, deliberately.

4. Stages beyond: shapes, objects, faces

   Higher visual areas along the temporal lobe pool these in turn, producing selectivity for curvature, texture, shape, and eventually for object categories, with the strongest known example being cells in the inferior temporal cortex that respond preferentially to faces and are relatively indifferent to the size, position, and viewpoint of the face.

One caution, because the field has learned it the hard way. The neat ascending hierarchy is real but incomplete. Cortical areas are not stacked in a clean line; they are massively interconnected, with abundant lateral and backward connections, and information does not simply flow up. The hierarchy is a good first approximation and a bad final theory, which brings us to the traffic problem.

Bandwidth spent on news

Put on a shirt and for a few seconds you feel it on your shoulders. Then you do not. The shirt has not moved and the mechanoreceptors in your skin are still being deformed by it, but their firing has fallen away and the sensation has gone with it. Walk into your own home and you smell nothing, though a visitor smells your home immediately and a stranger's home smells vividly to you. Sit in a room with a refrigerator hum and within a minute you cannot hear it, until it switches off and the silence is startling.

This is adaptation, and it is systematic. Receptors and the circuits above them reduce their response to a stimulus that does not change. It happens at every level, from the receptor protein itself up to the cortex, and it happens on timescales from milliseconds to hours.

The tempting explanation is fatigue: the cells get tired. That explanation is wrong, and the evidence against it is available for free. Adaptation is far too fast, too specific, and too reversible to be exhaustion. A neuron adapted to one orientation still responds briskly to another; a nose adapted to one odour still detects a different one at once. Fatigue would not be selective. Adaptation is.

The correct explanation is a design decision, and it falls out of the argument about contrast. Information lives in change. A stimulus that has not altered since the last report carries no news, and a channel with finite capacity that keeps re-transmitting old news is wasting itself. By adapting, a sensory neuron continually re-centres its sensitivity on the current baseline, so that its limited dynamic range is spent representing deviations from what is already true rather than re-representing what is already true.

The gain is enormous. It is what allows your eye to work across a range of light intensities spanning many orders of magnitude, from starlight to noon, using cells whose individual firing rates can only vary over a modest range: the system does not encode absolute brightness, it re-baselines and encodes departures from the current level. The same logic runs the motion aftereffect, in which staring at a waterfall for a minute leaves the rocks beside it drifting upward when you look away, because the downward-motion detectors have adapted and the now-unopposed upward detectors report a motion that is not there. The aftereffect is the adaptation becoming visible, and it is direct evidence that motion is coded by the balance between opposed detectors rather than by any one of them absolutely.

The cost is real, and it is worth naming. You are functionally blind to whatever is constant in your environment. Chronic problems, a persistent smell, a background noise, a mild ongoing discomfort, tend to disappear from awareness even though they persist. The system was not built to keep you informed about steady states. It was built to catch what is new, because what is new is what might need a response.

The traffic runs the wrong way

Everything so far has described a flow: receptors to thalamus to cortex, simple to complex, bottom to top. Now here is the anatomical fact that spoils that picture, and it is not a small one.

Count the fibres running between the visual cortex and the lateral geniculate nucleus of the thalamus, and the fibres running back down from cortex to thalamus vastly outnumber the ones running up. Indeed, in the lateral geniculate itself, the synapses actually carrying information from the retina, the ones supposedly doing the sensing, are a minority of the total. Most of what the visual relay is listening to is not the eye. It is the cortex, and the brainstem, telling it how to treat what the eye is saying.

You cannot fit that into a feedforward story. If perception were a matter of reading the input and passing it up, the cables would run mostly upward, and they do not. The wiring says, unambiguously, that a great deal of what determines what reaches your cortex is decided by what your cortex already expects.

The phenomenology matches the anatomy. Read a smudged word in a sentence and you will read it correctly, because the sentence told you what to expect; excise the same smudge and show it alone and it is illegible. Listen to a heavily accented speaker and, after a minute or two, the accent seems to clear, though nothing about the acoustics has changed. Play a degraded recording of a spoken sentence and it is noise; be told what the sentence says, play it again, and you will now hear the words clearly and will be unable to un-hear them. Your prior knowledge did not help you interpret an unchanged percept. It changed the percept.

Predictive processing. The most influential current framework for all of this reverses the usual arrow. On this account, the brain is continuously generating predictions about its sensory input, sending them down the hierarchy, and comparing them against what actually arrives. What flows up is not the raw input but the prediction error, the part the prediction failed to account for. Perception, on this view, is the brain's current best hypothesis about what is out there, revised whenever the errors get large enough to force a revision. The framework accounts elegantly for the backward-connection asymmetry, for the constructive character of perception, for illusions, and for the fact that expectation demonstrably alters experience.

It is important to be straight about the status of this. Predictive processing is a framework, and a productive one: it has generated a great deal of good experimental work and it unifies findings that were previously scattered. It is not an established fact. Serious researchers dispute whether the specific claim, that a canonical cortical microcircuit computes prediction error, is supported by the physiology; some of its formulations are difficult to falsify; and alternative accounts of the same phenomena exist. Treat it as the best available organising idea, not as a settled result, and be suspicious of any source that presents it as proven.

The senses are not separate channels

The tidy picture, one sense per pathway, one pathway per cortex, is convenient and false. The senses talk to each other, constantly, and they do it early enough and hard enough to change what you consciously experience. The cleanest proof is an experiment you can run on yourself.

Derive the principle from these. If the input to each sense is ambiguous, and it is, then each sense is making a guess under uncertainty. Two uncertain estimates of the same quantity are better than one, and the correct way to combine them is to weight each by its reliability. That is what the brain does. Vision is more reliable than hearing for location, so vision wins when they conflict, which is ventriloquism. Vision carries real information about speech, since the mouth's movements constrain what sound could have been made, so vision gets a vote, which is McGurk. Neither is a malfunction. Both are what a well-designed inference machine ought to do with two noisy sources of evidence about one event.

And notice that this closes the argument that opened the page. The senses combine because none of them, alone, determines the answer. Multisensory integration is not an added feature on top of perception. It is the inverse problem being attacked with every scrap of evidence available.

What illusions actually prove

Related topics