The Visual System

The input is poor and the experience is not

Start with an honest audit of what the eye actually delivers, because the standard picture, in which the eye is a camera and the brain develops the film, collapses the moment you take that audit seriously.

First, there is a hole. At one point on each retina, the axons carrying every visual signal must leave the eye and become the optic nerve. Nerve fibres and blood vessels cannot pass through a sheet of photoreceptors without displacing them, so at that point, the optic disc, there are no receptors at all. You have a blind region in each eye, several degrees across, large enough to swallow a face at conversational distance. You have never seen it.

Second, the sheet is sharp almost nowhere. Acuity is high only in the fovea, a pit at the centre of the retina roughly a millimetre and a half across, subtending a couple of degrees of the visual field: about the width of your thumbnail at arm's length. Everything outside it is progressively blurrier. Hold your gaze fixed on one word of this sentence and try to read the word four words along without moving your eyes, and the point is made immediately.

Third, colour is a central luxury. Colour depends on cones, and cones are packed into the fovea and grow sparse toward the edge. Your far peripheral vision is close to colour-blind, a fact so at odds with experience that most people refuse to believe it until they test it by bringing a coloured object in from behind the head and noting when the colour, not the movement, becomes apparent.

Fourth, the image never holds still. The eyes make rapid jumps, saccades, several times a second, and during each jump the retinal image smears. You never see the smear.

Now set that inventory against your experience: a broad, stable, sharply detailed, fully coloured, gapless world. Not one of those properties is present in the input. Every one of them is manufactured. The correct conclusion is not that the brain has cleverly rescued a bad camera. It is that vision was never a matter of capturing an image in the first place. Vision is the construction of a best explanation of the scene from evidence that is fragmentary by design, and the anatomy at every stage is evidence for that claim rather than against it. The sections that follow trace the pathway from the retina to the temporal lobe, and at each station the same question is asked: what is this part of the machinery adding that was not in the light?

The retina is not film: it is brain that computes

The single most consequential fact about the retina is developmental. During embryonic development, the retina does not form as part of the eye and then get wired to the brain. It grows outward from the brain: the optic vesicle buds from the developing diencephalon, the same embryonic region that produces the thalamus and hypothalamus, and it carries its neural character with it. The retina is central nervous tissue that happens to sit in a socket in your face. It is not a peripheral organ reporting to the brain. It is brain, reporting to more brain.

That fact predicts what we find inside it. Film has one layer. The retina has three layers of neurons stacked in series, with two further classes of cells wired sideways across them, and a signal must cross at least two synapses before it can leave the eye. Nothing is put in the way of a signal for no reason. The layers are there because work is being done.

There is one further oddity, and it is worth confronting because it looks like bad design and turns out to be a clue. The retina is wired backwards. Light must pass through the ganglion cells and the bipolar cells, through the tangle of blood vessels, before it reaches the photoreceptors, which point away from the light. The one place where this arrangement is relaxed is the fovea, where the overlying layers are swept aside to form the characteristic pit and let light reach the cones as directly as possible. The inversion is a consequence of how the retina develops out of the neural tube, and evolution cannot undo an early developmental commitment. It is also the reason there must be an optic disc: the axons are on the wrong side of the sheet, so they have to dive back through it to escape.

Why the eye sends contrast and throws brightness away

Here is the finding that turned the retina from a piece of optics into a piece of computation. Record from a single ganglion cell and map the region of the retina that influences it, its receptive field, and you do not find a simple light detector. You find a bullseye. A small central disc has one effect, and a surrounding annulus has the opposite effect. An "on-centre" cell fires hard when light hits its centre, and is silenced when light hits its surround. An "off-centre" cell does the reverse.

Now work out what such a cell actually reports. Shine a light over the whole receptive field, centre and surround together, and the two effects oppose one another and largely cancel: the cell barely responds. Turn the room lights up, so that everything gets uniformly brighter, and the cell says almost nothing at all. But place the boundary between a light region and a dark region across the receptive field, so that the centre is lit and the surround is not, and the cell fires vigorously.

So an on-centre ganglion cell is not a brightness meter. It is a difference detector. It reports that the light here is greater than the light immediately around here. And that is a statement about contrast, which is to say, about edges.

Two familiar experiences fall out of this, and they are not illusions in the sense of failures. They are the machinery becoming visible. The first is that a grey patch looks lighter against a dark surround than the identical grey looks against a light surround, because the surround is subtracted. The second is that at a boundary between two greys you see a faint bright band on the light side and a dark band on the dark side, edges exaggerated at the moment of transition. Your visual system did not measure those bands. It manufactured them, because it is built to shout about differences, and a boundary is where the differences are.

This is the first real payment on the debt announced at the top of this page. The system does not transmit the scene. It transmits an argument about the scene, and it begins making that argument before a single signal has left the eye.

The chiasm: why the field crosses and the eye does not

The standard shorthand says that the left brain handles the right eye, and it is worth being blunt: this is simply false, and almost every consequence of the anatomy is missed by anyone who believes it. What crosses at the optic chiasm is not an eye. It is a half of the world.

The anatomy is precise and the logic follows from it exactly. Divide each retina down the middle into a nasal half, nearer the nose, and a temporal half, nearer the temple. At the chiasm, the X-shaped junction sitting just above the pituitary gland, the fibres from the nasal half of each retina cross to the opposite side. The fibres from the temporal half do not cross; they continue on their own side. Half of each nerve crosses, and half does not.

Now add the optics. The lens of the eye inverts and reverses the image, so light coming from your right lands on the left, that is, the nasal, half of the right retina and on the right, that is, the temporal, half of the left retina. Everything on your right is therefore seen by the nasal retina of the right eye and the temporal retina of the left eye.

Put the two facts together. The nasal fibres of the right eye cross to the left. The temporal fibres of the left eye stay on the left. Both of those carry the right half of the world. So both arrive in the left hemisphere. Run the same argument on the other side and the left half of the world, seen by the nasal retina of the left eye and the temporal retina of the right, arrives in the right hemisphere.

1. The world is projected, inverted, onto the retinas

   An object in the right visual field casts its image on the nasal retina of the right eye and the temporal retina of the left eye. The optics do this, and they do it for both eyes at once.

2. The chiasm sorts fibres by retinal half, not by eye

   Nasal fibres cross. Temporal fibres stay. The sorting rule takes no notice of which eye a fibre came from; it notices only which half of the retina it came from.

3. The result is a half-world, from both eyes, on one side

   The left optic tract now carries the entire right visual field, contributed by both eyes. The right tract carries the entire left field. Each hemisphere holds one half of the world, seen twice.

4. And that is the precondition for depth

   To compute depth from binocular disparity, the brain must compare the two eyes' views of the same point. That comparison can only happen if both views arrive in the same hemisphere. The chiasm's sorting rule is what makes stereoscopic vision anatomically possible.

That last step is the payoff and the reason the design is not arbitrary. If the crossing had been done by eye, as the folk version has it, the two views of any given object would sit in opposite hemispheres and the comparison needed for stereo depth would have to be made across the corpus callosum for the entire visual field. Sorting by field instead of by eye puts the two images of every point in the same place, side by side, where a single neuron can look at both. The chiasm is not a wiring quirk. It is the operation that makes binocular vision possible.

The anatomy is the diagnosis

Here the argument earns its keep in the clinic, and it is one of the most elegant results in all of neurology. Because the visual pathway sorts the world by field at a single known point, the pattern of what a patient cannot see specifies the location of the damage, often to within a few millimetres, before a scanner is switched on. A neurologist with a pen and a piece of paper can localise a lesion by charting the visual fields. Nothing else in the nervous system offers quite so clean a map from symptom to site.

Work through the three canonical cases, and note that each one is derived, not memorised.

The pathway continues to be informative behind the tract. Fibres running to the visual cortex, the optic radiations, split around the ventricle: the lower fibres sweep forward into the temporal lobe in a loop before turning back, while the upper fibres run more directly through the parietal lobe. The lower fibres carry the upper part of the visual field. So a temporal lobe lesion tends to knock out an upper quadrant of the opposite field, a parietal lesion a lower quadrant. Even a quarter of the world missing tells you which lobe to look in.

The LGN listens more to the brain than to the eye

The optic tract terminates in the lateral geniculate nucleus of the thalamus, and the textbook description calls the LGN a relay. That word should be treated with the same suspicion it deserves everywhere else in the thalamus, and the reason is a number.

If the LGN were a relay, you would expect its input to come from the retina. It does not. Retinal fibres, the ones actually carrying the picture, account for only a minority of the synapses on LGN neurons. The bulk of its input descends from the visual cortex itself, with further input from the brainstem and from the inhibitory shell of the thalamic reticular nucleus. Most of what the visual thalamus is listening to is not the eye. It is the brain, telling it how to treat what the eye is saying.

A relay does not need that. A relay copies its input to its output. The moment you find a structure whose input is dominated by feedback from its own target, you are not looking at a relay; you are looking at a gate, and the thing being gated is the flow of visual information into cortex. This is why the LGN's activity changes with attention and collapses into rhythmic bursting during sleep, when the visual world stops arriving even though the eyes may still be open under the lids.

The structure of the LGN also preserves a distinction the retina made and the cortex will need. It is layered, six layers in the human, and the layers keep the two eyes apart: alternate layers receive input from the left and right eye respectively, and no LGN neuron in a normal brain is driven by both. The eyes are not combined here. They are kept separate, carefully, and delivered separately to the cortex, which is the first place in the entire pathway where a single cell sees through both eyes. Binocularity is a cortical achievement, and the LGN's job is to protect the ingredients until they get there.

The layers also sort by cell type. The two lower magnocellular layers take input from large ganglion cells: fast, sensitive to motion and to coarse form, colour-blind. The four upper parvocellular layers take input from small ganglion cells: slower, high acuity, colour-sensitive. Thin koniocellular layers between them carry further colour signals. Two kinds of information about the same scene, a fast sketch and a slow detailed portrait, travel up separately, and it is not a coincidence that the two cortical streams described later have roughly this division at their roots.

Why V1's map of the world is deliberately distorted

The LGN projects to the primary visual cortex, V1, which occupies the occipital pole at the very back of the brain, mostly buried in the deep calcarine fissure. It is also called the striate cortex, after a stripe of white matter visible in a section, and it is where the visual scene first touches the cortical sheet.

V1 is organised retinotopically: adjacent points in the visual field project to adjacent points on the cortical sheet. There is, in that sense, a map of the world laid out on the back of your head. But it is a map drawn by a cartographer with strong opinions. It is grossly, systematically distorted, and the distortion is the interesting part.

The central region of the visual field, the part the fovea sees, takes up a share of V1 that is enormously larger than its share of the visual world. The far periphery, which occupies most of the field by area, is squeezed into a fraction of the cortex. If you drew the world as V1 represents it, the middle would be inflated like a balloon and the edges crushed into a thin rim.

The naive question is why the brain would distort its own map. The question rests on a mistake, and correcting it is the point of this section. Acuity does not follow retinal area. It follows cortical territory. How finely you can resolve a part of the visual field is set by how many cortical neurons are analysing it, and therefore by how much cortex it has been given. The map is not distorted relative to some true map that the brain ought to have built. The map is the allocation of processing, and the brain has allocated it where the information is worth having.

And that allocation is exactly matched to behaviour. You do not need to resolve fine detail everywhere, because you can move your eyes. A system that put its foveal machinery everywhere would need a visual cortex of preposterous size and would spend most of it analysing regions you are not interested in. A system that puts the machinery in one place and then aims that place, several times a second, using saccades, gets the same result for a fraction of the cost. The distorted map and the darting eye are two halves of one design: a small high-resolution processor and a fast pointing mechanism.

V1 has one more structural feature that matters. It is organised in columns running through its depth. Cells in a column share a preferred orientation, and orientation preference rotates smoothly as you move sideways across the sheet. Interleaved with these are ocular dominance columns, alternating stripes preferring one eye or the other, which is where the two eyes' inputs, kept separate all the way from the retina, finally lie down next to each other. Move a millimetre or two across V1 and you have passed through every orientation and both eyes for one small patch of visual field: a complete analysis of one place, packaged in a block. This modular arrangement, one bit of the world analysed exhaustively by one block of cortex, then repeated across the sheet, is the cortex's characteristic solution to almost every problem it faces.

Hubel and Wiesel: how complexity is built from simplicity

In the late 1950s David Hubel and Torsten Wiesel set out to find what made cells in the cat's visual cortex fire. They had a projector, a screen, and glass slides with dark spots on them, since spots were what the retina and the LGN liked. The cortical cells were almost entirely unimpressed. For hours the recordings gave little.

Then, while sliding one of the glass slides into the projector, a cell fired hard. It was not responding to the spot on the slide. It was responding to the faint shadow of the edge of the slide, sweeping across the screen as it was pushed into place. The cortex did not want spots. It wanted lines, and it wanted them at a particular angle.

The work that followed was published in the Journal of Physiology in 1959 and 1962 and it defined the field. Hubel and Wiesel described two principal cell types in V1.

And here is the argument that made the discovery seismic, because it was not merely a description of what the cells do; it was a proposal for how they could be built.

1. Start with what the LGN offers

   Centre-surround cells. Each says: it is brighter here than immediately around here. A dot detector, in effect, and unable on its own to say anything about orientation.

2. Line several of them up and add them together

   Take a row of on-centre LGN cells whose centres happen to fall along a straight line in the visual field, and wire all of them onto one cortical cell. That cell now fires only when the whole row is illuminated together, which is to say, when a bright bar lies along the line. Change the angle of the bar and it no longer covers the row, and the cell falls silent. You have just built a simple cell, and it responds to orientation, a property that none of its inputs possessed.

3. Now sum several simple cells with the same preference

   Take a set of simple cells that all prefer the same angle but sit at slightly different positions, and wire them all onto one further cell, so that any one of them can drive it. That cell now fires for a bar at the right angle anywhere in the region covered by its inputs. You have built a complex cell, and you have built positional tolerance out of parts that had none.

4. Then repeat the trick, upward, without limit

   Nothing about this operation is specific to edges. Converge a set of detectors for feature X onto a cell and you get a detector for the arrangement of X's. Do it again and you get a detector for arrangements of arrangements. Each step buys a more abstract feature and a broader tolerance, paid for with more cells.

That is the hierarchy, and its importance runs far beyond vision. It shows that a cortex made of simple, near-identical units, each doing nothing cleverer than adding up its inputs, can construct representations of arbitrary sophistication purely by stacking layers of convergence. You do not need a homunculus, and you do not need a picture. You need depth. Hubel and Wiesel shared the Nobel Prize in 1981, and the architecture they described, layers of local filters converging into progressively more abstract and more invariant units, is the direct ancestor of the convolutional networks that dominate machine vision today, a lineage the field's own founders have always acknowledged.

Their other great finding concerned development, and it was in some ways more disturbing. Deprive a kitten of vision in one eye during a critical early window and the cortical territory serving that eye is captured by the other, permanently, even though the eye itself is intact. Do the same to an adult and nothing much happens. The wiring is not fixed at birth and it is not plastic forever: it is written by experience during a window, and then it sets. This is why a child's squint or congenital cataract is a surgical emergency of a kind that the same condition in an adult is not, and it is one of the foundational results in the study of neuroplasticity.

Two streams, and why the split is not arbitrary

V1 is the entrance, not the destination. Beyond it lie dozens of further visual areas, and they are not arranged as a single ladder. They form two great pathways, leaving V1 and running in different directions, described by Goodale and Milner in 1992 in the formulation that has shaped the field since. The brain lobes page introduces the split; this is the deeper account.

The ventral stream is where the hierarchy of the previous section is carried to its conclusion, and the progression is a straight extrapolation of Hubel and Wiesel's construction. In V1, cells care about edges at particular angles in tiny patches of field. In V2 and V4, receptive fields are larger and cells respond to contours, to combinations of angles, to shapes, and to colour in a way that begins to correct for the illumination. By the time you reach the inferotemporal cortex at the front of the stream, receptive fields are very large and cells respond to whole objects, and they keep responding when the object is moved, rotated, shrunk, or relit. Each step trades positional precision for abstraction, exactly as summing simple cells into a complex cell did, and after enough steps you have a cell that reports what is there while having thrown away nearly everything about where.

The most-discussed waypoint in this stream is the fusiform face area, a patch of the fusiform gyrus on the underside of the temporal lobe, characterised by Kanwisher, McDermott and Chun in 1997, which responds far more strongly to faces than to other objects. Whether it is a module dedicated to faces by evolutionary design, or a region that becomes tuned to faces because faces are the category we are most expert at discriminating, remains genuinely disputed, and honest treatments do not pretend otherwise. What is not disputed is that damage there costs you faces. In prosopagnosia, patients can see perfectly well, describe every feature of a face accurately, and remain entirely unable to recognise whose face it is, including sometimes their own in a mirror, while recognising the same person instantly from their voice or their walk. Vision is intact. A specific act of visual identification is gone. That dissociation is very hard to explain if seeing is looking at a picture, and easy to explain if seeing is a stack of constructions, one of which has been removed.

The dorsal stream's independence is demonstrated by the mirror-image dissociation, and it is the reason Goodale and Milner argued the division is one of purpose rather than of content. Patients with damage to the ventral stream may be unable to report the orientation of a slot in front of them, or to match it by turning a card, and yet will post a card straight through that same slot, rotating the wrist correctly, without hesitation. The information about orientation is plainly in the brain; it is simply not available to the system that reports and recognises. It is available only to the system that acts. Conversely, patients with parietal damage can describe an object perfectly and yet reach for it clumsily, with a badly shaped grip. Seeing for knowing and seeing for doing come apart, cleanly, in both directions, and this is why the streams are better labelled by what they are for than by what they are about.

What vision is not

What remains contested

The account above is standard, but it is not finished, and a reference that pretended otherwise would be misleading.

Whether the fusiform face area is really for faces. One camp holds that faces are so important, and so uniform in structure, that evolution built dedicated machinery for them. Another holds that the region is a general engine for fine discrimination within a category of expertise, and that it handles faces because faces are the category almost everyone is expert in; on this view, a bird expert's fusiform should light up for birds. Evidence has been offered on both sides for three decades and the question is not settled.

How far the streams really are separate. The two-stream account is powerful, but the anatomy shows extensive cross-talk between the streams at many levels, and the strong claim that action and perception use wholly independent representations has been challenged by re-analyses of the grasping experiments. The consensus has softened from "two separate systems" to "two heavily interconnected systems with different priorities", and where exactly on that spectrum the truth lies is an active argument.

What all that feedback is doing. The cortex sends more fibres down to the LGN than the retina sends up, and higher visual areas send more fibres back to V1 than they receive from it. Something enormous is being computed in the downward direction. Predictive-coding accounts propose that the higher areas are constantly sending down predictions and that what travels up is only the error, the part of the input that was not expected, which would explain both the volume of feedback and the retina's habit of transmitting differences rather than values. The framework is elegant, it is influential, and it is not yet established.

How the pieces are bound. Colour, motion, form, and depth are analysed in partly separate areas, and yet you do not experience a red thing, a moving thing, and a round thing: you experience one red ball moving. Why the fragments unify, and what makes the unification happen, is the binding problem, and it is unsolved.

Related topics