Words jump-start vision, psychologist’s study shows

Cognitive scientists have come to view the brain as a prediction machine, constantly comparing what is happening around us to expectations based on experience — and considering what should happen next.

“These predictions, most of them unconscious, include predicting what we’re about to see,” says Gary Lupyan, a University of Wisconsin–Madison psychology professor.

Work in Lupyan’s lab has demonstrated the predictive process through manipulating the connection between language and vision in the brain. A study published recently in The Journal of Neuroscience shows that words have a profound effect even on the first electrical twitches of perception.

Lupyan’s collaborator, Bastien Boutonnet of Leiden University in the Netherlands, showed people dozens of pictures from a group of 10 common objects or animals. Before each picture appeared, participants first heard a word (“dog,” for instance) or a nonverbal sound (such as a dog barking). The study subjects simply decided whether the word or sound matched the image.

In an earlier study, Lupyan showed that people are faster to recognize pictures after hearing words than after hearing highly familiar nonverbal sounds. The new study by Boutonnet and Lupyan may have revealed the mechanism behind this “label advantage.”

Their study subjects wore nets of electrodes on their scalps, recording electrical activity in their brains as they viewed the images. The results showed that hearing words versus non-word sounds made a significant difference in a well-known peak in brain activity occurring within one-tenth of a second after the eyes fall on an image.

The study, supported by the National Science Foundation, is the first to show that a word cue — or a cue of any kind — has such a basic effect on the way the brain processes visual information.

“Even in that first hundred milliseconds of the earliest stages of visual processing, half a second before they respond, you can see language shaping perceptual mechanisms to make more effective predictions of what is about to occur,” Lupyan says.

The results leave a puzzle: Nonverbal sounds like a bark can carry quite a bit more information than what is carried by a simple word like “dog.”

“The volume and pitch of the bark can tell you not only that there’s a dog nearby, but the size of the dog and whether it’s angry or frightened or playful, right?” Lupyan says.

So why are labels better than nonverbal sounds in helping us recognize what is to come?

In a study recently published in the journal Cognition, Lupyan and Pierce Edmiston, a graduate student at UW–Madison, write that all those specifics, while handy, are not necessarily relevant to the fundamental dog-ness of dogs in general. The word “dog” cuts through to the most simple and general information necessary for recognizing a picture as a dog.

“Words are ideal for activating categories in the mind,” says Lupyan. “Think about a guitar — you can say ‘guitar,’ leaving unspecified whether it’s an electric or acoustic kind. But if you were to see or hear a guitar, you would know immediately whether it’s an electric or an acoustic one. Our senses cannot be ambiguous about this.”

The only way to meaningfully convey the general notion of a guitar may be by using language.

“Language allows us this uniquely human way of thinking in generalities,” Lupyan says. “This ability to transcend the specifics and think about the general may be critically important to logic, mathematics, science, and even complex social interactions.”