The Neuroscience Behind Seeing Color

IRA FLATOW: This is Science Friday. I’m Ira Flatow. You’ve probably been in a hardware store and seen the wall of paint swatches, right? A giant color chart laying out the various shades of yellow that you’re going to possibly paint your kitchen.

But how do you know that your favorite mustard yellow really looks the same to everyone else? Interesting question, because how we perceive color is not, as you might say, so black and white. Producer Alexa Lim is here to tell us more. Hi, Alexa!

ALEXA LIM: Hey there, Ira.

IRA FLATOW: OK. So my experience of mustard yellow is it really that different from everyone else’s?

ALEXA LIM: Well, Ira, you might remember the viral white and gold dress photo from a few years ago.

IRA FLATOW: Yeah. Who can forget it? That’s where some people were seeing the colors of the dress as gold and white. And it was crazy. Other people were seeing it as black and blue.

ALEXA LIM: Yeah. People were really taking sides and saying, I’m this type of person or this type of person based on how they were seeing the dress. And it basically broke the internet for a few days. But it also broke people’s ideas about color perception.

And you would think it’s pretty straightforward. The company, you go to one of these color charts. You choose the specific dye. But what people were seeing and perceiving in that photo, they were very different colors.

IRA FLATOW: Yeah. I remember they certainly were. And it really caused fistfights sometimes about what people were actually seeing.

ALEXA LIM: Yeah, exactly. They were having different experiences. And people didn’t know how to wrap their brains around that. But it’s happened again.

IRA FLATOW: Again?

ALEXA LIM: Yes. It’s happened again. There was a shoe, the shoe of 2017 that kind of was the same phenomenon here. I want you– here’s a photo of it. And I want you to tell me what color you see.

IRA FLATOW: To me the sneaker looks like a combination of blue and greenish, sort of ghoulishly looking.

ALEXA LIM: OK. I mean, we’re on the same wavelength. No pun intended. You see it as blue and gray. So do I. But there were actually people who saw that shoe as pink and white.

IRA FLATOW: No! Pink and white?

ALEXA LIM: Yeah.

IRA FLATOW: No way you could see that as pink and white.

ALEXA LIM: Like I try to stare at it in different angles, but it’s always teal and gray to me.

IRA FLATOW: So why is color perception so complicated?

ALEXA LIM: Well, that’s a good question. It’s the usual culprit that complicates things, Ira. It’s your brain.

IRA FLATOW: Yeah, yeah. I’m with you on that also. Well, why is that? What’s happening in your brain there?

ALEXA LIM: Your eye takes in the signal, and then your brain interprets that data. And that is a complicated process. There’s actually research looking at how context and expectations play a factor into how your brain processes and perceives color.

IRA FLATOW: OK. I’m buying all of this, but I want you to tell me how we’re able to study how our brain perceives color.

ALEXA LIM: That’s a question I wanted to know, too, so I spoke to Bevil Conway, who is a neuroscientist at the National Eye Institute. He’s also an artist. And in his lab he’s mapping out how the neurons in our brain respond to color.

He’s trying to build a neurological color chart. It’s basically a Pantone system for our brain. And he’s trying to study color from a different perspective. I asked him how much of our color vision happens in our eye versus our brain.

BEVIL CONWAY: It’s a hard question to sort of parse out because it’s like a chain. How much of the car is dependent on the chassis versus the wheels versus the steering wheel? It’s the whole package that comes together that makes it an automobile. And color vision, our experience of it, depends on a whole integrated set of processes.

So the retina does a lot of heavy lifting to turn light signals into the electrical impulses that are the building blocks that our brain uses to construct a perception of color, but you couldn’t see color without the rest of your brain listening in on what the photoreceptors are doing. When I got into this business, I learned that there are two types of photoreceptors in the eye– rods and cones. And I thought that rods were for black and white vision and cones were for color vision. And it wasn’t until graduate school when I learned that, no, rods are just for low light vision. Cones are the photoreceptors you use for everything you think of as vision during normal daylight. Seeing black and white, and motion, and faces, and places, and navigation, and reaching movements are all dependent on the three types of cone photoreceptors.

So color vision really depends on the brain kind of extracting from the cone photoreceptor responses just the color component of the information. And that depends on a lot of neural circuitry, a lot of neuron circuits inside the brain that listen in on what happens at the photoreceptors. From an understanding of how photoreceptors work, one of the really key pieces of information is that the photoreceptors themselves are color blind. They can’t tell different wavelengths apart because each stimulus that you see not only has a wavelength, but it’s also got an intensity and amplitude of that wavelength. And both of those factors are going to affect how the photoreceptor works.

So it’s like going a certain distance. You can’t tell how far you’re going to go just by knowing the speed because you also have to worry about how long you were traveling, what your time was. And so the visual system kind of downstream of the photoreceptors doesn’t know all of these pieces. All it knows is, yep, a photoreceptor responded. The visual system has to do a lot of work to pull out the color information, given a really impoverished one little piece of information from the photoreceptor. And it’s that that means that the photoreceptor activity has to be compared across different photoreceptors with different spectral tuning functions for the brain to actually get a representation of color.

ALEXA LIM: Right. And you could say that like all of our senses have this perceptive quality to it. You hear sound, and your brain processes it, but you still have to perceive what exactly that is. What is special about color?

BEVIL CONWAY: What’s really quirky about color is that we just don’t have a very good objective measuring device, ruler for color. I started this business thinking that wavelength was it. When you look at an object, you could measure the amount of different wavelengths reflected from that object. And doesn’t that tell you the color? I mean, after all, we look at the rainbow and we see red, orange, yellow, green, blue. We label the different wavelengths with different colors.

But it turns out that actually wavelengths don’t really tell you what color you’re seeing. We can take exactly the same set of wavelengths and have two people see them very different colors, like that famous dress image. It’s the same physical light entering your eye under two different eyes. And they’re seeing it in two very different ways.

And we can go the other way. You can take two physically different things, like a yellow that’s in the middle of the rainbow. We call it monochromatic because it’s got a single peak. You can take that yellow, but you can mix a yellow with theater lights, just by projecting on the wall a red light and a green light, and having them overlap. Where they merge, that will be as vivid a yellow as the monochromatic yellow. Yet those two yellows are two physically very different sets of wavelengths.

So somehow the brain does this kind of alchemy to turn the photoreceptor signals into perception. It isn’t a simple reflex of translating wavelength into color. And it’s because really, ultimately, you need a brain in order to be able to see color, or reconstruct the colors as we experience them. Because there’s no measuring device that tells us what color there is, we’re sort of stuck on our own. We can’t appeal to some ruler out there and say, hey, tell me. Are you right or am I right? Because we’re sort of both right, and both stuck just with the color experiences we have.

ALEXA LIM: It was interesting to find out that the study of color perception and color is a fairly new field.

BEVIL CONWAY: Well, it’s tricky. I mean, on the one hand, the study of color psychology is a very old field that goes way back to the ancient Greeks. In some ways, a lot of people think that color is irrelevant because you can get so much out of vision just from black and white. I was surprised to learn recently that The New York Times only started publishing their newspapers in color in the late 1990s. That was long after you could pretty easily and cheaply make color photographs. And there was this whole debate in the editorial staff about whether or not it was valid to use color, like, did color actually carry some kind of meaning, or was it just sort of a visual cheesecake, a little bit of fluff because it was too subjective and squishy?

And I think that attitude permeates a lot of the kind of work that’s happened in visual neuroscience where we think of vision as telling us about what the stuff is out there in the world, like what are the identities of objects in the world. And to do that, you really don’t need color. I mean, you can do it perfectly well in a black and white picture. If you think vision is just to tell you what the stuff is there in the world, color doesn’t really do that for you.

And so I think a lot of people have kind of been scatty around in the dark thinking, well, maybe color just isn’t really what we should be thinking. Maybe it’s just a little extra frill that we acquired in the visual system that’s not really for anything. And that just rubbed me the wrong way when I was in graduate school. I was like, that can’t be true!

ALEXA LIM: Right. Well, that’s a good way to segue into I want to talk about what you’re doing in your lab. You’re studying color by looking at the neurons in the brain. What are you finding out?

BEVIL CONWAY: Yeah. We got really interested in this question of, what is color actually for in terms of behavior? If I ask you, what’s the color of a banana? A lot of people will say, well, a banana is yellow.

Then I sort of say, well, but hang on a minute. Are all the bananas you’ve ever seen yellow? And people are like, well, no, actually, I’ve seen green bananas, and black bananas. And actually, come to think of it, most of the bananas in the world are probably not yellow. So why is it that we think of bananas as yellow?

And I think the answer is that what you hear in my question, what are the colors of bananas that you care about? And it’s that little clause, what you care about, that I think points to something deep and fundamental about the role of color and behavior. And it’s why we’re studying it. So we’re interested in trying to understand how do visual objects acquire meaning. That is subjective meaning. And to do that, color becomes this really powerful tool.

A recent study, for example, what we were interested in doing is trying to figure out whether or not we could decode from the brain what colors people were seeing. Could we listen in on the brain activity directly, and then see from the pattern of brain activity could we decode, could we see what color people said they were looking at.

And to do that, we used this technique called magnetoencephalography. Each neuron fires an action potential. It communicates with electrical impulses. People just sit with one of these big caps over their head, and the cap measures these tiny little magnetic fields that are induced every time that the neurons fire.

And so we showed people colored stimuli. We asked them what colors they saw. And then we used these machine learning algorithms to basically learn the different patterns of activity, the association of those patterns of activity with what colors people saw.

ALEXA LIM: So can you then tell what color I am looking at by looking at my brain scan and seeing what neural pattern is happening, say, she’s looking at brown? Is that how it works?

BEVIL CONWAY: It’ basically exactly like that. So the way it works in real time is that we have to first present a whole bunch of colors. And then for each color, we get a pattern of activity in the brain. And we do it a whole bunch of times for the same color, and then lots of different colors. And then we train up these algorithms to learn the relationship between different patterns of activity that were elicited by each color and the color label that you assigned to it.

And then we can give you a brand new stimulus that you haven’t seen before, or a new example, or a new presentation of the stimulus. And we can say, OK, fancy algorithms, which color do you think that was? And our success, it varies a little bit, but it’s upwards of 80% or so of the time we get it right, which is way better than chance. And it’ll get better and better as the technology gets better and better.

So we can’t really do it in real time just yet, but for now, we can pretty reliably tell you what color you were seeing, given the pattern of activity in your brain.

ALEXA LIM: I’m Alexa Lim. And this is Science Friday from WNYC Studios.

And so then you also look at do people have similar relationships between different colors as well.

BEVIL CONWAY: So what we did is we looked at the pattern of brain activity to different colors. And we asked whether or not the colors that you think of as similar– so reds and oranges are kind of similar. They’re both warm colors. Do those colors cause similar patterns of brain response to colors in my brain and in your brain? And by the same token, colors like red versus green that are quite different colors, a warm color and a cool color, do they cause very different patterns of activity in your brain compared to my brain? And that turned out to be true.

Taking one step further is kind of fascinating because from those relationships, how the different patterns of brain relate to each other for different colors, we can then reconstruct a geometry of color space determined by the brain response itself. So this idea of a color space is a very old one. Anybody who uses a computer is familiar with the hue saturation value color space. And there are gazillions of different kinds of color spaces. Pantone has a very famous one.

Why are there so many different color spaces? If they’re all derived because of how the brain processes color information, shouldn’t there be like a color space defined by how the brain works? We could reconstruct the geometry of that neural representation. And it revealed all sorts of cool things, like there isn’t one static geometry. It’s actually kind of quite dynamic over time, which itself was like a big discovery. It’s sort of new to us.

ALEXA LIM: We’re taught that certain colors have certain meanings. A red bug can mean, I’m dangerous. Don’t eat me. I mean, how does this kind of biological aspect play into these ideas?

BEVIL CONWAY: Yeah. So we’ve spent a fair amount of time, as have lots of others, trying to figure out like, why do we have color vision in the first place? I mean, what are the evolutionary selective pressures that moved us towards having color vision? And some of the great old ideas are, well, color signals in a hard-wired way, as you’ve just described, like red means x. Red means danger.

It becomes a little complicated because red also means love. There’s lots of meanings that can be attached to colors. And so I think although it’s certainly true that there are selective pressures in evolution that pushed us to have color vision, I think the selective pressures are because color provides a nice way of capturing a lot of information about an object that’s divorced from its shape.

So there are lots of different red things. Not all dangerous things are red. And not all red things have the same shape. So it becomes a kind of abstract property that we can encode that we can then use to tell what things have meaning out there in the world.

And that’s– one of the great features of the nervous system, especially of the human nervous system, is its ability to solve problems that it hasn’t yet encountered. If we want to solve the problem of detecting the dangerous red bug, then you’ve got a problem. And evolution could work to try and solve how to detect the dangerous bug. And you might end up with a visual system that can detect red objects.

But it’s a harder problem to say, hey, nervous system, what I want you to do is to figure out how to get your wiring so that you can solve problems that you haven’t yet encountered, like bugs that may be different colors or different shapes, and so on. And I think that’s really what color vision is allowing us to do. It’s a kind of an adaptable system, a trainable system, where colors can take on new kinds of meanings depending on the environment, depending on what uses we place on the visual system.

ALEXA LIM: Well, we’ve run out of time. Thanks so much for joining us.

BEVIL CONWAY: It’s been fun to talk to you.

ALEXA LIM: Bevil Conway is a senior investigator at the National Eye Institute and an artist based out of Bethesda, Maryland. For Science Friday, I’m Alexa Lim.

Copyright © 2021 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of Science Friday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies/