How Signing Characters Help Deaf Children Learn Language

ARIELLE DUHAIME-ROSS: This is Science Friday. I’m Arielle DUHAIME-Ross. I want you to think back to your favorite childhood TV shows. Was it Blue’s Clues, Little Bear, Winnie the Pooh? Mine was The Simpsons, which is not exactly a kids’ show, but I did learn a lot. Animated TV shows are so important for kids because they can teach them to read, draw, spell, talk. And the creative ways in which shows tell stories, where they whisk kids away to some colorful, imaginary world, that contributes to the learning.

But shows like that aren’t accessible to every child, like deaf kids and children who are blind, for example, which could, in turn, affect how these kids learn language during those sensitive and formative early childhood years. My next guest is someone who is incorporating ASL, American Sign Language, into children’s media that’s made for and by the deaf community.

Melissa Malzkuhn is the founder and director of the Motion Light Lab at Gallaudet University in Washington, DC. Melissa is third generation deaf and is answering my questions in her native ASL. The person you’re hearing is an interpreter. Melissa, welcome to Science Friday.

MELISSA MALZKUHN: Thank you. Thank you for having me. Super happy to be here and to be part of this very awesome show, talking about science, culture, art, technology, all the fun stuff. It’s one of my favorite topics.

ARIELLE DUHAIME-ROSS: Yeah, thanks for coming on the show. So first of all, tell me about ASL access in early childhood.

MELISSA MALZKUHN: Oh, that’s a big question. [LAUGHS] So, speaking very, very generally, all children, deaf, hearing, otherwise, need language exposure to learn, to grow, to develop cognitively. And hearing children get language a bit more naturally, more easily, through being born into homes where they have access to the languages that their parents are using. Deaf children need visual language. And so for the most part, linguistically, that means a signed language.

And language acquisition, though, goes in the same way, whether it’s spoken language or sign language. Kids that are born into signing homes hit the same language developmental milestones that hearing kids born into speaking homes do. But now, what happens when deaf kids don’t have access to things on TV or media? There’s plenty of good content out there.

And a lot of people think that captions are the solution and that captions equal access for all. And of course, captions offer access to an extent, but we’re talking about kids that aren’t readers yet, kids that are still learning to read, kids that are still learning language, kids that are still developing vocabulary. So at that age, captions aren’t going to cut it.

And what we need, then, is sign language representation in children’s media. And there are such limited resources out there. There’s no TV show or series. There’s no real central core of access for deaf kids in children’s media. I think that if we provide that, not only deaf kids will benefit, but hearing kids will as well.

ARIELLE DUHAIME-ROSS: Right, so–

MELISSA MALZKUHN: So why not offer that? Why not give all kids more?

ARIELLE DUHAIME-ROSS: Right, so what you’re talking about is language deprivation.

MELISSA MALZKUHN: Yeah, it is certainly a consequence if a child doesn’t get access to language or good high quality language or if they get such minimal access to language that does result in that language deprivation. So to back up a little bit, I don’t think I shared this stat, but 95% of deaf people are born to hearing families. And that’s a global statistic.

So what that means is that deaf kids are born into an environment where their parents are navigating all kinds of questions about what it means to have a deaf kid, including the prospect of learning sign language. And what’s sad is that we do have a lot of resources in the deaf community to support these families, but the connection isn’t often made to hearing families and not as soon as it needs to be made.

And so that, I think, that there are real gaps in the early intervention system, the systems that parents interact with when they first find out that their kids are deaf. So hearing parents will sometimes not learn sign language at all or not learn how to sign until their kid is maybe two or three. And think about what that means, for a three-year-old to have barely had language exposure?

So that results in language deprivation. And you see lifelong consequences of that. They end up usually struggling to develop full literacy. And a lot of that can be traced back to what happens in early childhood. So that’s where we’re focusing our efforts. That’s what we want to fix to make sure that deaf kids have the same opportunities and access to life that anyone else does.

ARIELLE DUHAIME-ROSS: You mentioned something. You mentioned that some people feel like captions are universal access, but I want to talk about that a little bit more. What does most childhood entertainment look like for kids who are deaf and hard of hearing?

MELISSA MALZKUHN: So that’s another really good question. Well, so you do have some people that have put out ASL videos, videos in ASL. So my lab, for example, we have bilingual storybook apps that are available in sign languages and written languages. And it means that kids have access to great language models, high quality. Parents can use those as resources to interact with their kids. But it’s not like those storybook apps are going to substitute the full range of language exposure that a kid needs. It’s going to be more isolated if they’re just getting language exposure through these storybook apps.

On TV, I think a lot of deaf kids really primarily enjoy shows that are highly visual. There are some cartoons that can actually be more well-understood without language, so ones that are more in the slapstick genre of things, and that really are good for comedic value and entertainment value, but not educational value. So I think that what hearing kids get from shows like that are a lot more choices. They have shows that are entertaining and educational. Parents can choose what show they want their kid to watch at different times.

And with deaf kids, it’s more like, well, whatever is visually appealing, and then they’re filling in blanks and gaps. They’re relying on behavioral information that they’re seeing, the actions, but the meaning behind it, the labels for things, understanding the why isn’t there. So there’s a ton of guesswork on the part of deaf kids in their early years.

And I don’t want that to continue. I want kids to understand what things really mean. I want them to understand the context and really develop that knowledge base, start making those connections. Because, again, it’s a lot of filling in the blanks and making connections that might not always be accurate. So I don’t want their early years just to be a guessing game.

ARIELLE DUHAIME-ROSS: Yeah. What about the neuroscience of it all? Is there a difference in how brains process language when it’s seen versus heard?

MELISSA MALZKUHN: So I love this question. So it turns out that, actually– and I can say this because our lab is part of the Visual Language and Visual Learning Center, which is a research center based at Gallaudet University. There is a lot of neuroscience work among the labs in that center. So they use various types of imaging to see what’s going on in the brain. And over and over, they find that there actually is no difference. So the brain is actually just looking for patterns in language, the rhythms that are present in all language regardless of the modality.

So language can be spoken for hearing kids, and the brain will take that auditory input and process it. And for kids that are exposed to a sign language, they’ll take that visual language and process it in the same way. So it’s really fascinating that the brain doesn’t discriminate, and the brain isn’t choosing. It just wants those patterns.

And so ultimately, what it’s about is the timing of it, is getting that language exposure early. There really is a sensitive period, and it’s birth to three. So that’s really, really early. But those couple of years create such a foundation for the rest of our lives. And that foundation is what really allows for us to place all the rest of the building blocks that we get in life upon.

ARIELLE DUHAIME-ROSS: Right, and I mean you know this history so much better than I do as a third generation deaf person in the US, but ASL was not always considered on par with other languages. And that research that you’re talking about disproves that entirely.

MELISSA MALZKUHN: Exactly. So there are so many misconceptions, misinterpretations, misunderstandings about signed languages. I mean, even when my granddad was alive and when he was my age, deaf people, of course, valued using sign language, and they knew that it was integral to their communication, but they grew up in a society where their language was just perceived as signs.

And so even my granddad just said we used signs. They didn’t say sign language. There was definitely not any level of prestige or respect associated with the language. There was certainly a stigma to using a sign language. It wasn’t seen as something that you could use to express any sort of intellect or education. And that belief really was pervasive in the deaf community as well. So signs were seen as something more social.

It wasn’t until the 1960s or ’70s that linguists started to undertake research on what is now known as American Sign Language. And they realized, wait a minute, there is a grammatical structure here. And so they really flipped the whole conversation on its head to say this is actually a language, and specifically, American Sign Language because sign language is not universal.

And deaf people themselves at that time weren’t completely on board because they also were steeped in this belief that they were just using signs, that what they were doing didn’t have value. So that started a new movement. So after linguists could show that it really is a language, then bilingual education, the bicultural movement began.

By the time I grew up in the ’80s and ’90s, I was a part of that bilingual, bicultural philosophy in education, where people believed that you had to have a firm understanding of ASL and English. And it really led to better literacy outcomes. And so we’re now at a point where we certainly know better. And to add to that, there’s about 300 documented sign languages in the world. So linguists are still doing this research on sign languages globally.

ARIELLE DUHAIME-ROSS: Well, thanks for sharing that. That really puts it into context. So you and your team started out making storybook apps. If I opened one of them, what would I see?

MELISSA MALZKUHN: So that’s exactly how we got started. We designed storybook apps to help give a resource that would expose deaf kids and their families to sign language. And we wanted to help them own the reading experience. So we designed them to look like a traditional storybook in that you flip pages– of course, on a tablet. So you’re flipping the pages with your finger, but it looks like actual pages that are turning.

You see the words at the bottom of the screen, but then you can also hit the play button, and you’ll see a narrator that’s signing what is said on that page. So the kids can see the text. They can also click on individual words that will open video of just the narrator signing just that word. So they get word level exposure. They get the overall level of the sentence on the page.

And then we also have a part that they can watch the whole entire story be told. So that means they see part to whole. They might recognize the individual word, but not know what it means as part of the whole story. So they have the opportunity to interact with the storybook app in different ways so that they can build all the connections that they need from word to sentence to story meaning. And we’ve been doing that for over 10 years.

ARIELLE DUHAIME-ROSS: All right, so basically our learning– sorry, go ahead.

MELISSA MALZKUHN: Oh, I just was saying I can’t believe that we’ve been doing this for a decade now, a little over a decade. And oh, as part of that, we also have offered training for other people to make their own storybooks.

ARIELLE DUHAIME-ROSS: Oh, very cool. OK. And now you’re working on a TV show. Can you tell me about it?

MELISSA MALZKUHN: I’d love to. It’s so exciting. So this has been a huge dream of mine. It’s been one of those parts of my vision that’s been on the backburner. So as I said, we started these storybook apps about 10 years ago, and our very first one is called The Baobab. It’s a great story. It’s an original story. And the character’s name is Mavo. So we came up with the story. We came up with the character.

And again, Mavo is the main character. She’s deaf. She’s a curious, adventurous, spunky, little girl. And we had a couple of spinoff stories. So we’ve had subsequent storybook apps featuring Mavo. And so the community that our audience of people that download the storybook apps are already familiar with Mavo. They know her stories. She’s remained one of our most popular characters. So we figured why not make Mavo sign, instead of just featuring a storyteller in the apps?

So sort of a side story to this, while we were making the storybook apps, we also started working on motion capture in the lab to make 3D signing characters. We’ve been working on that for the past eight or nine years. And we did that because we certainly saw the need to have animated characters that could sign. We knew that it would be appealing to young kids, that it would offer representation.

And honestly, going into animation opens up worlds of possibilities in storytelling. So in terms of the imagination, you can show the power of imagination. Animation really is so powerful. It’s such a powerful tool in storytelling that can’t be overstated. And we wanted to really leverage that.

But the challenge that we faced in making high-quality, signing, animated characters is that we wanted to make sure that they were fluent, that the language that they could produce would be legible, believable. So to try to connect these two things, we had this amazing character model. We had been working on motion capture. And we now got to a point where we can have Mavo sign.

And she looks so great animated, signing, that we figured we’re ready to do a TV series. We’re ready to do a TV series, and she can be the central character of that series. She’ll be a signer. And we are partnering with an animation studio. It’s called Pig Mental Studios. They’ve got tons of experience in the industry that we don’t have, so we partnered to bring this vision to fruition and bring this series to the world. So we’re building a series called Here Comes Mavo. And that’s it.

ARIELLE DUHAIME-ROSS: All right. I mean, that sounds really incredible. I can’t wait to see it. But I do want to ask you, so would Mavo kind of be like Dora the Explorer, television in two languages except that in this case, of course, communication happens simultaneously.

MELISSA MALZKUHN: That’s a good question. In some ways, I’d say that you could say that they would be in the same category in that Dora is really well-known as a great educational show. We intend for Here Comes Mavo to also be educational. We have a curriculum as part of our series. We’ve worked with an educational consultant who worked on Dora the Explorer. So every episode certainly has an educational component to it. And that’s an integral part of our design. So in that regard, it’ll be similar to Dora.

But our goal is for Mavo to be a signer, but also to use gesture and visual communication in all of her interactions. So there won’t be spoken language as a component to the show. It’ll be entirely– all of the communication will be visual, and visual language will be centered. We also are placing this emphasis on visual communication, not spoken communication, in the series.

ARIELLE DUHAIME-ROSS: So what’s the process of filming and making Mavo? Do your actors have to get dressed up in a high-tech suit? We’re talking about motion capture, right?

MELISSA MALZKUHN: Exactly, yeah. So the process is, of course, that we need to have a live person create the content. So they’ll act out the scenes, and we’ll use motion capture. So we’ll have a team of talent. They’ll wear the high-tech suits that you’re talking about, capture them, and it’s in the post-production that we create the animation. And we assign characters to the data. It is pretty high-tech. And it’s exciting. It’s exciting to start working on, too.

Another really exciting thing about this is that we’ve done motion capture primarily with adults. But for the Mavo character, we’re going to be working with younger talent, younger actors. And because Mavo is going to be an 8-year-old character, we want there to be some authenticity there. So we aim to have talent who’s close to age 8 as the signer behind Mavo. So we’re really excited about working with talent, too.

ARIELLE DUHAIME-ROSS: That’s incredible. That sounds like so much fun. I do want to ask you, why is it so important to use motion capture for this? Why go through that extra layer of animation at all? Why not just draw the characters and have them sign that way?

MELISSA MALZKUHN: I can certainly understand that question. So, well, no, I mean, more hand-drawn animation or 2D animation has a lot of limitations for signing. So we’re focusing on 3D animation, partly because ASL and sign languages are 3D languages. Space is used as part of the language. And if we were to go with 2D animation, you would miss some of the information that needs to be conveyed with dimension.

So it’s not to say that there’s no place for 2D animation, and there’s no value. I think that people should always try to push the boundaries of whatever their mediums can do. But we wanted to design and build a series where people would be signers. And especially thinking about the fact that our audience is so young, they’re not already fluent in ASL. We needed to make sure that the language was as high quality as possible. We need them to be able to see the dimensionality of the characters, so to see a person signing from different angles.

And 3D motion capture allows us to do that, to move the camera, because you can still understand ASL if you look at someone from the side or even partially from behind. So that’s part of why we’re using motion capture. And then it also means that the language that you’re seeing in the series is based on real language. So all of the rhythm that’s part of the language, the synchrony with facial expression, is all present when you use 3D motion capture.

It is, of course, a more labor intensive process. But I think technology changes really fast. And any long or intensive process now is going to become more automated and more seamless over time. And so we just need people on it to make it better. So that’s the bottom line.

ARIELLE DUHAIME-ROSS: Melissa, you’ve been working on this project, these storybook apps and now this TV show for a really long time. What does it mean to you to create stories that deaf children will not only learn from, but also get to actually enjoy?

MELISSA MALZKUHN: I mean, it is sort of like my entire life’s work, beyond just my career. I absolutely love this work. And I kind of call it a spark. So there’s times where it’s like you see a kid’s face just light up because a character is funny or a kid reacts when a character does something wrong, or you see them– I don’t know– you know, like when a kid says, oops, what they’re really doing is making connections. You see those lightbulbs go off. You see those sparks get ignited. You see things spark their curiosity.

And when I see those moments, it just means so much to me. And that’s what always gives me my drive. It rejuvenates my belief in this work. And I believe in it just so much. And maybe it’s embedded in my DNA or something– I don’t know. I don’t quite know how to express it. Maybe this is too long of an answer, but to create an original ASL story– well, here, no, let me start over. There isn’t a ton of opportunity for this. There’s individual efforts. People have tried to do things on their own at home, like amateur efforts.

But there’s not been a place or space that really invests in a professional production like this, where storytellers are looked for, artists are brought in. There’s been no true investment in the creation or production of original work like this. And the process itself is a celebration of our language and culture, knowledge, history, community. And I think it’s not just about making cute stories or making things accessible. It’s bigger than that. It’s really about the human spirit from all kinds of perspectives.

What I really want is, as a person, I want deaf literature to have a place in human knowledge. And I want it to have a place as stories that are valued. I want us to have space. Our stories are real, and they’re cool, and they’re awesome. And yeah, that’s what I want. That’s what it means to me.

ARIELLE DUHAIME-ROSS: Absolutely. I think that’s a really good note to end on. Melissa, thank you for joining me. Oh, sorry, hang on. Let me do that again. Melissa, thank you for joining me.

MELISSA MALZKUHN: Certainly. It’s absolutely been my pleasure to be here. And thank you again for the opportunity to have the conversation. I’m really excited about this work. And I can’t wait for everyone else to get to see it. I believe that all children are going to enjoy and benefit from the magic of sign and visual communication in their very own homes, in children’s media on their own televisions. So I’m looking forward to that day.

ARIELLE DUHAIME-ROSS: Same here. Melissa Malzkuhn is the founder and director of the Motion Light Lab at Gallaudet University in Washington, DC. To watch my full interview with Melissa or see videos of her lab’s work, go to sciencefriday.com/lightlab. Thank you to Jennifer Vold for interpreting and to Jenna Beacom for consulting on this segment.

Copyright © 2023 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of Science Friday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies/