Will Pokémon Go Push Augmented Reality Forward?

IRA FLATOW: This is Science Friday. I’m Ira Flatow. Monster hunting game Pokemon Go is now more popular than Tinder and Twitter by the latest count. It has better engagement numbers than Facebook. And it’s getting kids and adults walking around outside for miles a day in search of rare finds, like the dragon Charizard.

On Wednesday, producers Xochitl Garcia and Brandon Echter took me to hunt Pokemon in Manhattan’s Bryant Park. We’re walking around now trying to get to my goldfish, because it’s out of range.

XOCHITL GARCIA: Your Goldeen.

IRA FLATOW: My Goldeen. Thank you, Sochi. My Goldeen. Goldeen. I did find it. One of the features that makes this game so enthralling is its augmented reality features. In other words, it adds stuff to what you’re doing. The game follows your actual location with items that you can pick up at local parks or on landmarks. And the Pokemon cute little cartoon characters show up in real life on your phone camera. So you can see them like sitting on your kitchen table. At Science Friday, we found a Squirtle– you’ll find out what that is– at our editorial meeting this week.

But that’s in some ways pretty basic for the possibilities of augmented reality. And my guests, who work to research this technology, say there’s a lot more we could do, even soon, with AR as they call it, augmented reality. From helping us repair our kitchen sinks, guiding doctors in surgery, there is a growing interest in the technology to allow us to enhance our interactions with the real world. Why not augment your tour of the museum with narration of graphics?

If you’d like to join us and talk about Pokemon Go, we’d like to hear from you. 844-724-8255. What fun– I want to know what fun or practical use you see for augmented reality. What would you like augmented reality to do that it doesn’t do for you now? Number again, 844-724-8255. You can also tweet us at scifri.

Let me introduce my guest to talk about Pokemon Go, augmented reality, what could come next, Dr. Blair MacIntyre. Professor of Interactive Computing at Georgia Tech. Welcome to Science Friday.

BLAIR MACINTYRE: Thanks for having me.

IRA FLATOW: Dr. Yan Xu, an augmented reality researcher who’s worked for Intel, and focuses on human game interactions. Welcome to Science Friday.

YAN XU: Thank you. I’m glad to be here.

IRA FLATOW: You’ve been playing Pokemon Go since it came out. What’s your take on what makes it so captivating? Dr. Xu.

YAN XU: So, I like the Pokemon Go a lot. And one thing that I really like about it is that these adorable characters– and I think that’s sort of like appealing to the audiences, like kids and casual gamers that is probably not the typical hardcore gamers that you see– but it brings a lot of social fun. Like the family fun that parents and kids are having together, or having fun with your friends exploring the environment. So, yeah, I enjoy this aspect of it. And another aspect is the exploring the environment around my neighborhood.

IRA FLATOW: Well, that’s what I’m going to ask. It’s built in that you have to get up and go walk around. And you can’t even drive, you have to be walking. Was that intentional?

YAN XU: And that’s great. And I get more reasons to walk around.

IRA FLATOW: Blair, what aspects of the game would you consider augmented reality?

BLAIR MACINTYRE: So I think– you could think of augmented reality across a spectrum from just augmenting what you know about your location, so that’s where you see on the map where the Poke Stops and the Poke Gyms and the Pokemon are.

And then sort of being a little bit more intimate. You’re augmenting your view of the world by overlaying the Pokemon on the video of the camera and using the phone’s orientation sensor so that when you look left and right, it looks like the Pokemon’s sort of sitting out there. And that is sort of the next level.

And then that’s about what you can do with phones right now. And I think, over time, we’ll see a lot more tighter integration of game characters like the Pokemon into the world. But right now, that’s about what you can do.

IRA FLATOW: You know, virtual reality was hot, hot, hot. And suddenly this augmented reality just appeared almost out of nowhere, Blair. Why augmented reality and not virtual reality?

BLAIR MACINTYRE: So for me, you know, I’d echo what Yan was saying, where it’s a question of whether you want to bring the virtual world, bring the stuff– the Pokemon, the games, the education– into the world with you. Or whether you want to sort of escape into a purely virtual environment.

And so, what augmented reality does is let us bring the computer into the world and share it with other people, versus say escape into a purely virtual world, which we might share with people who aren’t there with us right now, but it’s a sort of a trade off. It’s a different kind of thing.

IRA FLATOW: Yan, you’ll have to explain this to me, because this is just– as a geek myself– this is just so fascinating how the technology works to let you see a Squirtle or a Pidgey hanging out in your real life kitchen, or your office conference room. How does the game know to put those things there? And how does it get them there?

YAN XU: So there are– as Blair just mentionee– that there are two aspects. One is that it knows your location through the GPS. So the development team already have a lot of knowledge from their previous explorations of augmented reality location based games, like Ingress. So they know these kind of hot spots where people go to. And that’s where these creatures are living in. And so that’s one aspect of it.

And the second aspect is that they’re using– currently using gyroscope to place the character around you, so that your orientation matters when you have a close proximity to your characters. And when you rotate to the left, they will be there. And then when you rotate to the right, they would disappear because of the gyroscope based tracking.

IRA FLATOW: Could the algorithm in there make a mistake? I saw a report online, for example, that one of the characters was placed at the Holocaust Museum that gave off poison gas. It’s not a great choice for the Holocaust Museum.

YAN XU: No. I think it shows that there could be this kind of glitch between the understanding we have digitally vs. what’s going on in the real world.

IRA FLATOW: Blair, let’s talk about the real world. And this is fairly simple. It’s one of the first augmented reality games for the masses. What could more advanced AR for Pokemon, or any other kind, look like?

BLAIR MACINTYRE: Well, so, I described that there’s the first level of just using the map and knowing who they are. And then going to the camera and using the orientation of your phone to sort of make it look like it’s kind of around you.

The next step would be possible if the phone knew more about the world. So if the phone knew where the roads were, where the objects were, what the structure of the room around you– there’s a table here or whatever– then it could start actually making it look like the Pokemon, for example, was sitting on the table in front of you. And so that would mean that now I can’t just maybe turn left and right, I can move around and look at it from different viewpoints.

And when we can do that, that opens up all kinds of possibilities to, for example, create multiplayer or shared experiences because now, say, the three of us sitting around a table could look and see some Pokemon or we could each put one of our Pokemon down. And now we could actually have a combat on the table and see what each other are doing and so on.

And so I think the more you know about the world, the more the phone understands about the reality around you, the more opportunities you have to integrate the graphics into the world.

IRA FLATOW: I just mean– just wondering out loud. I’m sorry. I didn’t mean to interrupt, but we used to have augmented reality called Second Life. Science Friday was one of the first people, the first groups, in Second Life. We actually broadcast the show into there. That didn’t take off anywhere. It was not a great success. Why is Pokemon Go doing so much better than something like that?

YAN XU: So one that I can think of is that Second Life, you are still controlling an avatar. So you are like a puppet master, to some extent, that you are really–

IRA FLATOW: Oh, I think we lost her. We’ll try to get her back. Blair, what do you think?

BLAIR MACINTYRE: Actually, I’ll pick that up. So Second Life is much more like virtual reality, right? Where it’s a pure virtual world that you’re interacting with. We actually had, for a while, a bunch of islands in Second Life in our research lab. And we created an augmented reality version of the client, where you could take a part of Second Life, put it in our lab, and then anybody who walked in there in Second Life we would see them on a head mounted display in the lab.

And that created a very different kind of experience. But it also kind of highlights the difference. I mean, essentially, Second Life was a world like our world, but it didn’t correspond to anything in our world. And I think one of the reasons–

SPEAKER: [INAUDIBLE], can you still hear OK?

IRA FLATOW: Mm-hm.

BLAIR MACINTYRE: One of the reasons Second Life has not succeeded in the same way is that, at any given point in time, there’s not a lot of people in any given place. So you probably had this experience. You’d walk through a part of Second Life and it felt like a deserted wasteland of half completed projects.

IRA FLATOW: Yeah.

BLAIR MACINTYRE: Whereas, the real world, you’re walking around and we can map things to it, but there’s still the real world there, which is compelling in and of itself. And the graphics, the games, are just adding to it, hopefully.

IRA FLATOW: All right. Let’s go to the phones. Let’s go to Denver where Chris is waiting for us. Hi, Chris. Welcome to Science Friday.

CHRIS: Hi. Yeah, I just want to start off by saying I really appreciate you guys talking positively about Pokemon Go, one of the first news stories I’ve heard that said good stuff. But secondly, I just want to say, I think there’s great possibility for augmented reality in art projects, especially in sculpture. That they have an environment you can explore, an actual tactile environment. And then you augment that with, you know, a cloud floating by your head or rivers running by your feet. Or you can create multiple different landscapes using augmented reality.

IRA FLATOW: Do you– are you an artist?

CHRIS: Yeah. Yeah, I am. I make sculpture.

IRA FLATOW: So you’d like to be able to– people to get better vision of your art work and learn more about it.

CHRIS: And then other people can upload their own vision of the artwork if they want to.

IRA FLATOW: Great idea. Yan, any reaction?

YAN XU: Yeah, that’s really cool that you’re making, Dan. And I really like your take on this whole perspective, because you were talking about creating content for augmented reality. And it’s not just for artists, but also for everybody, for casual users. And I think with more and more capturing technologies and software pipelines down the road, we’re giving you this, this kind of capability that Dan just talked about.

IRA FLATOW: Could we see a combination of social communities, where people interact with each other, and augmented reality where we’re actually interacting with, really not people, but objects. Blair, what do you think?

BLAIR MACINTYRE: So what are you thinking about when you say interacting with people not objects?

IRA FLATOW: Well, let’s say you’re going to see people in your augmented reality, not just the stations you go to.

BLAIR MACINTYRE: Oh, I see.

IRA FLATOW: And maybe they would be your friends. Maybe you would opt in something like that.

BLAIR MACINTYRE: Yeah, I think there’s– so, the more we know about the world, the more we can augment it. And it’s really just a question of at some point we’re just limited by the kinds of worlds we want to create. One way to think of augmented reality is that we could have any number of virtual reality worlds all overlaid on top of the real world. The world is fixed, right?

And I think there is a lot of interesting possibilities for things like, instead of video calls or audio calls, actually having sort of an avatar of one of your friends appear here, or do what the caller suggested and start creating art projects and educational installations in space.

This– you know, you commented, hopefully that you’d want to opt into it. And this will be a huge issue, right? Because we can’t sort of just show everything everybody creates. Spaces like Times Square or other popular locations would quickly be flooded by tons of stuff that would make it impossible to be there. But the idea of having the ability to share these things would be– is one of the things that lots of researchers and designers think about when they think about augmented reality.

IRA FLATOW: This is Science Friday from PRI, Public Radio International. Talking about the augmented reality with Blair MacIntyre and Yan Xu. Let’s see if we go to the phones. A lot of people want to talk about this. Let’s go to Oakland. And Lock in Oakland. Welcome to Science Friday.

LOCK: Hey, Ira, thanks for taking the call.

IRA FLATOW: Go ahead.

LOCK: Hey, so as a longtime geocacher, I’m used to beating the bushes looking for hidden meanings in the physical world. But what’s really interesting to me is now you have all these people getting out there looking for this context that’s layered in through their phone. What happens when the most important information that people are finding in the physical world is controlled by this program that only people who are engaging to reality that way have access to.

IRA FLATOW: Give me an example of what you’re talking about.

LOCK: Well, you have– you have to go looking around for these Pokemon characters, right? And they’re getting out and they’re engaging with each other around these Pokemon characters that only they can see through their augmented reality screen, you know? There’s going to be layered, more meaning on that, as we start seeing advertising and other ways that people communicating with each other through this augmented reality pretty soon. You can imagine a world where the physical world becomes the pallet upon which the meaning is just painted. And that meaning is encountered through the telephone or through whatever device you’re using for that augmented reality experience.

IRA FLATOW: All right. Good.

LOCK: What does that say about our interaction with the physical world?

IRA FLATOW: Good question. Let me ask both Blair and Yan. Blair, go first.

BLAIR MACINTYRE: So, I think– I mean, it’s a really interesting, important question I think, because if there is only one entity and you know, Niantic or Google controlling what we see in our augmented reality and there’s no way for people to contribute their own meanings aside from being companies and advertisers, that could be a huge problem.

One of the things we’re doing in our research group is trying to create an augmented reality version of a web browser, so that anybody– just like the web– anybody can create augmented reality content, post it where they want, without having to get permission from anybody, and share it. Right? And it’s this idea of democratizing the content and the access to plays.

So I think it could be a huge problem. I also think that, as we move forward, more and more capabilities will be put into the operating systems, the phones, to let more apps that aren’t just controlled by a small number of companies create and share content. I mean, one example is Microsoft. As part of their hololens project, created a set of holographic APIs that are now part of the operating system, so anyone can create a holographic application that shares content in any of the ways that could eventually be to display it.

And I think we’ll see the same thing in Android. We’ll see the same thing in iOS. And we will get to a point where you open up Pokemon Go, and you’re looking at it, and maybe you also have a bunch of other AR applications for geocaching, for tours, for your community. And that content will appear as well, without the approval of, say, Google or Ingress– Niantic.

IRA FLATOW: Mm-hm. Yan Xu, do you have any problems with social interactions here?

YAN XU: Problems? So, I see a lot of potential. So right now people are walking around in Central Park. And if you look at the video, a lot of the time they’re spending a lot of time still with their small private screens. But we also hear a lot of very encouraging anecdotal stories that people start chatting with strangers. You know, Pokemon is a conversation starter.

And people with anxiety, depression, or autism go out of their houses more often because of this game. So it shows us the potential that bringing reality into games can bring people the health benefits, the social benefits. So I believe that with more and more understanding of the real world, combined was our intentional design to bring more of the in depth social interaction like friendship, teamwork, you know, different kinds of capabilities of the users can come together and work together as a team.

IRA FLATOW: All right we’ve run out of– this is exciting to look forward to, to see how this all pans out. Blair MacIntyre of Georgia Tech, Yan Xu, augmented reality researcher formally at Intel. Thank you both for taking time to be with us today.

We’re going to take a break and we’re going to talk about the new Ghostbusters that’s out there, and the real physicists who are here to tell us about the science in it. Stay with us. We’ll be right back after this break.

Copyright © 2016 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of ScienceFriday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies.