SCIENTIFIC AMERICAN FRONTIERS PROGRAM #1510 "Robot Pals"

AIRDATE: April 13, 2005

Hello and welcome to Scientific American Frontiers.I'm Alan Alda.

The problem with most robots is that they tend to be, well, robotic. They know nothing they aren't programmed to know, and can do nothing they aren't programmed to do. But for many applications where robots could be useful, they need to be more like humans, able to respond as a cooperative partner rather than a mindless machine.

In this program, we'll meet some robots that are learning to figure out for themselves what their human companions have in mind.

Ripley is a robot that's scrupulously polite as it tries to understand what you want…

Leonardo turns on the charm as he learns to become an active partner…

While Robonaut figures one day to be a fully-fledged member of an astronaut team, helping with construction tasks in space.

That's all coming up in tonight's episode, Robot Pals.

RIPLEY

ALAN ALDA Hand me the one on my right.

RIPLEY Sorry.

ALAN ALDA Hand me the one on my right.

ALAN ALDA (NARRATION) Believe it or not, this robot is called Ripley. He — somehow I'm assuming it's a he — lives at MIT's famous Media Lab, where right now he's trying to understand me.

ALAN ALDA Let go of it!

KAI-YUH HSIAO Yank. Yank harder. You can do it. There you go. That's it.

ALAN ALDA Hand me the blue one. Hand me the blue one.

ALAN ALDA (NARRATION) Ripley's world is confined to this circular white tabletop. But within that world he has a pretty good notion of what's where — though right now he seems to think the camera is me — and he apparently hates to give things up.

ALAN ALDA Good boy. Good boy. Let go, let go!

RIPLEY Sorry.

ALAN ALDA (NARRATION) Ripley is actually the test bed for an attempt to make robots better understand human speech…

ALAN ALDA You know, this is really a weird sight — a hand with eyes.

ALAN ALDA (NARRATION) …not just by listening to words but by trying to relate those words to what it sees. It's programmed to keep looking around, especially for human faces. But to keep its motors cool it also takes frequent rests. So in its quest to understand its world Ripley has to balance native curiosity with occasional naps, much like a human infant — and that's no coincidence.

DEB ROY Here's the basic idea. When you were an infant, when I was an infant, the way we learned language was by connecting words we heard to what we could see and feel and touch and taste, and those are the earliest bits of meaning that we picked up. And then we built on top of that levels of abstraction, where we could talk about things that were far more abstract. If you look at the way we build machines today that you talk to, we are trying to run before we can walk. We haven't built in those more primitive levels of meaning.

ALAN ALDA (NARRATION) The creature sitting next to Deb — a robot called Toco — was the first to try to connect words with what it could see. Toco was not much more than a camera on an arm…

DEB ROY That's a yellow ball.

ALAN ALDA (NARRATION) But by looking at objects from different angles it was able to learn both what things look like and what they are called.

DEB ROY Look, it's a red cup.

TOCO Look, it's a red cup.

DEB ROY So we used Toco as a way to develop some ideas. But then we went to an infant lab and we actually had mothers and infants play, we didn't tell them why, we jut gave them toys and said, "play with your infant." We put wireless microphones on the moms and recorded spontaneous speech.

NANCY Look what we've got. Peekaboo. He's got a blue hat, doesn't he? And what is it? It's a bear, it's a bear, he's a good hider.

ALAN ALDA (NARRATION) Before an infant can attach words to things she first has to recognize what a word is in the stream of sounds coming from her mother's mouth. Deb wanted Toco to so the same. So…

DEB ROY We took the speech recordings and we brought back the toys. We let Toco look at the toys and listen to the mothers' speech.

ALAN ALDA What happenened? It wet itself!

DEB ROY Toco learned.

ALAN ALDA (NARRATION) Once Toco learned what things are called it took the next step — recognizing objects and putting names to them — in other words it went from learning language to generating it.

TOCO Red ball.

ALAN ALDA (NARRATION) But Toco had reached its intellectual limit. It was time for a robot that could learn not just the names of things, but their context.

DEB ROY One of my favorite examples is, you've got an everyday simple use of words. We're sitting across the table from each other like right now, and there's a cup of coffee between us, and I say to you, "this coffee is cold." So what is the meaning of those words? On the one hand, there's something like, just imagine almost the words are floating between us as we've spoken them, you can think almost of an arrow, pointing down at the thing between us, that you can see and I can see…

ALAN ALDA "This" is already a clue.

DEB ROY So "this" is a clue, "is" is a clue, I'm talking about the coffee now, not yesterday, not tomorrow. The words "coffee" and "cold" have meaning for you because you've tasted the stuff probably and so have I. You know what cold means, not because you've looked up in a dictionary but because you can feel what things are like when they're cold and hot. So there's an experiential background that we're both relying on.

ALAN ALDA (NARRATION) Enter Ripley — sleepily.

ALAN ALDA He looks like a dog resting on the floor.

KAI-YUH HSIAO He's up, he's back.

ALAN ALDA Don't make any sudden movements.

KAI-YUH HSIAO Well he hasn't killed me yet so you figure you've got a little bit of leeway.

ALAN ALDA He's pretty heavy, you could really…

KAI-YUH HSIAO Oh yeah, totally, those are 300lb base motors down there.

ALAN ALDA Holy moly. Oh!

KAI-YUH HSIAO You alright?

ALAN ALDA Yeah I'm OK now. What do you suppose that was?

KAI-YUH HSIAO I reset the system…

ALAN ALDA You reset the system…

KAI-YUH HSIAO And it just kind of dropped.

ALAN ALDA (NARRATION) Once both Ripley and I recovered it was time to do a little work on context.

ALAN ALDA Hand me the heavy one.

ALAN ALDA (NARRATION) Now, Ripley can't tell which object is heavier just by looking at them. So…

KAI-YUH HSIAO So what it's got to do it's got to try it. Just like if you were holding an object like, it's pretty heavy, you know.

ALAN ALDA Yeah he's weighing it a little.

ALAN ALDA (NARRATION) This is just one example of what Deb Roy calls grounding — connecting words to the physical world, something we do naturally but that most speech recognition machines that look up a word's meaning in an internal dictionary are clueless about.

KAI-YUH HSIAO ÔCos he hasn't go two arms. If you had two arms you'd be like, well, this one's heavy, this one's light, you know…

ALAN ALDA (NARRATION) Which is why to really understand what things are, Ripley has to have a body to experience them with.

KAI-YUH HSIAO And with any luck he'll decide which one is the heavier one.

ALAN ALDA Ah, thank you, thank you.

RIPLEY You're welcome.

ALAN ALDA It's funny how easy it is to anthropomorphize it. Sometimes it looks like a puppy, sometimes it looks like a snake. And as these things start to get more sophisticated that's probably going to be a factor in … what is it, I'm talking, excuse me, talking to your maker, do you mind — close your mouth… the way it looks is probably going to have an effect on how I relate to it and how it relates to me.

DEB ROY Absolutely. You can't completely separate function from form. So it's not just here's a piece of software that's running on a computer. The computer has a body, and so just like we're not just brains in a vat, we're brains that are embodied and the particulars of how you look and how you sense and how you interact with the world, they don't just shape the way that you perceive the world, they shape the way that people perceive you.

ALAN ALDA (NARRATION) This is a theme we'll pick up in our next story. Meanwhile…

ALAN ALDA Da da da-da da.

RIPLEY Sorry.

ALAN ALDA You know, I tried that in the army and it didn't work.

RIPLEY Sorry.

ALAN ALDA Watch out, watch out, don't get excited.

RIPLEY Sorry.

ALAN ALDA Some day being a robot will mean never having to say you're sorry.

RIPLEY Sorry.

ALAN ALDA Not yet!

LEONARDO THE LOVABLE

CYNTHIA BREAZEAL So sad when I take away your toy.

ALAN ALDA (NARRATION) Kismet is a robot dedicated to being cute. Built by Cynthia Breazeal, Kismet's deliberately cartoon-like responses were designed to encourage us humans to play with it. A couple of years ago, Cynthia joined forces with a Hollywood special effects wizard to build Kismet's successor.

STAN WINSTON It was probably the most exciting thought concept that had ever come to me in my career. I mean, we have created the most organic looking and acting robots ever in the films that we've done. But they don't have any brains. No offense to the puppeteers. But the robot itself doesn't have any brains. And this was a chance to do something that was groundbreaking and historic.

ALAN ALDA Oh my God. Look. What?

ALAN ALDA (NARRATION) This is Leonardo, who made its first public appearance exclusively for Scientific American Frontiers.

LEONARDO La la la. ALAN ALDA You can make it do that? Or are they making it do that?

STAN WINSTON They're doing it.

ALAN ALDA Yeah, right. Can you do "woo woo woo woo…"?

ALAN ALDA (NARRATION) At this point, Leonardo was still brainless.

ALAN ALDA Wow. Look at that tongue action.

ALAN ALDA (NARRATION) His 61 different moving parts were controlled by puppeteers. The goal was to replace them with a rack of computers.

STAN WINSTON This is the birth of the first truly artificial intelligent character that will have the ability to emote, that can reach every range of expression. That can hear you and respond to you as an artificial intelligence. It's pretty amazing.

CYNTHIA BREAZEAL What we really are exploring here is social connectedness. Social interaction, the human-robot connection.

STAN WINSTON What it is right now is a brand new instrument that has yet to be learned how to play.

ALAN ALDA Right.

STAN WINSTON It's…we've created a grand piano but it's never been… This particular character has never been built before. So all of what's happening with him is a learning process. This is like the first day anybody's seen it. Puppeteers that are dealing with it right nowÑthis is a new character. So it's like "what things make it do this?" So we have to learn how to use this instrument and then once we have learned as human beings how to create an expression, how to create a phonetic, how to make it go "woo", how to make it go "hello", whatever it is…we put it all together…

ALAN ALDA …And the computer will know how to do it?

STAN WINSTON Exactly.

ALAN ALDA (NARRATION) His skin and fur back in Hollywood for further primping, Leonardo — or rather the computers that control him — had to acquire for himself the skills of Stan Winston's puppeteers. Cameras in his eyes help Leonardo pick out human faces and gestures, while microphones in his ears recognize voices and translate the words they are saying. But Leonardo's ears can do more than just listen.

CYNTHIA BREAZEAL You can do the backstroke, the crawl, it can wave, hello. You can see all the technology that's gone into creating a robot that is really optimized for expression. The ability of these robots to interact with people in a way that's natural for them, and that they accept the robot, is going to be really critical. And I think that always Leonardo is pushing the idea of not making it look human, but kind of pet-like but intriguingly anthropomorphic, like a pet but perhaps even smarter than a pet.

ALAN ALDA (NARRATION) With his fur back on — and even with only his mouth able to move — Leonardo was certainly hard to resist.

CYNTHIA BREAZEAL When you were in that little interaction, notice how much you were adapting to it.

ALAN ALDA Right. Yeah.

CYNTHIA BREAZEAL That's what I'm getting at. That's the kernel of the learning interaction I'm trying to capture. And you need something like this to make you act that way.

ALAN ALDA Yeah.

STAN WINSTON And how much fun are you having….? He will learn Ôcause you're so interested in playing with him…

CYNTHIA BREAZEAL That's right.

STAN WINSTON That the machine, the computer, the brain will learn. And because of your interest in his organic reality, you'll make him more real. Imagine having to act with a character that doesn't exist, who isn't thereÑand pretend it's not there. Or to actually have that fantasy character sitting there and acting with you. It allows you a better performance.

ALAN ALDA That's it.

STAN WINSTON Therefore, you as an actor are going to be more believable on screen, we're going to believe the story, we're going to believe the fantasy character and the best performance is going to come out of you.

ALAN ALDA You know, you just made me understand Cynthia's work better because as an actor I understand how much more will come out of me if the other actor in the scene with me is behaving like a real person. Now I understand through that what you mean about how I'll feel when I sit down or stand up with my computer and talk to it and it talks back like a person. And responds to me with facial expressions like a person, I think I'll be in a different relationship to the computer or the robot than I've ever been before.

STAN WINSTON Absolutely.

STAN WINSTON The wonderful thing about this is that we're sort of seeing its birth. The robot is there. The brains are now coming in and the learning process is gonna go year after year. It'll be so interesting I cannot wait for the deadline, which I kind ofÑbecause I have gray hairÑI said "In six months, those brains have got to be connected." Then in the next six months we see how much we can do.

ALAN ALDA You're talking about in the robot…?

STAN WINSTON I'm talking about in the robot, yes. ÔCause mine are gone.

STUDENT Press the red button.

ALAN ALDA (NARRATION) Since we first met Leonardo his brains have indeed been connected, and he has begun to collaborate with his human designers in learning a simple task.

STUDENT Turn all the buttons on.

ALAN ALDA (NARRATION) Leonardo appears obviously baffled by this instruction…

STUDENT I can teach you how to turn the buttons on.

ALAN ALDA (NARRATION) …and it's here that his social skills are serving him well. STUDENT Press the green button.

ALAN ALDA (NARRATION) Leonardo uses his gaze both to show what he's done and to seek approval.

STUDENT Now all the buttons are on.

ALAN ALDA (NARRATION) The relationship between a teacher and learner is at its best a constant exchange of social signals, enabling each to see into the mind of the other.

STUDENT Can you turn all the buttons on?

ALAN ALDA (NARRATION) And that's one of the goals for Leonardo — for him to become a better learner by being able to keep checking in with his human instructor for feedback, and by being able to signal his own impressions of how well he's doing.

ALAN ALDA (NARRATION) Leonardo is gradually emerging as the first ever robot to achieve the goal so beloved in science fiction…

STUDENT Not quite. Push the red button.

ALAN ALDA (NARRATION) Of becoming a friendly, smart and helpful mechanical companion, socially adept and eager to learn. The collaboration between Hollywood chutzpah and MIT know-how is unique — and if Stan Winston and Cynthia Breazeal can pull it off, may indeed result in a robot that's not only useful but that you can care about — and that will at least give the impression of caring about you.

ALAN ALDA Want to go out for a cup of coffee later?

ROBONAUT

ALAN ALDA (NARRATION) We're at the Johnson Space Center in Houston, where astronauts are trained. But Josh Mehling isn't planning to be an astronaut. He's training to become a Robonaut — or rather a Robonaut's puppet master. The Robonaut itself has recently acquired a set of wheels — odd, you might think for a robot designed to help out in space. We'll come back to those wheels in a moment. Right now Josh is gradually bringing the rest of his robot counterpart on line. When Josh moves his head, Robonaut moves his head. When Josh moves his arms, Robonaut follows along. And as Josh flexes his fingers, the exact motion is picked up by sensors in his gloves and sent to the electric motors controlling Robonaut's fingers. In fact, Robonaut's body is as much like a human's — at least from his waist on up — as his chief creator, Rob Ambrose, can make him — with exceptions.

ROB AMBROSE So what you won't find is the appendix — didn't copy that from the human anatomy. There's no reason to copy things if they don't have some function, so we really started with the functions. And the functions are, two dextrous hands with a sweet zone right here where two arms can work together, and it's not an accident that that zone right here where the two arms are so nice, the head sees that zone perfectly.

MYRON DIFTLER OK, Robonaut, reach out and take this handrail please.

ALAN ALDA (NARRATION) Josh shares Robonaut's view of the sweet zone through his goggles.

MYRON DIFTLER That's a good grasp. Thank you.

ALAN ALDA (NARRATION) This tool in NASA-speak is an EVA Tether Hook — what astronauts use in space to attach themselves to the handrails on the outside of their spacecraft.

MYRON DIFTLER Now Josh is going to do a more complicated task. He's going to remove it from the handrail without me helping him. His is simulating things floating in space. What's really nice about this is that Josh only has about 12 or 14 hours experience working with Robonaut, but because it's such a natural system to use, it was designed with body parts similar to what a human has, it's a very intuitive system to work with. All the skills we've developed since we were children transfer very nicely to the robot.

ALAN ALDA (NARRATION) Robonaut started out as one arm and a head, suspended in Rob Ambrose's lab. It was here that the team developed Robonaut's stereo vision and began work on the most complex and sophisticated of its body parts — its hand. Dozens of electric motors control Robonaut's fingers, allowing the thumb and first two fingers to flex and rotate like human fingers. The ring finger and pinkie simply open and close — but because the palm can also flex, the hand can grasp objects securely. The robot Josh is controlling is Robonaut B, the younger brother of the original. And while the gloves covering Robonaut B's hands are much more than cosmetic, they are not as fancy as those now being tested on Robonaut A.

ROB AMBROSE Younger brothers seem always to get the hand-me-down gloves, but we try out the latest here on A. You can tell by the dirtiness of the glove this robot's been put to some serious work. This is the third generation glove that we've developed, it has tactile sensors all along the underside of the fingers and on the palm, kind of like callous points, and out in the palmar fascia. It's able to sense where objects are after it's grasped them and pulled them into its palm. And that's part of the most important signature of what it means to have a good grip on an object.

ALAN ALDA (NARRATION) Getting a good grip matters more than usual when working in space, and here Robonaut has an advantage over his human co-workers by being designed with a single plug-in leg. In this NASA animation, Robonaut is helping build the International Space Station by setting up equipment before a human astronaut arrives — and dutifully putting it away when the astronaut's work is done. Which brings us back to Robonaut B, which has no legs at all, but wheels. And not just any wheels, but those of a self-balancing Segway scooter.

ROB AMBROSE No, you're not going to see a Segway on the Moon, and it makes no sense in zero-G, they'd just be fly wheels causing it to pivot around. But what it allowed us to do very quickly was to learn how to coordinate moving the torso and arms on the torso. When we first built it we weren't even sure if one person would be enough to drive it. We were pleased to find that a single person could both coordinate moving the body and reaching out with the arms.

ALAN ALDA (NARRATION) But the ultimate goal is to eliminate Robonaut's puppet master altogether. The NASA researchers have begun working with the Media Lab team behind Leonardo, to give Robonaut the ability to work autonomously as a member of a team, able to interpret it's fellow astronaut's voice commands and even gestures.

ROB AMBROSE Recently we've been working with a human paired with the machine, where the human is more of the experienced master technician, able to task the junior member of the team, the robot, as an apprentice. "Robonaut, go torque that bolt, that one right there, I'll be back in a few minutes, I'm going over there, call me if you have a problem." We know when we've gotten there when the person standing next to it, working, floating next to it, is able to just pretend it's another person.

ALAN ALDA (NARRATION) A robot astronaut would not only be helpful building structures in space. It — or more likely they— could be sent ahead on missions to the Moon or Mars, perhaps mounted on wheels like a centaur, to prepare the way for humans.

ROB AMBROSE What I would like, if I were going to be sent to Mars, is I would like the robots already to have set up the habitat, have already manufactured the fuel for the return vehicle, have tested the rockets of the return vehicle to make sure that they're working fine, the habitat is already filled with oxygen and at that point I would be really excited leaving Earth to go live in this habitat, knowing everything is working fine. That's what we'd like the robot to do, long before the people get there, is to get everything set up.

ALAN ALDA (NARRATION) Already NASA has an active research program in robotic vehicles, designed to accompany astronauts on lunar or Martian expeditions. But the growing skill of robots — especially robots that can do jobs unsupervised — brings up the obvious question. If robots can do so much to explore other worlds, why send humans at all?

ROB AMBROSE There's a reason humans go to the beach. It's fun, it's exciting, sometimes you discover things at the beach. I could send the robot to the beach and it could do a much better job recording the temperature there than me, do a much better job taking pictures of the beach. But at some point, I want to go to the beach. Because I'm a human.