Visit Your Local PBS Station PBS Home PBS Home Programs A-Z TV Schedules Watch Video Donate Shop PBS Search PBS
Scientific American Frontiers Logo
TV Schedule
Alan Alda
For Educators
Previous Shows
Future Shows
Special Features

The Intimate Machine

Scientific American magazine links
Show credits


SCIENTIFIC AMERICAN FRONTIERS PROGRAM #1303,"The Intimate Machine" will air October 22, 2002.



ALAN ALDA Now I know I'm a little unusual in this, but I've always had a close personal relationship with my personal computer, one that even gets a little emotional at times. I have to admit, though, that the feeling isn't reciprocated - and most people find their personal computers anything but personal. But in this show we're going to explore research aimed at making computers, or machines controlled by computers, as sensitive, gentle and caring as a human companion - or at least a devoted family pet. Which is why I'm here among some of the creatures created at Stan Winston Studio, and famous from the movies they've starred in. Because one of them - Teddy from the Steven Spielberg film A.I. Artificial Intelligence - was the inspiration for an ambitious experiment to give computers not only brains, but a heart. Stay tuned…


ALAN ALDA (NARRATION) It was six years ago that I had my first encounter with a sociable computer - or at least one that tried to be sociable.

ALAN ALDA OK. KRIS THORRISON How does that feel?

ALAN ALDA Now these wires go from the gloves down my back? KRIS THORRISON Down your back and into the computer. And this eyetracker will allow it to see your eye movements.

ALAN ALDA (NARRATION) I was visiting MIT's Media Lab - a visit which in those days often involved getting rigged up in elaborate outfits designed to keep track of what I was doing - and in this case, where I was looking.

ALAN ALDA Oh, my god! Look at my eye on that monitor.

ALAN ALDA (NARRATION) The gloves were to allow the machine to follow my gestures.

ALAN ALDA Oh, it speaks Italian

ALAN ALDA (NARRATION) The idea of all this was to allow me to gesture to my computer.

ALAN ALDA I actually gesture at my computer quite a bit, but the computer doesn't respond. In that regard I can understand how it would be good to have gestures understood by the computer, but why are you engaged in this? Why do you want the computer to read my gestures?

JUSTINE CASSELL Because we do gesture. We gesture at computers, we gesture at each other, like I'm doing right now. We even gesture when we talk on the telephone. So yeah, there you are, you're saying to your computer, not that, the... I told you...

ALAN ALDA But mostly it's like this. I can go like that.

JUSTINE CASSELL And it still doesn't respond.

ALAN ALDA No, it doesn't. It doesn't understand.

JUSTINE CASSELL A slow computer. Well, we'd like to speed it up a little bit.

ALAN ALDA (NARRATION) The Media Lab is famous for its "demos", demonstrations of technology that is still far from perfected, but gives us a glimpse of the possible. The idea here was for a character I could converse with.

GANDALF Hello, I am Gandalf, your guide to the galaxy.

ALAN ALDA Take me to Mars.

GANDALF Why not?

ALAN ALDA I love that eye action.

GANDALF Goodbye.

ALAN ALDA No wait, where are you going? Wait. Are you going to leave me out here in space?

ALAN ALDA (NARRATION) Gandalf had trouble recognizing my speech, let alone my gestures.

ALAN ALDA Tell me about Mars.

GANDALF Mars has two moons.

ALAN ALDA Tell me more.

GANDALF Mars is 6,000 kilometer in diameter.

ALAN ALDA (NARRATION) But he did have a cute personality.

GANDALF Mars is a cool planet.

ALAN ALDA You said it. Uh, what was the point of wearing all this? What did that do? How did that change anything?

JUSTINE CASSELL So we talked earlier about interaction of verbal and nonverbal behavior. You saw an example of that. He knows, Gandalf knows when you're looking at him, and he knows when you're looking at the screen. And when you looked at the screen, he looked at the screen too, just to be friendly. And when you looked at him, he got ready to answer a question, or to interact with you. When you looked away, he took a powder. You can...

ALAN ALDA Did my making gestures change the way... I didn't see a change in the way he answered questions when I used my hands.

JUSTINE CASSELL In the future he'll understand the content of your gestures. For the moment, what he understands is that you're speaking.

ALAN ALDA The ways we've thought of computers up until now has sort of kept computers in the box, and the way you're thinking of computers is giving them a chance to get out of the box by thinking of them more as people. I mean, you're making them listen and feel and hear and see like people, so then they can be all over the place. They don't have to be only accessed by a keyboard and a mouse.

JUSTINE CASSELL Absolutely. That's the point. They should be able to be social actors. Actors in our social world. They're not going to be people because they're not people and we don't want to think that they're people ultimately. I don't.

ALAN ALDA But they might be easier to use if they got information from us the way other people do.

JUSTINE CASSELL Absolutely. That's the point.

ALAN ALDA (NARRATION) Today, Gandalf has retired, to be replaced by REA, whose expertise is much more down to earth.

TIMOTHY BICKMORE I'm starting up her brains. It takes couple of minutes. She's in a deep sleep.

ALAN ALDA This is Gandalf's grand-daughter.

TIMOTHY BICKMORE That's right. We've made a number of advances over the years. She does full gesture and speech generation. So when she has something to say she actually decides whether she should put it into her speech or her hand gestures or both. So she's more natural in that way in interacting with her. She recognizes everything… well, not everything you say but everything you say about real estate.

ALAN ALDA She's got a one-track mind.

TIMOTHY BICKMORE That's right, that's right.

ALAN ALDA Her grandfather needed me to get all duded up in an outfit. Gloves on my hands and a helmet on. She doesn't need that?

TIMOTHY BICKMORE She doesn't need that because she has two cameras up here that recognize the position of your head and hands in space. And from that she can determine where you're standing so she can gaze as you move around and look at you, she can recognize…

REA Hello.

TIMOTHY BICKMORE …if you're gesturing with your hands or not.

ALAN ALDA Hello, hi.

REA How are you doing?

ALAN ALDA Pretty well. Pardon me?

REA Have you been in the Media Lab before?

ALAN ALDA Yes, I have.

REA Cool.


REA They are doing some crazy things in here.

ALAN ALDA (NARRATION) Tim and Justine have found that people tend to trust computer characters more if they chit-chat before getting down to business.

ALAN ALDA What did she say?

ALAN ALDA (NARRATION) But Rea's patience with me quickly runs out.

TIMOTHY BICKMORE She said goodbye.

ALAN ALDA She said goodbye?

JUSTINE CASSELL Yeah, I've never seen her do that before.

ALAN ALDA Look, I swear to God, I really think it's cool.

ALAN ALDA (NARRATION) So Tim resets Rea's brain and we try again. REA Hello


REA How do you like Boston?

ALAN ALDA Oh I like Boston. (Pause.) What? Don't go. No, no. I want to buy a house. What, what?

REA How about this weather?

ALAN ALDA Yeah, it's cool. Uh, do you sell houses or anything? Rea, real estate. Does that strike a chord? This is the best example of soft sell I've ever seen.

REA She figures if you don't want to buy, she doesn't want to push you.

ALAN ALDA I think she just sees in me that I'm not really serious. REA I think so. Look at her, she looks sad.

ALAN ALDA She really does.

REA I have a house.

ALAN ALDA (NARRATION) But finally Rea gets down to business.

REA It has one bedroom, one bathroom, one living room, one kitchen.

ALAN ALDA Could I see the kitchen?

ALAN ALDA (NARRATION) In theory, she's listening to what I'm saying - with both my voice and my hands…

ALAN ALDA Where is the living room?

ALAN ALDA (NARRATION) …and then figuring out how to respond - again with both voice and gestures.

REA The living room is downstairs across from the kitchen.

ALAN ALDA Uh, uh. Could I see the living room?

REA Were you talking to me?

ALAN ALDA Yes, I was.

REA I'm sorry, I did not quite catch that.


ALAN ALDA (NARRATION) Like Gandalf before her, Rea's dependent on an off-the-shelf speech recognition system - which is having trouble understanding me.

ALAN ALDA Show me the living room. Ah, here we go. Ah, nice. Nice TV set there. Tell me about the living room.

REA I'm sorry I did not quite catch that.

ALAN ALDA I bet your granddad was Gandalf.

JUSTINE CASSELL You have this remarkable effect on her and I think it's actually because you have a low pitched voice. I would suggest that you watch Tim interacting with her.

ALAN ALDA Yeah, sure.

REA Let me see what I have available.

ALAN ALDA (NARRATION) Rea has had months of practice recognizing Tim's speech, so their interaction goes much more smoothly.

REA What can I tell you about those rooms?

TIMOTHY BICKMORE Tell me about the house.

JUSTINE CASSELL Once we get recognized speech from this commercial recognition system, we're understanding what the person is saying, and then Rea in real time, as you're having this conversation, is using her intelligence to decide what an appropriate response would be, to formulate that response in English, and also to know how to use her body. And that had never been done before.

TIMOTHY BICKMORE Where is the bedroom?

REA The bedroom is upstairs.

ALAN ALDA (NARRATION) Justine hopes that Rea will evolve into virtual characters that could make all sorts of interactions with the computer easier and more natural. She's already developing on a fitness instructor that could keep an eye on you as you work out at home, a tutor for distance learning…

REA Take care.

ALAN ALDA (NARRATION) And Sam, a virtual playmate for children.

JUSTINE CASSELL When Geneva starts to play, Sam is all about eliciting from her the best of her language, the best of her story-telling, and then giving her what she needs to be encouraged, to be sophisticated in her language use and to be actually focused on the kind of language use that are going to help her as she goes through school.

ALAN ALDA So Sam hears her speaking, not only understands the story part of it but understands what she means and picks out the best parts of her story telling and encourages that? That sounds like an awful lot for Sam to know.

JUSTINE CASSELL Right. And Sam doesn't do any of that.

ALAN ALDA Who does that?

JUSTINE CASSELL Tune back in in a couple of years when we can do that.

ALAN ALDA (NARRATION) What Sam can do already is share the Magic Tower with Geneva…

SAM Cool. OK, my turn…

ALAN ALDA (NARRATION) Even sharing toys across the boundary that divides the real and virtual worlds.

ALAN ALDA How does Sam know what she just did?

JUSTINE CASSELL There are sensors in the bottom of the toys, and sensors in the Magic Tower. And that's what lets Sam know whether one toy is in there or two toys.

SAM Long ago there was a fairy an old man and a wolf…

ALAN ALDA (NARRATION) Sam's background is generated by a video camera in Geneva's world, so even the room is shared between them.

SAM Help, I'm a fairy…

ALAN ALDA (NARRATION) Sam is happy to tell his own stories, or to listen when it's Geneva's turn - prompting her if she pauses.

SAM What happened next?

GENEVA So she took the watering can and she poured it on each of her fishes…

ALAN ALDA She's just totally inspired by this. Would she be that active in her story telling if she wasn't prompted by Sam?

JUSTINE CASSELL That's exactly what we looked at. So we ran a big long study in a local school, and we had four conditions. One kid playing alone without Sam, two kids playing with each other but without Sam, one kid with Sam and two kids with Sam. And what we find is that without Sam they stall, they hum, hey look up at he ceiling and that's it, they don't tell. And one kid with Sam, even with this minimal feedback, and the fact that when she's finished Sam is going to take a turn, seems to inspire children to be really creative.

GENEVA She gave the boy a radish and a lemon…

SAM Then what happened?

GENEVA And picked them up and put them down.

ALAN ALDA It's very interesting. She's moving these figures around the house and every time she puts it in a new room she looks up at Sam and gets a reaction.

JUSTINE CASSELL Right. One of the reasons that I launched this project was in response to what I saw as a series of games that talked at children. Computer games and other games that just bombard the kid with images and noise, and I wanted something that listened and that evoked real physical play in the physical world, that got kids to move around.

ALAN ALDA Geneva, how much fun is Sam as a playmate? Do you like playing with Sam?


ALAN ALDA What do you like about Sam?

GENEVA Because he listens and he says what happens next and stuff.

ALAN ALDA (NARRATION) While Sam encourages his playmates to make up stories, his own are all pre-recorded. But another character in Justine's lab - a lot less wholesome-looking than Sam - is able to put together sentences on the fly.

BEAT If you want it to speak, you just have to type something.

HANNES VILJHALMSSON In a way, Beat actually is sort of like text-to-speech, except that it's not only producing speech, it's producing the gesture that goes along with the speech. So it takes an input, text, and outputs an animation script that a character can execute. BEAT This is both good news and bad news.

ALAN ALDA OK, well that's really interesting, because this is both good news and bad news…

JUSTINE CASSELL Which is by the way exactly what you did…

ALAN ALDA What I did, I know. But how did he…wait a minute, he got it from "both," huh?


ALAN ALDA How did he know that there was this and this?

JUSTINE CASSELL Right. Because one of the things that it knows about the world is that there are sets of things in which one stands in contrast to the other. So we'll let you type in a bunch of other sentences that have contrasts in them, and you'll see that it will always understand, if it has that knowledge in its data base.

BEAT Sometimes I like blue things and sometimes I like red things.

ALAN ALDA Does this and that.

JUSTINE CASSELL Why did he point towards the truck?

ALAN ALDA Oh wait, wait. The truck is blue. But he pointed to it when he said red. Let me see that again.

HANNES VILJHALMSSON He actually likes blue better.

BEAT Sometimes I like blue things and sometimes I like red things.

JUSTINE CASSELL Now, why did he do it differently the second time?

ALAN ALDA Don't ask me. I didn't invent it!

JUSTINE CASSELL No, I'm going to tell you. It's the Socratic method. I'm seeing if it's in you somewhere. But since it isn't I'll tell you. So in fact, not only does this system have a model of the world, it also has something else that we have, which is a model of what we've said previously. And he knows that he already said blue thing. So there's no need to gesture again. You now share it.

ALAN ALDA (NARRATION) Beat also knows a little about where he's standing.

HANNES VILJHALMSSON He actually knows what the scene looks like.. BEAT I was just visiting the Media Lab.

HANNES VILJHALMSSON So he knows where the Media Lab is and is able to point at it.

ALAN ALDA (NARRATION) Let's see how he says a new sentence mentioning the lab.

BEAT Soon the Media Lab will have another building attached to it.

ALAN ALDA Just a little half gesture toward it.

JUSTINE CASSELL Exactly, and that half gesture, he's gesturing when he's saying the new thing, another building attached to it, because that catches your interest and pulls you to the contentful part of the sentence.

ALAN ALDA Let me see what he does if I let him say it again. He probably won't make that same gesture, will he?

BEAT Soon the Media Lab will have another building attached to it.

ALAN ALDA It's almost in response to the unspoken question, what was that you said, because he's just telling you the words.

JUSTINE CASSELL Exactly, yeah.

ALAN ALDA If I do it again will he give me a look like, I told you already.

JUSTINE CASSELL He'll start rolling his eyes.

BEAT I really like the "automated actor."

ALAN ALDA (NARRATION) Giving a cartoon character the ability to interpret typed speech as if he understands it may be useful to animators, but it isn't yet a threat to the acting profession.

BEAT Now this is a lot easier than dealing with a real actor.

JUSTINE CASSELL That's you he's pointing at. And what we did to do this was, he already knows that in his world is this building called the Media Lab, he's got a blue truck, he knows that he's a so-called actor. Hannes just went in and stuck another object into his world, an object called Alan Alda equals actor, and a real one.

ALAN ALDA Let's see…

ALAN ALDA (NARRATION) Well, if I'm going to be just another object in Beat's little world, at least I want better billing than his truck.

BEAT Alan Alda is a real actor, and a great one.

ALAN ALDA I think this is a wonderful program.

JUSTINE CASSELL Yeah, you want to take it home.

ALAN ALDA I think this gives you confidence.


ALAN ALDA (NARRATION) Bruce Blumberg loves dogs - especially his silky terrier Sydney. And dogs have long inspired Bruce's efforts to make computers more sociable.

ALAN ALDA Hello Silas.

ALAN ALDA (NARRATION) Silas, for instance, another of Bruce's dogs, once lived happily in a room at the Media Lab.

ALAN ALDA Where are you going? Hey, come here.

ALAN ALDA (NARRATION) Although only a virtual dog, he'd been trained to respond to the gestures of the people who entered his life.

ALAN ALDA Sit, sit. Oh wow, look at that.

BRUCE BLUMBERG He's like a real dog, that sometimes he'll respond to your gestures and sometimes he won't.

ALAN ALDA Does he beg?

BRUCE BLUMBERG Yeah. Put both your hands over your head.


BRUCE BLUMBERG Now sometimes Silas will do what he wants. So right now I think he's heading off to get a ball, because he figures…

ALAN ALDA Here he is, here he is. Give me the ball, give me the ball. OK, now I'll throw the ball for him.

ALAN ALDA (NARRATION) Several years -- and several thousands of lines of computer code -- later, Silas left his cramped little room and grew up to become Duncan, a sheepdog with an entire Scottish estate to roam around and a flock of virtual and rather flighty sheep to look after.


ALAN ALDA (NARRATION) Bruce is trying to train Duncan much as he would his real dog.

BRUCE BLUMBERG When I tell him "away" for example, he understands that command, but then it's up to his own intelligence in order to figure out how far away from the sheep to go, how to avoid the obstacles that he sees. Very much as if you had a puppy and the puppy comes with some basic abilities and then you're training it to use this action to get particular…

ALAN ALDA Now he's got them in kind of a lump over there.

BRUCE BLUMBERG Yeah, that's good. Maybe I shouldn't do anything. He's doing a great job of herding if I don't do anything. Would you like to try it?

ALAN ALDA Yeah, sure.



ALAN ALDA (NARRATION) Things start off promisingly…

ALAN ALDA Steady. I've lost a sheep here.

ALAN ALDA (NARRATION) But then Duncan and I seem to go our different ways.

ALAN ALDA Bye. Pay attention. Bye. No… He's just separated the whole herd of sheep here.

BRUCE BLUMBERG Someone's separated the whole herd of sheep.

ALAN ALDA Well, if you don't mind I'll blame him. Now this really calls to mind an interesting question. Let's say I got really good at this…


ALAN ALDA Yeah, why? Why have you gone to all this trouble?

BRUCE BLUMBERG Well, the fundamental thing I'm interested in is sort of the nature of intelligence. What is it that sort of is going on in our brains that allows us to have the every day kind of intelligence that gets us through our day? And for me, dogs are a perfect model of that. Because dogs really have dinky little brains -- their brains are the size of lemons if they're a big dog -- and yet they have the kind of common sense that underlies 90% of our behavior.

ALAN ALDA (NARRATION) Bruce uses a clicker to give Duncan a virtual reward when he does the right thing. Now I'm going to see if I can teach him something.


ALAN ALDA (NARRATION) I first wait till he jumps, then say the word jump as I reward him with a click.


ALAN ALDA (NARRATION) After several training runs…


ALAN ALDA (NARRATION) Will Duncan treat the word as a command?

BRUCE BLUMBERG Cool. Cool. Do it again. Do it again.


BRUCE BLUMBERG Awesome. Awesome. You know you are the… this is a pretty fresh demo, meaning you are the first person outside of the group that's actually trained this dog.

ALAN ALDA Oh that must make you feel great that an amateur can do it, that's great. (Wuff.)

ALAN ALDA (NARRATION) As a graduate sheepdog trainer, my next assignment is tougher - raising a wolf pup.

BILL TOMLINSON OK, so now he's woken up.

ALAN ALDA I'm sorry, I'm laughing like a wolf. Do they laugh? Now what?

BILL TOMLINSON Try howling for him.


ALAN ALDA (NARRATION) This cute little black wolf pup is mine to mold - to shape his social interactions with two other pups, one being raised by Bill Tomlinson, the other by Marc Downie. We can move our pups with a mouse, but the only way to control how they behave is through the sounds that wolves themselves make.

ALAN ALDA (Grr. Wuff. Wuff.)

BILL TOMLINSON Barking means play so you're giving him a mixed signal.

ALAN ALDA I'm giving him mixed signals…

BILL TOMLINSON You are. But just a nice grr, grr…

ALAN ALDA OK, I see. You know, you're threatening me when you do that. No seriously, I'm a little frightened of you.

ALAN ALDA (NARRATION) Just as Duncan the sheepdog models animal intelligence, these wolf pups are intended to model social behavior.


ALAN ALDA That's better. (Grr, Grr.) Yeah, that's more like it.

BRUCE BLUMBERG Wolves are really pretty amazing creatures. A lot of a wolf's day is taken up sort of sorting out the social relationship between it and the other members of the pack. And what we're trying to do here is to understand what sort of things might be going on in the brain to be able to do that.

ALAN ALDA (NARRATION) Bruce and Bill got their introduction to real wolf behavior through a visit to Wolf Park in Battle Ground Indiana, where several packs of grey wolves live in a semi-natural environment. Each pack has a strict social hierarchy, with an alpha wolf at the top. The Media Lab researchers are now trying to model the way wolf puppies find their place in that hierarchy.

BILL TOMLINSON If you have your pup coming at me and growling a lot, my pup will learn that you tend to be a more dominant individual and it will in turn form a submissive relationship with your pup.

ALAN ALDA Now, how is your pup going to learn that if you keep giving him sounds to make that go against what he's learning?

BILL TOMLINSON Well, it exhibits that relationship by the emotional style in which it performs the actions that you ask it to take. So if you tell it to growl at somebody who is always dominant to it, it will do so but it will growl kind of like this, knowing that it's a bad move and that no good can come from growling at this individual. Whereas if you try to growl at somebody that you're traditionally dominant to, it will run over and do it very confidently.

ALAN ALDA Well let me see what happens if I don't give up and you don't give up.


ALAN ALDA I mean, I want to see, can get to a kind of stand-off here. Let me see what happens. (Wuff.) There, wake up (Howl). I'm going to go visit you (Howl). There, that's the boy.


ALAN ALDA (Grr.) Get up, get up.

BILL TOMLINSON I was all over you on that one.

ALAN ALDA Alright, well I'll try that again. (Grr.)


ALAN ALDA (Grr.) There he's getting all wimpy again. What is this? Is it me?

BILL TOMLINSON No, I got a head start, because I beat you down the last time and so it's just easier this time. One of the problems too is that human speech gets recognized as whining., which is an interesting thing.

ALAN ALDA Human speech gets recognized as whining? You know this has been my problem in life. I should just keep growling.

BILL TOMLINSON If you want a dominant wolf you have to make sure not to talk at all.

ALAN ALDA I'll talk like this.

BILL TOMLINSON There you go.

ALAN ALDA Hey, turn around. (Grr.) I think I won that one. I think I finally got the idea now. Now I know how to dominate these guys.

ALAN ALDA Uh oh, now that's the adult. Now what if I tried to get the adult to back down?

BILL TOMLINSON Try growling at him.

ALAN ALDA (Grr.) It doesn't do any good. Look at this. I'm totally cowed by this. So what they've learned as puppies, that personality will carry on. For instance, if I had been dominant as a puppy, will I be the dominant adult?

BILL TOMLINSON Exactly. Your adult and my adult will continue to manifest the same relationships that we trained into them.

ALAN ALDA This is fascinating and fun. What do you think it could possibly lead to?

BRUCE BLUMBERG Well, I think there are a couple of different ways that I would love to see this work out. Number one, as a tool ultimately for understanding more about the social behavior of wolves. You know, all the work that we do, I really want it to inform our understanding of the natural world. Number two is that one of the things that computational systems don't do is social behavior. And social behavior is an important aspect of natural systems, so I'm very interested in what would be the benefits of incorporating social behavior into computational systems.

ALAN ALDA (NARRATION) These wolf pups are of course nothing more than complex software programs. But they're programs that exhibit socially appropriate behavior, emotionally charged memories, and the ability to learn from their experiences - all features that would be desirable in the computer now sitting on your desk. Just as long as you don't have to growl at it - and have it growl back.

BILL TOMLINSON People are often worried about making computers that are sentient and social, in that, you know, like the Terminator, they're all going to come back and kill us all and it's going to be this horrible thing. But people all around the world are making entities that are stronger and smarter and more interesting and more capable than us. We have children. And if the process of making computational systems can be more like making children that making bombs, hopefully it will make computational systems that don't turn out to be bombs but turn out to be more like children in the way that they interact with us. And just as I don't grow up to want to kill my parents, hopefully the computational systems will want to have wonderful interactions with us over the course of our lives.

ALAN ALDA So we're going to have to deal with adolescent computers?

BILL TOMLINSON I mean that is one of the challenges. If you have a system that learns, at some point, it hasn't learned yet. And that is a problem.

ALAN ALDA (NARRATION) Once our pups grew up, they sat around and howled.

ALAN ALDA What are they all howling at?

BRUCE BLUMBERG Actually, real wolves do this, it's called a rally howl, particularly in the early morning and late afternoon.

ALAN ALDA (NARRATION) And since my wolf is one of them, it seems appropriate to join in.

ALAN ALDA It's a fade out.


DOOR Welcome to Ted Selker's office. What is your name?

TED SELKER Ted Selker. DOOR Hi Ted Selker. Welcome to your office.

TED SELKER I'll go into my office here. Why don't you try to be Ted Selker?

ALAN ALDA OK, I'll try to be Ted Selker.

ALAN ALDA (NARRATION) Ted Selker invented that little button that acts like a mouse in the middle of the keyboard of IBM laptops. Now he's here at the Media Lab, where his mission is to make computers aware of where we are and what we're doing. The door knows there's someone here - and it's not about to be fooled.

DOOR Hi Ted Selker. Wait, Ted Selker is already in his office.

TED SELKER Let's see. I guess I'll let you in.

DOOR Ted Selker is ready to see you. Please come in.

ALAN ALDA Thanks, my heels are really cool by now. Thank you.

TED SELKER Now, if somebody else comes up, it's going to do something a little different. Why don't you try to go in again?

ALAN ALDA I'll be somebody else. I get to play two parts in this, it's great.

DOOR Welcome to Ted Selker's office. What is your name?


DOOR Sorry, I do not understand you. Please type your name here.

ALAN ALDA (NARRATION) As usual, I'm having trouble getting the voice recognition system to understand me.

DOOR Hi. You don't have an appointment with Ted Selker, and he is busy right now. He has just started a meeting. If this is important, please knock. Otherwise please arrange an appointment or leave a message.

ALAN ALDA This is the knocker?

DOOR I will inform him that you are here. Please wait. Sorry, it seems that Ted Selker is very busy.

ALAN ALDA Does he know it's about insurance?

ALAN ALDA (NARRATION) Well, I'll try my luck at making an appointment.

DOOR No appointment set. Goodbye.

ALAN ALDA Do you know if he has insurance?

ALAN ALDA (NARRATION) Even after you've made it into Ted Selker's lab, your every move is tracked - first by a swarm of virtual bees, then by a clown-face stepping on your foot.


TED SELKER We can measure how heavy a person is and where they're standing and it's neat because it's only about 20 cents a square foot. So we're able to know these things and we use them to tell, you know, social things too…

ALAN ALDA So in other words if the floor knows how much I weigh then when I walk around does it track me by how much I weigh?


ALAN ALDA So it knows where I am all the time?

TED SELKER So if you lose your keys, we can just tell it, hey, where's that guy been?

ALAN ALDA Every floor could be made like this.

TED SELKER That's a good idea.

ALAN ALDA Then you really could track everything that you do around the house.

TED SELKER Well, you could, like, when you pour some milk into a recipe, you could know when you've poured a cup in.

ALAN ALDA By the change in weight?


ALAN ALDA It's that sensitive?


ALAN ALDA (NARRATION) Intelligent doors, intelligent floors…

ALAN ALDA This is an intelligent bed.

TED SELKER That's right. So you're going to put your head underneath this Luxor lamp over here. I bet you've got an eye. There it is.

ALAN ALDA (NARRATION) They are all part of Ted Selker's goal of using simple, cheap sensors everywhere around us to keep track of what we're doing and anticipate our wants.

TED SELKER If you kind of blink rapidly it changed the music. And the reason it changed the music is because it decided you weren't happy. People blink when they're not happy. Now if you just close your eyes for a moment, the system will think you're asleep and now it puts z's across the screen, Now if you open your eyes, the sun rises and you wake up kind of thing. Here we'll try to have you do a couple of other things. Why don't you look at this newspaper for a moment, stare at it.

ALAN ALDA (NARRATION) By picking up my stares, the sensor in the lamp over the bed allows me to browse my email, read a book, even control a video game.

ALAN ALDA I imagine that eventually you're going to find a way to find my eye no matter where I am in the room, I don't have to be lying on a bed.

TED SELKER Yeah, yeah, yeah.

ALAN ALDA (NARRATION) The lab has already developed a simple pair of eyeglasses that are able to tell if I'm staring at something - or even somebody. Selker sees the day when people could exchange much more than a glance when they gaze at each other.

TED SELKER You being interested in me gives me your business card. In a party, the person that's paying attention to someone is giving all these people their cards.

ALAN ALDA I can just imagine that certain people would go home with an awful lot of business cards!

ALAN ALDA (NARRATION) But first the glasses would have to be much less obtrusive.

RESEARCHER It can be reduced to a very very small size. We use it just to demonstrate that we have something, otherwise it could be too small.

TED SELKER There are lots and lots of things that are coming along that are going to make it very easy, almost for nothing, recognize where you are, what you're looking at, how you're looking at it, without even having to wear anything, without you're having to be in any particular place.

JENNIFER HEALY Clench your jaw for a second for me so I can find the muscle.

ALAN ALDA (NARRATION) Six years ago when I was here at the Media Lab, they were making a start not only on letting computers know what you are doing but even how you are feeling.

JENNIFER HEALY Go ahead and clench your jaw. See, so that's how I know it's on a good place. Yeah, great. Good job.

ALAN ALDA Wow, look at that.

ALAN ALDA (NARRATION) As well as the tension in my jaw, Jennifer Healy measured the clamminess of my hands.

JENNIFER HEALY Yes, if you've ever noticed when you're nervous, your palms get all sweaty, and that sweat helps the electrical conductivity.


ALAN ALDA (NARRATION) Another sensor picked up my heart rate. The idea of all this was to let the computer know my emotional state - which unknown to me was about to be aroused.

JENNIFER HEALY Look at that, isn't that cool?

ROZ PICARD He went off the scale there again.

ALAN ALDA (NARRATION) Now as anyone who knows me can tell you, the secret to manipulating my emotions is food.

ALAN ALDA All right, all right. I'm just going to think of a saltine. I bet you a saltine gets me right back... it went down, see?

ROZ PICARD Now you went down. You just dropped all the way down.

ALAN ALDA Saltines are not very exciting to me. Now watch this. Pasta. Oh, red sauce, a little ricotta on the side. Some hot red peppers.

ROZ PICARD The peppers went up.

ALAN ALDA Did they? I could have told you that.

ROZ PICARD I don't know about the ricotta though.

ALAN ALDA (NARRATION) The point of having my computer know my emotions is that it could adjust itself to my mood, running faster when I'm bored, trying another tactic if I'm frustrated. But back then in 1996, reading my emotions was a cumbersome and intrusive business.

ALAN ALDA Does this meant that at the next stage of this, somebody will have to be hooked-up like this in order for the computer to recognize what they're going through?

ROZ PICARD No, eventually we hope that these sensors will disappear into your clothing or into the twiddler itself, into the devices that you're naturally in physical contact with. In fact, people are physically in contact with computers more than with other human beings, frequently, so there's a lot more opportunity than you realize for these sensors to be collecting information.

ROZ PICARD Today you're going to see some of the progress we've made along these lines. One of the things we did, for example, was to take the blood volume pressure sensor that before was this big old clunky thing on your finger, and now you can wear it on your ear. Now I see that you love to wear earrings, and maybe I can just attach this to your ear, OK?

ALAN ALDA And this is shining a light through my ear to see what the blood flow is in there?

ROZ PICARD Yeah, it shines an infrared light, it's called a photoplethysmyograph and it picks up the amount of light going through and the amount changes with your heart rate and with stress information.

ALAN ALDA (NARRATION) And now the clamminess of my hands can be measured with no wires at all - just a glove equipped with a light-emitting diode.

ALAN ALDA But it's not lighting up yet.

ROZ PICARD No it's not lighting up. Well, we can debug this together. First what you can do is try taking some really quick deep breaths. It may take a second to get you to glow. I'm seeing a little teeny bit of glimmer there, very slight.

ALAN ALDA I'm going to pass out from hyperventilation.

ROZ PICARD Don't do that. The other thing you can do is tell me about something significant that's happened to you lately.

ALAN ALDA Something significant?

ROZ PICARD Or something that arouses you.

ALAN ALDA Um, I'm trying to think of something I can tell you on camera.

ROZ PICARD That made him glow, look at that.

ALAN ALDA Look at that!

ROZ PICARD You didn't actually have to say anything.

ALAN ALDA This is the hot pepper phenomenon. Whoa!

ROZ PICARD This signal doesn't tell people what you're feeling. It just tells what are the dimensions of the emotion. So emotions such as anger, fear, excitement, joy, strong interest, those map to a heightened arousal. So this tells us something about the height of that arousal level but it doesn't tell us whether it's a positive or negative response, and that's a key piece of information that we don't seem to get from the surface of the skin right now. There we have to look at the face.

ALAN ALDA Now this takes a picture of my face and it finds my eyes right away.

ALAN ALDA (NARRATION) Right now the system can only tell when I nod, shake my head or blink…

ALAN ALDA So how would this be used?

ROZ PICARD Well, obviously you wouldn't be looking at a picture of your face, you'd be looking at some other task that you're doing. And it's going to be watching you as you engage with it. It will be looking at you to see if you're looking at it or not, because if you're not even looking at the screen, why should it continue to present you with new information. It could be watching to see if you are looking at it intently, if you're raising your eyebrows, or if you're frowning by pushing them together, lowering like you're concentrating. These kinds of changes in the muscles around the eyes are part of the information we want to use to try to determine if you're interested, if you're on the task or not.

ALAN ALDA (NARRATION) Roz Picard's lab has also developed a mouse that can tell how hard I'm gripping it - a measure, perhaps, of how annoyed I am with my computer. These images may also give hints at how I'm feeling about my computer -- or the programs it's running.

ROZ PICARD Right now you're sitting in a special chair. This chair contains a pad at the bottom and the back that contains an array of pressure sensitive sensors and it senses how… the points of pressure and the intensity of those. So there's a rather unusual portrait of you on the screen right now…

ALAN ALDA This is what I'm sitting on?

ROZ PICARD That's what you're sitting on.

ALAN ALDA And this is my back?

ROZ PICARD That's your back. So lean back and we should see more blue lighting up there, you see the weight shifting to the back of the chair. And as you slump down the pattern shifts that way. The dynamics of this pattern we think will give us some clues to things like are you starting to get fidgety, you know, and antsy or are you on the edge of your seat, literally.

ALAN ALDA (NARRATION) Sitting on the edge of his seat is 6-year-old Adam, testing out the sensor chair as it analyzes his shifting weight and decides on his most likely posture. Graduate student Selene Modo has had scores of children sit in the chair while working at a computer. It's one more way Roz Picard's group is trying to make our computers sensitive to our mood and interest…

ADAM Ooh, look, a twenty. Cool, cool.

ALAN ALDA (NARRATION) So that by the time Adam is an adult, his computer will know when it's losing him - and try harder not to put him to sleep.


ALAN ALDA (NARRATION) For almost a decade now we've been following the evolution of an idea that began with "It."

ALAN ALDA I'm sorry. Could you just say that again.

ALAN ALDA (NARRATION) "It" was the creation of MIT researcher and famous robot builder Rodney Brooks. It could react if you got too close with caricatured shock.

ALAN ALDA Heh, heh, it opens its mouth in utter surprise.

RODNEY BROOKS Yeah, and it raises its eyebrows. It's sort of got this reaction of, "Oh, what's happening here, get away from me."

ALAN ALDA (NARRATION) "It" was on of the first attempts to put a human face on a computer with the express purpose of making it easier for us to relate to it.

RODNEY BROOKS It tries to appear to be human, so that we can interact with it in a way that's human.

ALAN ALDA (NARRATION) It soon evolved into a robot head called Kismet, built by a student of Rodney Brooks, Cynthia Breazeal. We first met Kismet three years ago.

ALAN ALDA These robots, like babies, are going to seem especially appealing to us. We spend a lot of time with babies and we can't resist them.

ALAN ALDA (NARRATION) Kismet's expressions -- ranging from boredom… to happiness… to sadness… to interested -- were deliberately over-the-top.

CYNTHIA BREAZEAL It's very much a caricature to make it that much easier for you to read the robot. So that when it looks happy, it's obviously happy; when it looks sad, it's obviously sad. You kind of go, "Oh, I did something to upset it, I should do something to make it happy." So it's really trying to get you involved at this kind of unconscious emotional level.

ALAN ALDA (NARRATION) Cynthia spent long hours in front of Kismet, working on giving it the ability to recognize objects and react with its cartoon-like responses.

CYNTHIA BREAZEAL Ooh, you don't like it when I take away your toy…

ALAN ALDA (NARRATION) And her willingness to invest the effort is itself a demonstration of the goal of the research -- to make people want to help their robots, so that robots can more easily learn the ways of humans.

CYNTHIA BREAZEAL Ooh, you're so sad. So sad when I take away your toy…

ALAN ALDA (NARRATION) Robots learning the ways of humans also happened to be a central theme of a movie then in production. MOTHER This belonged to Martin, my son.

ALAN ALDA (NARRATION) In Steven Spielberg's A.I. Artificial Intelligence, the teddy bear is a robot.


ALAN ALDA (NARRATION) In the movie, the boy is also a robot. But while he's played by an actor, the teddy bear was created at Stan Winston Studio - which also made the dinosaurs for the Jurassic Park movies.

MOTHER His name is Teddy. Teddy, this is David.

DAVID Hello Teddy

TEDDY Hello David.

ALAN ALDA (NARRATION) Teddy was one of the most lifelike puppets ever made in Hollywood…

TEDDY I am not a toy.

ALAN ALDA (NARRATION) While Kismet was certainly the most expressive robot ever built by computer scientists. So when Cynthia Breazeal was hired as a consultant to A.I. and met Stan Winston, they hatched a plan - to combine their skills to create the most lifelike and expressive robot ever built.

STAN WINSTON It was probably the most exciting thought concept that had ever come to me in my career. I mean, we have created the most organic looking and acting robots ever in the films that we've done. But they don't have any brains. No offense to the puppeteers. But the robot itself doesn't have any brains. And this was a chance to do something that was groundbreaking and historic.

ALAN ALDA Oh my God. Look. What?

ALAN ALDA (NARRATION) Meet Leonardo, making its first public appearance, exclusively for Scientific American Frontiers.

LEONARDO La la la.

ALAN ALDA You can make it do that? Or are they making it do that?

STAN WINSTON They're doing it.

ALAN ALDA Yeah, right. Can you do "woo woo woo woo…"?

ALAN ALDA (NARRATION) Today, Leonardo is still brainless.

ALAN ALDA Wow. Look at that tongue action.

ALAN ALDA (NARRATION) His 61 different moving parts are controlled by puppeteers. The goal is to replace them with a rack of computers.

STAN WINSTON This is the birth of the first truly artificial intelligent character that will have the ability to emote, that can reach every range of expression. That can hear you and respond to you as an artificial intelligence. It's pretty amazing.

CYNTHIA BREAZEAL What we really are exploring here is social connectedness. Social interaction, the human-robot connection.

STAN WINSTON What it is right now is a brand new instrument that has yet to be learned how to play.


STAN WINSTON It's…we've created a grand piano but it's never been… This particular character has never been built before. So all of what's happening with him is a learning process. This is like the first day anybody's seen it. Puppeteers that are dealing with it right now-this is a new character. So it's like "what things make it do this?" So we have to learn how to use this instrument and then once we have learned as human beings how to create an expression, how to create a phonetic, how to make it go "woo", how to make it go "hello", whatever it is…we put it all together…

ALAN ALDA …And the computer will know how to do it?


ALAN ALDA (NARRATION) It's a few weeks later. His skin and fur back in Hollywood for further primping, Leonardo sits patiently in the Media Lab, waiting for Cynthia Breazeal to plug him into his control system. It's the start of another day of Cynthia's translating the skills of Stan Winston's puppeteers into computer code. Of the 61 different moveable parts in Leonardo's body, twelve are in his arms alone.

CYNTHIA BREAZEAL It gives us a very intuitive physical sense of, you know, as I'm moving Leonardo's arms, to what kind of motion the robot can do well to what's kind of pushing its limits and so forth. So I think it will give us additional insights as we write the code to make sure that Leonardo moves in a way that doesn't damage its motors, it doesn't overheat them too fast.

ALAN ALDA (NARRATION) But making Leonardo move like a living creature rather than a robot isn't enough. Leonardo isn't just a robot. It's a character.

CYNTHIA BREAZEAL Character is expressed through the entire body. Often characters have their own little idiosyncrasies, so probably elements of those will be put into Leonardo. We don't know what they are yet, that's kind of Stan Winston's domain I think. But anything regarding, I mean what is the character of Leonardo is really going to be a joint effort between us as well as Stan Winston Studios to try to define, what does this robot sound like when it talks? How does it express? How does it react to things? What is the character of Leonardo, how do you encode that? And how do you encode it in a way that the robot can learn and extend its repertoire over time?

ALAN ALDA (NARRATION) For Leonardo to know what's going on around it, cameras in its eyes will be able to recognize faces and facial expressions, while microphones in its ears will recognize individual voices and the sounds they are making. And its body will be covered with touch sensors.

CYNTHIA BREAZEAL You can imagine people are going to be petting Leonardo, it's a very lovable little guy, and it should be able to know that and respond to that, know when it's being touched in a way that it likes, that's good for it. And the way when people are touching it that's not good for it, it should recoil, I don't like the way you're touching me.

ALAN ALDA (NARRATION) One of the robot's most expressive features are its ears - which can also perform a few tricks not often seen in nature.

CYNTHIA BREAZEAL You can do the backstroke, the crawl, it can wave, hello. You can see all the technology that's gone into creating a robot that is really optimized for expression. The ability of these robots to interact with people in a way that's natural for them, and that they accept the robot, is going to be really critical. And I think that always Leonardo is pushing the idea of not making it look human, but kind of pet-like but intriguingly anthropomorphic, like a pet but perhaps even smarter than a pet.

ALAN ALDA (NARRATION) With his fur back on - and even with only his mouth able to move - Leonardo is certainly hard to resist.

CYNTHIA BREAZEAL When you were in that little interaction, notice how much you were adapting to it.

ALAN ALDA Right. Yeah.

CYNTHIA BREAZEAL That's what I'm getting at. That's the kernel of the learning interaction I'm trying to capture. And you need something like this to make you act that way.


STAN WINSTON And how much fun are you having….? He will learn 'cause you're so interested in playing with him…


STAN WINSTON That the machine, the computer, the brain will learn. And because of your interest in his organic reality, you'll make him more real. Imagine having to act with a character that doesn't exist, who isn't there-and pretend it's not there. Or to actually have that fantasy character sitting there and acting with you. It allows you a better performance.

ALAN ALDA That's it.

STAN WINSTON Therefore, you as an actor are going to be more believable on screen, we're going to believe the story, we're going to believe the fantasy character and the best performance is going to come out of you.

ALAN ALDA You know, you just made me understand Cynthia's work better because as an actor I understand how much more will come out of me if the other actor in the scene with me is behaving like a real person. Now I understand through that what you mean about how I'll feel when I sit down or stand up with my computer and talk to it and it talks back like a person. And responds to me with facial expressions like a person, I think I'll be in a different relationship to the computer or the robot than I've ever been before.

STAN WINSTON Absolutely.

STAN WINSTON The wonderful thing about this is that we're sort of seeing its birth. The robot is there. The brains are now coming in and the learning process is gonna go year after year. It'll be so interesting I cannot wait for the deadline, which I kind of-because I have gray hair-I said "In six months, those brains have got to be connected." Then in the next six months we see how much we can do.

ALAN ALDA You're talking about in the robot…?

STAN WINSTON I'm talking about in the robot, yes. 'Cause mine are gone.

ALAN ALDA (NARRATION) The collaboration between Hollywood chutzpah and MIT know-how is unique - and if Stan and Cynthia can pull it off, may indeed result in a computer - or a computer-controlled creature - you can not only talk to but care about - and that will at least give the impression of caring about you.

STAN WINSTON Leonardo will actually be able to see, be able to hear, and be able to touch and feel. It will respond to you. It's amazing. And I'm going to take credit for all of it.

CYNTHIA BREAZEAL Oh no you're not. Oh no you're not.

ALAN ALDA Want to go out for a cup of coffee later?


      return to show page
© 1990-2002 The Chedd-Angier Production Company, Inc. All rights reserved.



Friendly CharactersAlpha WolfGetting to Know UsLeonardo Teaching guide Science hotline video trailer Resources Contact Search Homepage