"INVENTING THE FUTURE"
SHOW 701

Episode Open
Virtually Real
Body Talk
Smart Car
Private Eyes
Brain Music

EPISODE OPEN

ALAN ALDA (Narration) This is MIT's famous Media Lab. This is the lab's pet dog, Silas. And this is me. Right now we're all virtual. But the real world here at the Media Lab is even more extraordinary. Silas does tricks, while Gandalf takes me to the stars. I get my own cooking show... drive a smart car. And, for a brief, embarrassing moment, become a cyborg. This is Alan Alda. Join me for "Inventing the Future", on Scientific American Frontiers.

back to top

VIRTUALLY REAL

ALAN ALDA (Narration) My first glimpse of MIT's Media Lab came - appropriately enough - on the screen of my trusty laptop, via the Internet. The Lab's reputation is that life there runs on fast forward... That here today is a preview of tomorrow. And the word is that the tomorrow now being hatched with a stark exterior is a startling one - in which computers, as we've come to love them or hate them, disappear. Now as a computer buff myself, this was something I had to see. If it's good-bye laptop, it's hello... what? Well, here I was, moments after arriving, balancing on my empty hand a virtual version of the lab complete with its own welcoming host.

ALAN ALDA Sandy, can you wave your arms about a little?

SANDY PENTLAND OK?

ALAN ALDA Yeah, yeah, there you are.

SANDY PENTLAND Hi.

ALAN ALDA How are you?

ALAN ALDA (Narration) The man behind the curtain was Sandy Pentland.

SANDY PENTLAND Virtual me in the virtual lab. Virtually on your hands.

ALAN ALDA (Narration) His answer to my very first question turned out to capture the lab's trademark research philosophy.

ALAN ALDA Sandy?

SANDY PENTLAND Yes?

ALAN ALDA Why are we doing this?

SANDY PENTLAND Well, we don't know. We figured that...

ALAN ALDA You don't know? Ha.. ha. No, I mean, what is this sort of a glimpse of here, what is... I mean, it's an amazing thing, but how will it help us get to the future?

SANDY PENTLAND What we're showing here is that we can take real things and virtual things and put them all together. You can think of this sort of the next generation of virtual teleconferencing. We could have a little crowd of people and we could all explore the building together. You could maybe pick parts of the building up and move it, and so forth.

ALAN ALDA Oh, in other words, I could be here, and you could be in Toledo... Spain, and we could both go through this virtual building together.

SANDY PENTLAND Yeah, right. That's right. And we could grab things and move them. There could be more than one person that's virtual like me. We could have people not just from Spain, but from England and Japan and we could all get together with our virtual representations and talk about things and design things and...

ALAN ALDA I mean, I could be working at my computer and then all these little guys could show up and I could tell them to go do things.

SANDY PENTLAND That's right. That's the idea... is to be able to combine reality with virtual reality, to augment reality.

ALAN ALDA Well, that's attractively weird.

ALAN ALDA (Narration) What's even weirder is my playing Doom. But in this game neither I nor my plastic gun are in any way connected to the computer. The computer knows what I'm doing because it's watching me.

ALAN ALDA Got him, captain. You can't, surely you can't have been working all these years to make this game safe for democracy. What's this going to lead to?

SANDY PENTLAND No, no. What really happened is, we were working on being able to understand human emotion and recognize gestures and things, and I came in one day and someone, one of my students, had hooked it up to Doom.

ALAN ALDA (Narration) Bruce Blumberg is another of the team working on an environment that can figure out what someone in it is doing - a Smart Room.

ALAN ALDA And it seems to know I'm here, because that blue blob is me and you, right?

BRUCE BLUMBERG Right. And yeah, if you look at the top of the screen, or what we call the magic mirror, you'll see a video camera. And the video camera, when the system first starts up, records a whole bunch of pictures of the background, and then, when you walk into the world, it's looking for differences. So it's comparing the picture with you in the world with the picture without and saying, wherever it's different, it must be a person.

ALAN ALDA Now what are those little squares, the green square, the red square?

BRUCE BLUMBERG Right. Well, what we're doing is, once we have the blob - it doesn't seem quite right to call you a blob - but once we have the blob, we do some very simple gesture recognition. So part of that is figuring out where the hands are. So if you hold out the hand, the computer program has some idea of how long your arm should be, and the green cube there corresponds to where it thinks your hands are. And similarly, if you knelt down, it will think that you're sitting. And that's basically done on height.

ALAN ALDA What do you use the blob for? What good is the blob?

BRUCE BLUMBERG Well, the blob in itself is, only so interesting. But the purpose behind it is that we can use this as a front end, say, to a virtual reality system where we have creatures.

ALAN ALDA Could I see the creature?

BRUCE BLUMBERG Of course.

ALAN ALDA (Narration) And there, in the Magic Mirror, is Bruce Blumberg's animated creation, Silas.

ALAN ALDA Hello, Silas.

ALAN ALDA (Narration) Now Silas of course is simply a computer program that's getting data about what I'm doing via my blob. But share a Smart Room with him for a while and that's easy to forget.

ALAN ALDA Hey, Silas... Silas, come here. Sit. Sit. Oh, wow, look at that.

BRUCE BLUMBERG He's like a real dog, but sometimes he'll respond to your gestures and sometimes not.

ALAN ALDA And will he beg?

BRUCE BLUMBERG Yeah. Put your both hands over your head.

ALAN ALDA Come on.

BRUCE BLUMBERG Wow, you're a pro. Now, sometimes Silas will do what he wants. So right now I think he's heading off to get a ball, because he figures...

ALAN ALDA Here he is. Give me the ball. OK, now I'll throw the ball for him.

BRUCE BLUMBERG There you go.

ALAN ALDA He always goes after the ball?

BRUCE BLUMBERG He goes after the ball unless he really had to get a drink of water or something else that was more important.

ALAN ALDA How does he know he wants a drink of water?

BRUCE BLUMBERG He has a number of built in motivations. And these motivations go up over time and so he's balancing sort of these internal goals with what's happening out in the world.

ALAN ALDA After he has a certain number of drinks of water, does he head for the fire hydrant?

BRUCE BLUMBERG Yeah. And you see, that's what makes animals, and this kind of guy different than regular robots, is that animals are constantly juggling a zillion goals, whereas robots typically only focus on one thing at a time. And part of the serious science here is, well, how do you build systems that can deal with multiple goals. With the same sort of common sense that dogs have.

ALAN ALDA Ha, ha. What's this thing on the corner there?

BRUCE BLUMBERG He's a hamster. He's another little creature too. You may be able to pick him up. It may be a little problematic.

ALAN ALDA I think I've got him.

BRUCE BLUMBERG Ah, no.

ALAN ALDA I'm looking over there to see, and it's not there. I'm starting to believe this.

BRUCE BLUMBERG But I think that within a couple of years, we'll be able to...

ALAN ALDA (Narration) But no sooner did we decide to forget about the hamster....

ALAN ALDA You know, the hamster has just suddenly flown off the table.

BRUCE BLUMBERG Well, let's take advantage of this. Hold out your, point with your right hand. OK. Now Silas is going to respond to your gestures, but he doesn't really like the hamster, because everyone likes the hamster, and that takes away attention...

ALAN ALDA He's jealous.

BRUCE BLUMBERG He's jealous. And eventually it gets too much and he has to... now why don't you just point, like that. And maybe Silas...

ALAN ALDA He loves to walk through my legs. What's...

BRUCE BLUMBERG See, Silas is sad now, so he's going away because you punished him.

ALAN ALDA Oh, I see. All right, come back. Come over here, Silas. Let's make up. No, Silas... Silas. He really has a mind of his own. Come here. Come over here. Come here.

ALAN ALDA (Narration) So here I am, playing virtual ball with a virtual dog, and there isn't a computer in sight. But computers, of course are running everything - figuring out my intentions as well as endowing Silas with his doggy behavior.

BRUCE BLUMBERG One of the things that you didn't see there was training Silas, because, after all, animals learn, and so I'm very interested in how can you have these virtual creatures learn like real animals?

ALAN ALDA That's the hamster?

BRUCE BLUMBERG Yup. This circular area here, this gray area, represents an electrified mat. What happens is, a sequence of lights will come on and then when the red light comes on, the hamster gets shocked.

ALAN ALDA Once he gets shocked, what makes him want to come back on this gray thing at all?

BRUCE BLUMBERG Well, he's not that smart. But also, it's... these things represent food sources. So really what he's doing is, he's foraging, and in between foraging, he's periodically having to deal with the fact that he's getting shocked. So you'll notice now, the red light never comes on, because he's always fleeing before he can get shocked.

ALAN ALDA You mean to say that we're going to have like virtual dogs roaming around our houses?

BRUCE BLUMBERG Well, I think it's a possibility. I mean, I have a dog at home, and I'm not sure I'd trade, well, maybe, she's an old dog, but um, I'm not sure I'm willing to trade a real dog for a virtual dog. But you could imagine sort of a St. George and the Dragon kind of environment where a kid could have a pet dragon at home.

ALAN ALDA Let's say I'm not interested in a virtual pet. But there's other things you can do with this knowledge, right? Like what?

BRUCE BLUMBERG Well, I think that fundamentally the problem I'm trying to solve is, how do you build intelligent things that do the right thing for the right amount of time over extended periods of time and adapt?

ALAN ALDA (Narration) The latest challenge to Silas' intelligence is to get him to figure out his own way through a computer-game maze. He's getting very good at it. But it may turn out that Silas - or intelligent computer creations like him - will have their greatest impact in helping the rest of us find our way through real life mazes. Media Lab professor Patti Maes.

ALAN ALDA How is he going to be useful to you in your work?

PATTI MAES Well, I admit that Silas is, the only sort of reason for his existence is entertainment at the moment, and science of course, but we're actually making similar types of software that can help people with real hard problems.

ALAN ALDA Like what?

PATTI MAES Well, one such problem is finding things that you may be interested in, there's actually more and more information available every day, there's more and more products available on the markets, and it's actually really hard for a consumer to find the things that they may like in this mass of information, mass of products out there, so we've been building software agents that sort of are on the lookout for you, trying to find things that you may be interested in. Actually, there's some things that we can show you right now and that the viewers can try out themselves on the Web.

ALAN ALDA Oh, that's now. It's on the Web now? Oh, let's see.

ALAN ALDA (Narration) As someone who's spent hours searching for information on the Web, having a notional Silas fetch things for me is appealing. Patti Maes' intelligent agent is called Firefly, designed to help people find music or movies they might like.

ALAN ALDA Now, if I click on this I can rate it.

PATTI MAES Yup, you can always rate it.

ALAN ALDA And that's my rating.

PATTI MAES Yes. Lots and lots of people tell the system what they like and dislike on the order of 10,000 people, and so everybody has their agent that that particular person likes and dislikes. And all these agents can get together and compare their users with one another, so my agent will talk to your agent and try to find out whether we are similar, whether we have similar tastes.

ALAN ALDA So if we have a basically similar taste, as the agents match our tastes, then it'll tell me about movies you've seen and liked that I might not have thought of.

PATTI MAES Yes, exactly. Click on one of the movies and then ask the system to predict how much you would rate that movie.

ALAN ALDA (Narration) Knowing this wasn't a good idea, I was talked into checking on movies I've appeared in. While Firefly in its current form only works to help find music or movies that people with similar tastes to mine have enjoyed, it's easy to see how an intelligent agent like this could search out restaurants or stocks or vacation resorts - even politicians.

PATTI MAES What people really need help with is finding other people, finding other people that share some of their interests or that have different interests, or that are an expert in a particular area, etc. And so that's actually one of the other functions that this system of agents provides. Agents can do matchmaking, not so much romantic matchmaking, although you could argue that if you go and check out the people that like the same movies and the same music or whatever, books that you like that maybe they're...

ALAN ALDA Yeah, well are you working on a matchmaking...

PATTI MAES You'll at least have a good start.

ALAN ALDA Are you working on literally matchmaking?

PATTI MAES It's not, sort of, we didn't build this to do romantic matchmaking, but as it turns out I do know a lot of couples that have found each other...

ALAN ALDA That met this way, really?

PATTI MAES Yeah.

back to top

BODY TALK

ALAN ALDA (Narration) Now you might well be wondering - as I was by now - what was all that about computers as we know them disappearing?

ALAN ALDA OK.

KRIS THORRISON How does that feel?

ALAN ALDA Now these wires go from the gloves down my back?

KRIS THORRISON Down your back and into the computer. And this eyetracker will allow it to see your eye movements.

ALAN ALDA (Narration) If this is what's going to replace them, I'm not sure I'm ready to give up my mouse and keyboard just yet. Graduate student Kris Thorrisson is helping me into an outfit that will keep track of where I'm looking.

ALAN ALDA Oh, my god! Look at my eye on that monitor.

ALAN ALDA (Narration) The gloves, meanwhile, are to allow the machine to follow my gestures.

ALAN ALDA Oh, it speaks Italian

ALAN ALDA (Narration) The idea of all this is to allow me to gesture to my computer.

ALAN ALDA I actually gesture at my computer quite a bit, but the computer doesn't respond. In that regard I can understand how it would be good to have gestures understood by the computer, but why are you engaged in this? Why do you want the computer to read my gestures?

JUSTINE CASSELL Because we do gesture. We gesture at computers, we gesture at each other, like I'm doing right now. We even gesture when we talk on the telephone. So yeah, there you are, you're saying to your computer, not that, the... I told you...

ALAN ALDA But mostly it's like this. I can go like that.

JUSTINE CASSELL And it still doesn't respond.

ALAN ALDA No, it doesn't. It doesn't understand.

JUSTINE CASSELL A slow computer. Well, we'd like to speed it up a little bit. So the idea is that...

ALAN ALDA (Narration) The Media Lab is famous for its "demos", demonstrations of technology that is still far from perfected, but gives us a glimpse of the possible. The idea here is for a character I can converse with.

GANDALF Hello, I am Gandalf, your guide to the galaxy.

ALAN ALDA Take me to Mars.

GANDALF Why not?

ALAN ALDA (Narration) Gandalf has a simple off-the-shelf speech recognition program to know what I'm saying. The novelty here is that he's supposed to respond to my gaze and my gestures as well as my voice.

ALAN ALDA Here's Mars, right? It's red. Tell me about Mars.

GANDALF Mars has two moons.

ALAN ALDA Tell me more.

GANDALF Mars is 6,000 kilometer in diameter.

ALAN ALDA Tell me more about Mars.

ALAN ALDA (Narration) He's also got something of a personality of his own.

GANDALF It's a cool planet.

ALAN ALDA You said it. Tell me about Saturn.

GANDALF Saturn has three rings.

ALAN ALDA Tell me more.

GANDALF The rings are made from rocks.

ALAN ALDA I love how it goes like this. It goes, I say, tell me more, he goes, the rings are made of rocks, like, what more do you want to know? Hey, I already told you about Saturn. And this tapping of the fingers, what is that, like...

KRIS THORRISON It's just to show you that it's alive.

ALAN ALDA That it's alive, yeah. What was the point of wearing all this? What did that do? How did that change anything?

JUSTINE CASSELL So we talked earlier about interaction of verbal and nonverbal behavior. You saw an example of that. He knows, Gandalf knows when you're looking at him, and he knows when you're looking at the screen. And when you looked at the screen, he looked at the screen too, just to be friendly. And when you looked at him, he got ready to answer a question, or to interact with you. When you looked away, he took a powder. You can...

ALAN ALDA Did my making gestures change the way... I didn't see a change in the way he answered questions when I used my hands.

JUSTINE CASSELL In the future he'll understand the content of your gestures. For the moment, what he understands is that you're speaking.

ALAN ALDA The ways we've thought of computers up until now has sort of kept computers in the box, and the way you're thinking of computers is giving them a chance to get out of the box by thinking of them more as people. I mean, you're making them listen and feel and hear and see like people, so then they can be all over the place. They don't have to be only accessed by a keyboard and a mouse.

JUSTINE CASSELL Absolutely. That's the point. They should be able to be social actors. Actors in our social world. They're not going to be people because they're not people and we don't want to think that they're people ultimately. I don't.

ALAN ALDA But they might be easier to use if they got information from us the way other people do.

JUSTINE CASSELL Absolutely. That's the point.

ALAN ALDA (Narration) OK, we have smart rooms that can track what I'm doing, intelligent agents that can act on my behalf, characters I can talk to like another person. And this guy - who here is simply mimicking my movements, but is really part of a demo of a smart desk, which tracks my gestures without wires - as Gandalf will once he knows what gestures mean.

ALAN ALDA What a stupid grin.

ALAN ALDA (Narration) I was beginning to see where my computer is going - off my desk and into the rest of my life.

SANDY PENTLAND In one sense, what we're trying to do here is build a virtual butler, right, somebody who'd be around with a towel and say, oh, master is getting up now, I'd better start the coffee. So if we can duplicate those sort of abilities, I think that'll be fun. It'll be nice, it'll be convenient.

ALAN ALDA You think that'll be like an early step, an actual thing that'll walk around and do things for us?

SANDY PENTLAND Oh, I don't think it'll walk around. I think what you'll have is little sensors in the corners of the rooms that know when you're asleep and know when you're getting ready to get out of bed.

ALAN ALDA And then parts of the house will just do it?

SANDY PENTLAND That's right. And you know another application for something like this is if you have young children it might warn you when the kids are playing near the stairs so you can go and shush them away from the stairs. So we're building eyes and ears to be helpful.

CLAUDIO PINHANEZ Welcome to the Media Lab cooking show.

ALAN ALDA (Narration) You never know what's around the corner in the Media Lab, and before I know it I'm guesting on a cooking show.

ALAN ALDA So, we're going to do artichokes on top of pasta, right? The artichoke is a sauce. The artichoke's been soaking in...

ALAN ALDA (Narration) It's actually another of those Media Lab demos, this time of what one day might be a TV studio where the cameras themselves get the shots, with no one operating the controls.

ALAN ALDA See, it's starting to get yellow...

ALAN ALDA (Narration) I'd been asked by Claudio Pinhanez to prepare one of my favorite recipes, complete with the sort of details that require tight close-ups.

ALAN ALDA The choke is that fuzzy stuff that's in the... By the way, please don't try to follow the recipe. It's one of my favorites, but since we didn't want to shut down the Media Lab by setting off the sprinkler system, we weren't actually able to cook anything. Then you take this and pour it over... that's the sauce. You pour it over the pasta. This doesn't look it, but it's delicious. It's really much better than this when it's chopped up and it looks nice.

ALAN ALDA (Narration) Supervising the project is lab professor Aaron Bobick.

ALAN ALDA Why were we doing this? Why was this, how will this turn into something in the real world?

AARON BOBICK Our goal here is to build what we call Smart Cams, which is basically a camerman-less camera. Now a cooking show is particularly nice for us because we have a basic notion of a script - that is, the sequence of steps, a recipe, in fact, of what the actions are, so we can make use of knowing something about those actions to actually make the computer able to do the vision that it needs to finally give a close up of those delicious looking...

ALAN ALDA (Narration) Claudio has already starred in a solo demonstration of the Smart Cam system. As with all the Media Lab demos, it's meant to show how something can be done, more than how it will be. But the key point is that the system finds each shot from a simple written description, rather than having a camera operator interpreting the director's instructions.

AARON BOBICK Well, what you have here is a switcher, the kind of thing a director would normally see where he's got the input from the three different cameras and the director gets to decide which of these cameras he's going to send to the viewers. But this is a Smart Cam system where the director can call whatever shot he wants and the cameras respond.

ALAN ALDA So the director can type in directions. Like I could say on this camera, give me a close up of the cook? Can I say that?

AARON BOBICK Absolutely.

ALAN ALDA (Narration) I'm proud to say that my guest appearance with Claudio - if it produced nothing edible - will be useful in testing the system.

AARON BOBICK A question that's been asked to us is how specific was it to him and the way he did things, and it'll be very nice for us to take a different recipe to show how much or how little work we need to do to make this exact system deal with the ultimate Alan Alda cooking show.

ALAN ALDA OK, now you cut it in half...

ALAN ALDA (Narration) What's remarkable about the system is that it's able to recognize actions like mixing or cutting, as well as find objects like people or bowls. And as with everything at the Media Lab, who knows what it could lead to - perhaps even home videos that people will actually want to watch! Meanwhile, it's one more demonstration of how computers can tell what we are doing or intending without our spelling it out.

ALAN ALDA Then you take this and pour it over...

ALAN ALDA (Narration) Of course, computers still have no sense of taste.

ALAN ALDA This doesn't look it, but it's delicious.

back to top

SMART CARS

ALAN ALDA (Narration) After smart rooms, smart desks and smart cameras - you guessed it, smart cars.

ALAN ALDA I seem to be doing OK now, why would I need a smarter car?

SANDY PENTLAND Well, you know, 40,000 to 50,000 people a year die in this country alone in accidents. Most of those accidents are bad judgment, or people didn't see things.

ALAN ALDA Where'd that guy come from?

SANDY PENTLAND Yeah, exactly.

ALAN ALDA Wait a minute, I mean it!

SANDY PENTLAND This is Boston, you have to remember that. They come at you from all angles...

ALAN ALDA How would a Smart Car have helped me there?

SANDY PENTLAND Well, it probably would have warned you that someone was coming up through your blind spot.

ALAN ALDA (Narration) Car makers are already working on inexpensive radar systems that will warn you if a vehicle is lurking in your blind spot.

SANDY PENTLAND The trouble is of course is if it's beeping all the time, you're going to rip it out first chance you get, and then of course it's not so smart.

ALAN ALDA So I don't want something that's always telling me about the blind spot. I want it to know when I need to know it.

SANDY PENTLAND That's right.

ALAN ALDA It's got to be pretty smart to know when I need to know it.

SANDY PENTLAND Yeah, it's got to know what you're up to and what it is you're trying to do and only when you're going to do something that's going to get you in trouble does it say, oh, watch out. OK, well, this is the half-car we that we use to develop our Smart Cars. It's a little bit safer than driving around on Memorial Drive, so why don't we get in and we'll start it up and go for a drive.

ALAN ALDA (Narration) Media Lab professor Sandy Pentland is collaborating with Nissan on a project to make cars smarter. This driving simulator allows people's driving habits to be closely monitored by computers checking every movement of the steering wheel or touch of the pedals.

SANDY PENTLAND Don't hit anything.

ALAN ALDA When I put my foot on the accelerator...

ALAN ALDA (Narration) But the simulator isn't exactly like the real thing.

SANDY PENTLAND You got a speedometer, it works, and, everything works here. The one thing that's different in a simulator is that you can't, first of all you can't look around and see somebody coming from the side. The other thing is, you don't actually get the feeling of motion the way you do when you really move. Oh, you see the red light there?

ALAN ALDA (Narration) And until you get used to it, the steering is alarmingly light to the touch.

SANDY PENTLAND Turn left...

ALAN ALDA Oh, oh...

ALAN ALDA (Narration) It's OK, it's only the movies! Sandy's goal is to make a car smart enough to know what its driver is about to do before you do it - like changing lanes.

ALAN ALDA How can you tell how people are going to change lanes? How can a computer tell?

SANDY PENTLAND So what we did is, we had a number of different people drive around in this world for a fairly long period of time and we told them to do things, like we told them to pass, like once you... there we go.

ALAN ALDA I think I clipped the front end of that truck.

SANDY PENTLAND Ah, it's OK, it's covered by insurance. And we looked at what they did just before they did something like pass, or just before they turned.

ALAN ALDA Now, what did they do?

SANDY PENTLAND Well, it turned out that everybody had more or less the same pattern of behavior just before they did something, so for instance, when you pass, you pull over, sort of look around the guy, slow down to match speed and only then do you go to pass.

ALAN ALDA I'm a little nauseated here.

SANDY PENTLAND That happens.

ALAN ALDA That happens, huh?

SANDY PENTLAND Yeah. You get used to it if you do it, but it happens.

ALAN ALDA So what does the computer look for? How does the computer know I'm going to turn or pass?

SANDY PENTLAND It's looking for the little motions you do with a steering wheel and with a brake and accelerator that indicate that you're getting ready to pass or turn.

ALAN ALDA Now these nutty cowboys on the roads who seem to shoot over to that lane apparently out of nowhere... are they giving signals too that a computer could read, or do they just go like a nut and go to the right?

SANDY PENTLAND Even they will have a pattern that you can recognize. But probably for cases like that, the car would have to learn to specialize itself to the particular person, so there might be a generic way it works for most people, but that for the cowboys, the car would have to learn that this is a cowboy, we have to look for these things and not those things.

ALAN ALDA (Narration) So a smart car would check its blind spot and warn the driver only if it knew you were about to change lanes. But it could also keep a watchful eye on you for other signs of impending trouble.

ALAN ALDA Is this a camera here?

SANDY PENTLAND Yeah, there's a camera right near where the speedometer is looking back at you to see if you're looking sleepy or not.

ALAN ALDA How can it tell if I'm sleepy?

SANDY PENTLAND Well, it watches your face and looks at your blinking. If you're blinking a lot and your blinking gets longer and longer, that's an indication that you're getting sleepier and sleepier.

ALAN ALDA So then what? What does it do about that?

SANDY PENTLAND Well, there's a couple of things it can do. One thing it can do is turn on the air conditioning and wake you up or give you a warning that says get some coffee. Another thing it can do is actually put a little peppermint spray into the air conditioning, because peppermint's been shown to wake people up.

ALAN ALDA (Narration) By now I was over my nausea and beginning to enjoy the ride.

SANDY PENTLAND Want to pass this guy?

ALAN ALDA How fast can you go in this thing?

SANDY PENTLAND It goes up to about 120.

ALAN ALDA Does it?

SANDY PENTLAND Yeah.

ALAN ALDA Oh, let's see.

SANDY PENTLAND A little scary at that point.

ALAN ALDA (Narration) But soon the thrill of simulated speed gave way to curiosity about that camera in the steering wheel. So it was back to Sandy's lab to learn how a computer can not only recognize you but even tell how you're feeling, all by simply gazing at your face.

SANDY PENTLAND So here's a camera up here, right on top of the computer, and the first thing we need to do is we need to record a picture of you. There we are. Got you. Now we need to learn a little bit what you look like in different situations, so, why don't you look really tired.

ALAN ALDA Wait a minute, I have to get into this.

SANDY PENTLAND Got to get in... acting comes in here, right? OK. Let's record tired.

ALAN ALDA Tired.

SANDY PENTLAND So now we know when to say go home. And now we try and look confused.

ALAN ALDA Try and look confused. Try not looking confused.

SANDY PENTLAND OK, what's going on here? OK, good. So now we have you normal, we have you laughing actually also, we have you tired and we have you confused.

ALAN ALDA Right.

SANDY PENTLAND So this is the picture we just took of you to try and learn what your face looks like. So what the computer does, is it has a model of what a human face looks like that it's learned by looking at hundreds and hundreds of faces and using that, it's able to find where your eye and nose are. It then scales your head and rotates it into a position that's a standard position so that we can compare you against other people. So we get this ghostly "chip" we call it, face chip, which is just the center if your face, no hair, so if you combed your hair different, it's not going to get confused or anything like that. And now we can actually compute the description of what your face looks like as opposed to anybody else.

ALAN ALDA (Narration) It does this by comparing my face with a set of standard faces - these are just eight out of the full panel of one hundred - and assigning a value to each match. This set of values then describes my face. What's more, these values can be used to construct a sort of synthetic me from all those other faces - that's the real me on the left, the artificial me on the right. Another program maps the movement of key parts of your face as your expression changes, and can put a label on how you're feeling. And having your computer know how you feel is the goal of another Media Lab professor, Roz Picard.

ALAN ALDA This brain is really fun to see sitting here on your desk. Do you refer to that in your work?

ROZ PICARD Well, we refer to parts of it, we don't refer to the duct tape and hooks that are actually in that particular model.

ALAN ALDA I'm missing a little duct tape in mine. What... how do you relate that to what you do?

ROZ PICARD Actually, it's become very important to notice that, well, most of the work people are focused on, especially in computer science, focuses on the cortex, the outer part of the brain. People have largely ignored the inside of the brain, which includes a bunch of parts that are called the limbic system. One of the main parts, for example is this amygdala right here, that's removed for the moment. This is, functions kind of like the fire alarm in your house would. This fires and it can hijack the cortical part of the brain, your reasoning processes, and basically take over and cause you, for example, to be scared and to run...

ALAN ALDA And go woo, woo, woo...

ROZ PICARD Well, or whatever, yeah. And so what we're finding, actually is not only is this important for just fear and traditional emotion, but it's also important for decision making and for rational processes.

ALAN ALDA What you're saying is that this little part of the brain reacts to something on a way that's prior to thought and affects the thought process. Something that's sort of emotional in some way. It's an immediate response. So do you get that into the interaction between me and the computer?

ROZ PICARD Well, that's what we're trying to do right now. We're trying to give the computer the ability that humans have to pick up on emotional cues, and use those on its interaction with you.

JENNIFER HEALY Clench your jaw for a second for me so I can find the muscle.

ALAN ALDA (Narration) Here we go with the wires again, this time administered by graduate student Jennifer Healy.

JENNIFER HEALY OK, now you can see this on the blue here, so go ahead and clench your jaw. See, so that's how I know it's on a good place. Yeah, great. Good job.

ALAN ALDA Wow, look at that.

ALAN ALDA (Narration) As well as the tension in my jaw, she measures the clamminess of my hands.

JENNIFER HEALY Yes, if you've ever noticed when you're nervous, your palms get all sweaty, and that sweat helps the electrical conductivity.

ALAN ALDA I see.

ALAN ALDA (Narration) Another sensor picks up my heart rate. The idea of all this is to let the computer know my emotional state - which unknown to me was about to be aroused.

JENNIFER HEALY Look at that, isn't that cool?

ROZ PICARD He went off the scale there again.

ALAN ALDA (Narration) Now as anyone who knows me can tell you, the secret to manipulating my emotions is food.

ALAN ALDA All right, all right. I'm just going to think of a saltine. I bet you a saltine gets me right back... it went down, see?

ROZ PICARD Now you went down. You just dropped all the way down.

ALAN ALDA Saltines are not very exciting to me. Now watch this. Pasta. Oh, red sauce, a little ricotta on the side. Some hot red peppers.

ROZ PICARD The peppers went up.

ALAN ALDA Did they? I could have told you that.

ROZ PICARD I don't know about the ricotta though.

ALAN ALDA (Narration) The point of having my computer know my emotions - through my facial expression or my vital signs - is that it could adjust itself to my mood, running faster when I'm bored, trying another tactic if I'm frustrated. It's all part of making our interactions with computers more like interacting with other people.

ALAN ALDA Does this meant that at the next stage of this, somebody will have to be hooked-up like this in order for the computer to recognize what they're going through?

ROZ PICARD No, eventually we hope that these sensors will disappear into your clothing or into the twiddler itself, into the devices that you're naturally in physical contact with. In fact, people are physically in contact with computers more than with other human beings, frequently, so there's a lot more opportunity than you realize for these sensors to be collecting information.

ALAN ALDA (Narration) And thereby hangs another tale. Stay tuned.

back to top

PRIVATE EYES

ALAN ALDA This cafe is just around the corner from MIT in Cambridge. And as you can see, computers are everywhere here. I can plug my laptop into the Internet while I drink my tea. But if you think this is the ultimate in computers being everywhere, this Thad Starmer, who doesn't just take his computer everywhere he goes, he wears it wherever he goes. You wear it everywhere?

THAD STARNER Well, if I'm at a conference or out on the street, pretty much so, yes. You get information at the strangest times, and it helps you manage it during your day.

ALAN ALDA Well, explain that to me. Let's say you're passing somebody in the hall and they say something to you, interesting. What happens to that information? It just goes into this little thing in your eye here?

THAD STARNER Basically. What we have here is a keyboard...

ALAN ALDA Oh, what's that?

THAD STARNER This is my keyboard. It uses more than one button at a time to get letters or words.

ALAN ALDA Oh, it's like playing the clarinet or something.

THAD STARNER Yeah, almost exactly like that.

ALAN ALDA When you click those words in, do you see them up on your eye there?

THAD STARNER Yeah, this is a private eye, and what it does...

ALAN ALDA Doesn't it bother you to just see out of one eye all day long?

THAD STARNER No, what's nice about this device is that it looks like I see words on top of your nose right now, because I'm sitting here with you. I've typed in like the dates and who this is for and who you are and where we are and that then starts pulling up files from my past to remind me about what I want to talk about.

ALAN ALDA It would be really useful taking a test.

THAD STARNER Yeah, actually they let me use it on tests now. If I have it with me...

ALAN ALDA Get out! What do you mean, they let you use it on tests?

THAD STARNER On tests. I have all my books on line....

ALAN ALDA Naturally.

THAD STARNER But the point that the professors make, say if I have this with me all the time, then it's part of me. And I'll have it out in the field, so there's no reason why I shouldn't be...

ALAN ALDA In a way, your professors already accepted this as an auxiliary brain.

ALAN ALDA (Narration) And already in local MIT hang-outs, Media Lab students with their auxiliary brains slung in pouches across their bodies no longer raise neighborhood eyebrows. Most wear their private eyes like Thad Starmer, mounted on eyeglasses. Brad Rhodes is setting his own, jauntier, fashion trend. Right now, the computers worn by this group of "cyborgs" as they call themselves, aren't networked together, but already they're experimenting with wireless links both within the group and to the Internet. And in fact one of the Media Lab cyborgs is already online most of the time. Steve Mann's wearable computer is even equipped with a head-mounted camera, and he is continually transmitting images from his everyday life to his very own Web page.

ROZ PICARD Steve is evidently over at the food trucks wearing the Wear Cam on his head, and we can see what's he's doing. All right, Steve's in line right now, he's looking at the menu here and it looks like there's a few people ahead of him and it's not too long of a wait. We can click on the next image he sent most recently to see the menu up closer, and see what our choices are at this particular food truck.

ALAN ALDA (Narration) I was about to receive what to the best of my knowledge is my first e-mail from a cyborg.

ROZ PICARD So here's a piece of mail from Steve, and he's asking if you want falafel.

ALAN ALDA You know what's great about this? If all I saw was his e-mail, all I'd see was I had a choice of whether I wanted falafel or not. Falafel is nice, I like falafel, but when I look at the picture and I see this picture of the whole menu, I can see I can get tabouli if I want. So tell him I want some tabouli.

ROZ PICARD OK. Some tabouli it is.

ALAN ALDA (Narration) A few weeks later I got to meet Steve Mann in person.

ALAN ALDA Steve, hi.

STEVE MANN Hi.

ALAN ALDA What are you working on?

STEVE MANN It's Alan.

ALAN ALDA Yes, Alan. How can you tell?

STEVE MANN Well, my computer recognized you. How are you doing?

ALAN ALDA (Narration) Captured by Steve's video camera, my face was sent back to the main computer at the Media Lab, run through the face recognition program, and the result flashed back to Steve's computer. Steve, by the way, has a ham radio license, since as well as everything else, he's a walking radio transmitter.

STEVE MANN Yeah, this thing on my head is an antenna. It's just a short whip antenna. There's a copper screen here...

ALAN ALDA Oh, there's a whole lot of screen in there. What is it, I take it that's not to keep out mosquitoes? What does that do?

STEVE MANN That's the ground plane for the antenna.

ALAN ALDA (Narration) Steve Mann's been experimenting with personal, wearable sensing devices for almost a decade now... and most of them are still stashed away in his dorm room at MIT. This is his first helmet-mounted rig, dating from the mid-1980's, and already possessing a period charm all of its own.

STEVE MANN ... over my head like this, and then I could see my screen here. These are the shoes that I was experimenting with...

ALAN ALDA (Narration) The shoes were able to monitor his footsteps.

STEVE MANN So inside here I had various sensors that produce an electrical signal...

ALAN ALDA (Narration) But for sheer inventiveness it would be hard to beat these thermostat-equipped briefs, able to wirelessly control his dorm-room air conditioner.

ALAN ALDA You never had to get out of bed to change the thermostat because your underwear was changing the thermostat for you?

STEVE MANN Yeah, yeah.

ALAN ALDA That's unbelievable.

ALAN ALDA (Narration) But if I thought I'd just seen the ultimate in wearable computers...

NEIL GERSHENFELD Let me ask you to put that on the ground and rest your foot on it.

ALAN ALDA (Narration) I'm in the lab of MIT professor Neil Gershenfeld.

NEIL GERSHENFELD Let me ask you to shake hands with me. Now look at the screen here. That's data in your body. Let go, it'll stop. Shake hands... our bodies are data networked right now.

ALAN ALDA You mean this thing, the thing under my shoe...

NEIL GERSHENFELD ...is putting data into your body.

ALAN ALDA Into my foot, up my leg, through my arm, down...

NEIL GERSHENFELD Yes. And then when you touch me, it goes through my body. And it goes up to my shoe to my shoe computer.

ALAN ALDA Oh, come on. You've turned me into a computer here.

NEIL GERSHENFELD A data network.

ALAN ALDA I am the Internet.

NEIL GERSHENFELD You're a bodynet. It's your... actually, it's very important. If you let me have that back, I'll show you how it works.

ALAN ALDA I can't believe this.

NEIL GERSHENFELD It's actually very important. You're not the Internet. You're a bodynet.

ALAN ALDA The worldwide Alan.

NEIL GERSHENFELD You're the person-wide Alan, because...

ALAN ALDA Because it never goes, it doesn't go to anybody else but who I touch.

NEIL GERSHENFELD It's your bits.

ALAN ALDA My bits, yeah. What's in here, this is...

NEIL GERSHENFELD Let's think about your shoes. First of all, there's plenty of space there. Nobody's fighting over getting access to your shoe right now. That'll change, but right now nobody is. Second, there's power. When you walk, you make watts of power. And in a couple minutes I'll show you a shoe power source. With this mechanism, your shoe can talk to your body, turning your body into a data network. Suddenly, a foot bottom makes more sense than a laptop. You have your shoes when you get dressed, rather than carrying technological clutter around, just get dressed and let your shoes do all this work.

ALAN ALDA I don't even think about it, right? I just put my shoes on and all of a sudden I'm wired up.

NEIL GERSHENFELD That's right. Now, for example, when we shake hands you can give me your business card. If you pick up a telephone, you can give it phone numbers. If you touch a doorknob you'll authenticate...

ALAN ALDA This can have a really large amount of data stored in here.

NEIL GERSHENFELD It's a computer. Your body's a network and it's a computer. This is where we're working on your shoe power source. When I walk, I'm making up to maybe fifty watts - a lot of power. And we think we can get at least a watt out from that. So what we're developing here are the shoe inserts, so this is... there are a couple of different types. This is a piezo-polymer laminate, and when you step in this, the spring stores energy, puts it into stretching this, which makes a charge this way.

ALAN ALDA (Narration) And sure enough, with each simulated footstep, the power builds up.

NEIL GERSHENFELD Here we're starting to become a sneaker manufacturer, working with Nike we made our own shoe mold to start molding soles, where we're going to start taking these things, putting them into the soles, connecting then to the shoe computer I showed you there to make it something that lives in your shoe and you can forget. And also one of the other nice implications is, with a shoe power thing like this, eventually you'll run down the power if you're sitting, so you need to go for a little walk and clear your head and power up your shoe.

ALAN ALDA I love this... all this technology comes down to taking a few steps with your feet.

ALAN ALDA (Narration) OK, it was finally time to step into the future myself, and see what the cyborgs see. I was about to experience "augmented reality". All around the lab are little color-coded signs.

ALAN ALDA This way to the egress.

ALAN ALDA (Narration) When the camera on my head mount sees one, it triggers my on-board computer which then displays in my video goggles a pre-determined message.

ALAN ALDA Do not throw away any of the rolls from this printer.

ALAN ALDA (Narration) That message needn't be just text. In this case, an electrical panel, I'm treated to a computer animation of what's inside. And when I look at a video screen, my on-board computer plays me a video. Eventually all sorts of things in my environment- say a painting in a museum or a copier machine in my office - might trigger my wearable computer to provide me with useful background information. Surrounded by most of the cyborgs presently on the planet, I wondered about the next generation of wearable computers.

ALAN ALDA Do you have like, fashion conversations about... Do you give much thought to different ways that this stuff looks?

JENNIFER HEALY People give it a lot of thought, actually. There's a lot of debate whether it's good to cover the eye at all, whether you might want it on your arm, a screen. We have an undergraduate working on an even better private eye which would only take up a fraction of this space. It's a little prism that'll directly show you the screen and yet be here, and allow your eyes to be seen by the person you're having a conversation with.

STEVE MANN Hopefully be completely invisible. One won't be able to tell that we're wearing special glasses. We won't have these, this display's about six years old with a big cathode ray tube up on top of it. You know, they're getting smaller and smaller and eventually we'll have them built into a regular pair of spectacles and the only evidence that somebody is wired will be the fact that their eyes sort of go across in a sort of reading motion. We'll think that they're possessed by the devil or on the Internet.

LENNY FONER And there's some precedent for this in that it used to be that when Walkman's first came out, everybody said, you know, who'd want to wear this thing on their head, these headphones all the time and then all of a sudden it became stylish and nobody gives it a second thought, I mean, you see somebody wearing a Walkman, big deal...

ALAN ALDA (Narration) In fact, fashion designers have already started to play with the idea of incorporating computers into clothing and accessories. These sketches were submitted by students of the French fashion designer Jean Claude Gaultier for a contest held in his Paris studio late spring. But for today's cyborgs - that's already the boring stuff. For them, what's really interesting isn't what's already possible - it's what's out there just beyond.

back to top

BRAIN MUSIC

ALAN ALDA (Narration) At the heart of the Media Lab is a large dark space called the Cube - on my last day there filled with eerie sounds.

TOD MACHOVER This is what we think of as a music videogame, actually. It's called harmonic driving. The idea here is that it's a piece of music that you actually drive through, and by the way you drive and the decisions you make while driving, the piece actually changes. You want to give it a try?

ALAN ALDA Yeah, if I crash, this piece is over, or what?

TOD MACHOVER Oh, it's a terrible thing that'll happen...

ALAN ALDA Oh all right, let me see. Let me do it...

ALAN ALDA (Narration) This is the prototype of one of several music-making machines, called hyperinstruments, designed by a team led by the Media Lab's Tod Machover.

TOD MACHOVER Hey, all right. We start by building special instruments for the world's best musicians, like Yo Yo Ma, and Peter Gabriel and orchestras. For the last five years or so, we've been trying to build instruments for audiences.

ALAN ALDA Where the audience makes the music.

TOD MACHOVER Where the audience in the very least experiments with the music, sometimes performs the music and makes the music. These are sensors, one, two, three, four, and then under each of your feet, those are measuring how much electrical current is being picked up from your body. So as you move, it knows where you are in this field.

ALAN ALDA This, can I make the same... if I go to the same place, it does the same thing every time?

TOD MACHOVER The exactly...

ALAN ALDA So it's really like playing an instrument. There's no randomization involved.

TOD MACHOVER No randomization, plus right in the middle there's supposed to be a sort of cymbal sound.

ALAN ALDA (Narration) This sensor chair, like the other hyperinstruments here, was being readied for an appearance at New York's prestigious Lincoln Center Festival, just a few weeks after my visit. Luckily, it was still just a prototype...

TOD MACHOVER You broke it! If you want to come over for a second to what we call the rhythm tree, that's a different idea. That's one that's designed to have something physical that you hit that gives you a sound exactly when you hit it.

ALAN ALDA OK.

ALAN ALDA (Narration) The performance Tod Machover and his team are preparing is called The Brain Opera, and several of his hyperinstruments - designed to be played by the audience - look like bits of brain.

TOD MACHOVER This one is, you really have to work to get the kind of regular rhythm that you got with the sensor chair. Why don't you give it a try?

ALAN ALDA Right. Hello? Ladies and gentlemen, my fingers never left my hands.

ALAN ALDA (Narration) Teresa Marrin has invented a hyperinstrument called the digital baton, which makes different orchestral sounds depending on where it's pointed.

TERESA MARRIN Electric guitars, trumpets down here.

ALAN ALDA What's this again?

TERESA MARRIN That adds string sounds.

ALAN ALDA (Narration) An infrared detector follows the tip of the baton in space.

TERESA MARRIN I like that. Now take it down there, you'll get some strobe bass.

ALAN ALDA (Narration) Now on this instrument, supposedly the better I sing, the louder my angelic accompaniment.

TOD MACHOVER That was really good.

ALAN ALDA I think I just wrote Chariots of Fire.

ALAN ALDA (Narration) Three months after my visit to the Media Lab and Tod Machover and his band of hypermusicians and hyperinstruments were ready to meet the public. By now the heavenly choir was accompanied by angelic video images. The idea of The Brain Opera is that audience member first get to generate their own hypermusic on what were now fully-tested hyperinstruments.

ALAN ALDA The original was like being inside a skeleton and this is like being inside the real thing. It really does change the quality of the music.

ALAN ALDA (Narration) Tod Machover's vision for the Brain Opera is that the sounds generated here by the audience will be incorporated into the performance that follows.

ALAN ALDA It's hard to believe that this is going to become a symphony. In just ten minutes.

ALAN ALDA (Narration) Despite the plan to employ some of the audience's input in the performance, I can't say I noticed any input from my own earlier efforts in the lobby. I listened in vain for my perfectly sustained E-flat. But at one point in the performance, an even wider audience had their say when Teresa Marrin used her digital baton to select contributions from people listening in over the Internet. The Brain Opera, by stretching our conventional ideas about musical instruments, about music, about the whole relationship between audience and performers, struck me as a an interesting metaphor for the Media Lab itself - exploring a future in which computers change the very nature of everyday experience.

ALAN ALDA That's all for this edition of Scientific American Frontiers. See you next time.

back to top