ALAN ALDA (Narration) This is MIT's famous Media Lab. This is the
lab's pet dog, Silas. And this is me. Right now we're all
virtual. But the real world here at the Media Lab is even
more extraordinary. Silas does tricks, while Gandalf takes
me to the stars. I get my own cooking show... drive a smart
car. And, for a brief, embarrassing moment, become a cyborg.
This is Alan Alda. Join me for "Inventing the Future", on
Scientific American Frontiers.
back to top
ALAN ALDA (Narration) My first glimpse of MIT's Media Lab came
- appropriately enough - on the screen of my trusty laptop,
via the Internet. The Lab's reputation is that life there
runs on fast forward... That here today is a preview of tomorrow.
And the word is that the tomorrow now being hatched with a
stark exterior is a startling one - in which computers, as
we've come to love them or hate them, disappear. Now as a
computer buff myself, this was something I had to see. If
it's good-bye laptop, it's hello... what? Well, here I was,
moments after arriving, balancing on my empty hand a virtual
version of the lab complete with its own welcoming host.
ALAN ALDA Sandy, can you wave your arms about a little?
ALAN ALDA Yeah, yeah, there you are.
SANDY PENTLAND Hi.
ALAN ALDA How are you?
ALAN ALDA (Narration) The man behind the curtain was Sandy Pentland.
SANDY PENTLAND Virtual me in the virtual lab. Virtually on
ALAN ALDA (Narration) His answer to my very first question turned
out to capture the lab's trademark research philosophy.
ALAN ALDA Sandy?
SANDY PENTLAND Yes?
ALAN ALDA Why are we doing this?
SANDY PENTLAND Well, we don't
know. We figured that...
ALAN ALDA You don't know? Ha.. ha. No, I mean, what is this sort
of a glimpse of here, what is... I mean, it's an amazing thing,
but how will it help us get to the future?
What we're showing here is that we can take real things and
virtual things and put them all together. You can think of
this sort of the next generation of virtual teleconferencing.
We could have a little crowd of people and we could all explore
the building together. You could maybe pick parts of the building
up and move it, and so forth.
ALAN ALDA Oh, in other words, I could be here, and you could be
in Toledo... Spain, and we could both go through this virtual
SANDY PENTLAND Yeah, right. That's right.
And we could grab things and move them. There could be more
than one person that's virtual like me. We could have people
not just from Spain, but from England and Japan and we could
all get together with our virtual representations and talk
about things and design things and...
ALAN ALDA I mean, I could be working at my computer and then all
these little guys could show up and I could tell them to go
SANDY PENTLAND That's right. That's the idea...
is to be able to combine reality with virtual reality, to
ALAN ALDA Well, that's attractively weird.
ALAN ALDA (Narration) What's even weirder is my playing Doom. But
in this game neither I nor my plastic gun are in any way connected
to the computer. The computer knows what I'm doing because
it's watching me.
ALAN ALDA Got him, captain. You can't, surely you can't have been
working all these years to make this game safe for democracy.
What's this going to lead to?
SANDY PENTLAND No, no. What
really happened is, we were working on being able to understand
human emotion and recognize gestures and things, and I came
in one day and someone, one of my students, had hooked it
up to Doom.
ALAN ALDA (Narration) Bruce Blumberg is another of the team working
on an environment that can figure out what someone in it is
doing - a Smart Room.
ALAN ALDA And it seems to know I'm here, because that blue blob
is me and you, right?
BRUCE BLUMBERG Right. And yeah, if you
look at the top of the screen, or what we call the magic mirror,
you'll see a video camera. And the video camera, when the
system first starts up, records a whole bunch of pictures
of the background, and then, when you walk into the world,
it's looking for differences. So it's comparing the picture
with you in the world with the picture without and saying,
wherever it's different, it must be a person.
ALAN ALDA Now what are those little squares, the green square,
the red square?
BRUCE BLUMBERG Right. Well, what we're doing
is, once we have the blob - it doesn't seem quite right to
call you a blob - but once we have the blob, we do some very
simple gesture recognition. So part of that is figuring out
where the hands are. So if you hold out the hand, the computer
program has some idea of how long your arm should be, and
the green cube there corresponds to where it thinks your hands
are. And similarly, if you knelt down, it will think that
you're sitting. And that's basically done on height.
ALAN ALDA What do you use the blob for? What good is the blob?
BRUCE BLUMBERG Well, the blob in itself is, only so interesting.
But the purpose behind it is that we can use this as a front
end, say, to a virtual reality system where we have creatures.
ALAN ALDA Could I see the creature?
BRUCE BLUMBERG Of course.
ALAN ALDA (Narration) And there, in the Magic Mirror, is Bruce
Blumberg's animated creation, Silas.
ALAN ALDA Hello, Silas.
ALAN ALDA (Narration) Now Silas of course is simply a computer
program that's getting data about what I'm doing via my blob.
But share a Smart Room with him for a while and that's easy
ALAN ALDA Hey, Silas... Silas, come here. Sit. Sit. Oh, wow, look
BRUCE BLUMBERG He's like a real dog, but sometimes
he'll respond to your gestures and sometimes not.
ALAN ALDA And will he beg?
BRUCE BLUMBERG Yeah. Put your both hands
over your head.
ALAN ALDA Come on.
BRUCE BLUMBERG Wow, you're a pro. Now, sometimes
Silas will do what he wants. So right now I think he's heading
off to get a ball, because he figures...
ALAN ALDA Here he is. Give me the ball. OK, now I'll throw the
ball for him.
BRUCE BLUMBERG There you go.
ALAN ALDA He always goes after the ball?
BRUCE BLUMBERG He goes
after the ball unless he really had to get a drink of water
or something else that was more important.
ALAN ALDA How does he know he wants a drink of water?
He has a number of built in motivations. And these motivations
go up over time and so he's balancing sort of these internal
goals with what's happening out in the world.
ALAN ALDA After he has a certain number of drinks of water, does
he head for the fire hydrant?
BRUCE BLUMBERG Yeah. And you
see, that's what makes animals, and this kind of guy different
than regular robots, is that animals are constantly juggling
a zillion goals, whereas robots typically only focus on one
thing at a time. And part of the serious science here is,
well, how do you build systems that can deal with multiple
goals. With the same sort of common sense that dogs have.
ALAN ALDA Ha, ha. What's this thing on the corner there?
BRUCE BLUMBERG He's a hamster. He's another little creature too.
You may be able to pick him up. It may be a little problematic.
ALAN ALDA I think I've got him.
BRUCE BLUMBERG Ah, no.
ALAN ALDA I'm looking over there to see, and it's not there. I'm
starting to believe this.
BRUCE BLUMBERG But I think that
within a couple of years, we'll be able to...
ALAN ALDA (Narration) But no sooner did we decide to forget about
ALAN ALDA You know, the hamster has just suddenly flown off the
BRUCE BLUMBERG Well, let's take advantage of this.
Hold out your, point with your right hand. OK. Now Silas is
going to respond to your gestures, but he doesn't really like
the hamster, because everyone likes the hamster, and that
takes away attention...
ALAN ALDA He's jealous.
BRUCE BLUMBERG He's jealous. And eventually
it gets too much and he has to... now why don't you just point,
like that. And maybe Silas...
ALAN ALDA He loves to walk through my legs. What's...
See, Silas is sad now, so he's going away because you punished
ALAN ALDA Oh, I see. All right, come back. Come over here, Silas.
Let's make up. No, Silas... Silas. He really has a mind of
his own. Come here. Come over here. Come here.
ALAN ALDA (Narration) So here I am, playing virtual ball with a
virtual dog, and there isn't a computer in sight. But computers,
of course are running everything - figuring out my intentions
as well as endowing Silas with his doggy behavior.
One of the things that you didn't see there was training Silas,
because, after all, animals learn, and so I'm very interested
in how can you have these virtual creatures learn like real
ALAN ALDA That's the hamster?
BRUCE BLUMBERG Yup. This circular
area here, this gray area, represents an electrified mat.
What happens is, a sequence of lights will come on and then
when the red light comes on, the hamster gets shocked.
ALAN ALDA Once he gets shocked, what makes him want to come back
on this gray thing at all?
BRUCE BLUMBERG Well, he's not that
smart. But also, it's... these things represent food sources.
So really what he's doing is, he's foraging, and in between
foraging, he's periodically having to deal with the fact that
he's getting shocked. So you'll notice now, the red light
never comes on, because he's always fleeing before he can
ALAN ALDA You mean to say that we're going to have like virtual
dogs roaming around our houses?
BRUCE BLUMBERG Well, I think
it's a possibility. I mean, I have a dog at home, and I'm
not sure I'd trade, well, maybe, she's an old dog, but um,
I'm not sure I'm willing to trade a real dog for a virtual
dog. But you could imagine sort of a St. George and the Dragon
kind of environment where a kid could have a pet dragon at
ALAN ALDA Let's say I'm not interested in a virtual pet. But there's
other things you can do with this knowledge, right? Like what?
BRUCE BLUMBERG Well, I think that fundamentally the problem
I'm trying to solve is, how do you build intelligent things
that do the right thing for the right amount of time over
extended periods of time and adapt?
ALAN ALDA (Narration) The latest challenge to Silas' intelligence
is to get him to figure out his own way through a computer-game
maze. He's getting very good at it. But it may turn out that
Silas - or intelligent computer creations like him - will
have their greatest impact in helping the rest of us find
our way through real life mazes. Media Lab professor Patti
ALAN ALDA How is he going to be useful to you in your work?
PATTI MAES Well, I admit that Silas is, the only sort of reason
for his existence is entertainment at the moment, and science
of course, but we're actually making similar types of software
that can help people with real hard problems.
ALAN ALDA Like what?
PATTI MAES Well, one such problem is finding
things that you may be interested in, there's actually more
and more information available every day, there's more and
more products available on the markets, and it's actually
really hard for a consumer to find the things that they may
like in this mass of information, mass of products out there,
so we've been building software agents that sort of are on
the lookout for you, trying to find things that you may be
interested in. Actually, there's some things that we can show
you right now and that the viewers can try out themselves
on the Web.
ALAN ALDA Oh, that's now. It's on the Web now? Oh, let's see.
ALAN ALDA (Narration) As someone who's spent hours searching for
information on the Web, having a notional Silas fetch things
for me is appealing. Patti Maes' intelligent agent is called
Firefly, designed to help people find music or movies they
ALAN ALDA Now, if I click on this I can rate it.
PATTI MAES Yup,
you can always rate it.
ALAN ALDA And that's my rating.
PATTI MAES Yes. Lots and lots of
people tell the system what they like and dislike on the order
of 10,000 people, and so everybody has their agent that that
particular person likes and dislikes. And all these agents
can get together and compare their users with one another,
so my agent will talk to your agent and try to find out whether
we are similar, whether we have similar tastes.
ALAN ALDA So if we have a basically similar taste, as the agents
match our tastes, then it'll tell me about movies you've seen
and liked that I might not have thought of.
PATTI MAES Yes,
exactly. Click on one of the movies and then ask the system
to predict how much you would rate that movie.
ALAN ALDA (Narration) Knowing this wasn't a good idea, I was talked
into checking on movies I've appeared in. While Firefly in
its current form only works to help find music or movies that
people with similar tastes to mine have enjoyed, it's easy
to see how an intelligent agent like this could search out
restaurants or stocks or vacation resorts - even politicians.
PATTI MAES What people really need help with is finding other
people, finding other people that share some of their interests
or that have different interests, or that are an expert in
a particular area, etc. And so that's actually one of the
other functions that this system of agents provides. Agents
can do matchmaking, not so much romantic matchmaking, although
you could argue that if you go and check out the people that
like the same movies and the same music or whatever, books
that you like that maybe they're...
ALAN ALDA Yeah, well are you working on a matchmaking...
PATTI MAES You'll at least have a good start.
ALAN ALDA Are you working on literally matchmaking?
It's not, sort of, we didn't build this to do romantic matchmaking,
but as it turns out I do know a lot of couples that have found
ALAN ALDA That met this way, really?
ALAN ALDA (Narration) Now you might well be wondering - as I was
by now - what was all that about computers as we know them
ALAN ALDA OK.
KRIS THORRISON How does that feel?
ALAN ALDA Now these wires go from the gloves down my back?
KRIS THORRISON Down your back and into the computer. And this eyetracker
will allow it to see your eye movements.
ALAN ALDA (Narration) If this is what's going to replace them,
I'm not sure I'm ready to give up my mouse and keyboard just
yet. Graduate student Kris Thorrisson is helping me into an
outfit that will keep track of where I'm looking.
ALAN ALDA Oh, my god! Look at my eye on that monitor.
ALAN ALDA (Narration) The gloves, meanwhile, are to allow the machine
to follow my gestures.
ALAN ALDA Oh, it speaks Italian
ALAN ALDA (Narration) The idea of all this is to allow me to gesture
to my computer.
ALAN ALDA I actually gesture at my computer quite a bit, but the
computer doesn't respond. In that regard I can understand
how it would be good to have gestures understood by the computer,
but why are you engaged in this? Why do you want the computer
to read my gestures?
JUSTINE CASSELL Because we do gesture.
We gesture at computers, we gesture at each other, like I'm
doing right now. We even gesture when we talk on the telephone.
So yeah, there you are, you're saying to your computer, not
that, the... I told you...
ALAN ALDA But mostly it's like this. I can go like that.
JUSTINE CASSELL And it still doesn't respond.
ALAN ALDA No, it doesn't. It doesn't understand.
A slow computer. Well, we'd like to speed it up a little bit.
So the idea is that...
ALAN ALDA (Narration) The Media Lab is famous for its "demos",
demonstrations of technology that is still far from perfected,
but gives us a glimpse of the possible. The idea here is for
a character I can converse with.
GANDALF Hello, I am Gandalf,
your guide to the galaxy.
ALAN ALDA Take me to Mars.
GANDALF Why not?
ALAN ALDA (Narration) Gandalf has a simple off-the-shelf speech
recognition program to know what I'm saying. The novelty here
is that he's supposed to respond to my gaze and my gestures
as well as my voice.
ALAN ALDA Here's Mars, right? It's red. Tell me about Mars.
Mars has two moons.
ALAN ALDA Tell me more.
GANDALF Mars is 6,000 kilometer in diameter.
ALAN ALDA Tell me more about Mars.
ALAN ALDA (Narration) He's also got something of a personality
of his own.
GANDALF It's a cool planet.
ALAN ALDA You said it. Tell me about Saturn.
GANDALF Saturn has
ALAN ALDA Tell me more.
GANDALF The rings are made from rocks.
ALAN ALDA I love how it goes like this. It goes, I say, tell me
more, he goes, the rings are made of rocks, like, what more
do you want to know? Hey, I already told you about Saturn.
And this tapping of the fingers, what is that, like...
KRIS THORRISON It's just to show you that it's alive.
ALAN ALDA That it's alive, yeah. What was the point of wearing
all this? What did that do? How did that change anything?
JUSTINE CASSELL So we talked earlier about interaction of
verbal and nonverbal behavior. You saw an example of that.
He knows, Gandalf knows when you're looking at him, and he
knows when you're looking at the screen. And when you looked
at the screen, he looked at the screen too, just to be friendly.
And when you looked at him, he got ready to answer a question,
or to interact with you. When you looked away, he took a powder.
ALAN ALDA Did my making gestures change the way... I didn't see
a change in the way he answered questions when I used my hands.
JUSTINE CASSELL In the future he'll understand the content
of your gestures. For the moment, what he understands is that
ALAN ALDA The ways we've thought of computers up until now has
sort of kept computers in the box, and the way you're thinking
of computers is giving them a chance to get out of the box
by thinking of them more as people. I mean, you're making
them listen and feel and hear and see like people, so then
they can be all over the place. They don't have to be only
accessed by a keyboard and a mouse.
JUSTINE CASSELL Absolutely.
That's the point. They should be able to be social actors.
Actors in our social world. They're not going to be people
because they're not people and we don't want to think that
they're people ultimately. I don't.
ALAN ALDA But they might be easier to use if they got information
from us the way other people do.
JUSTINE CASSELL Absolutely.
That's the point.
ALAN ALDA (Narration) OK, we have smart rooms that can track what
I'm doing, intelligent agents that can act on my behalf, characters
I can talk to like another person. And this guy - who here
is simply mimicking my movements, but is really part of a
demo of a smart desk, which tracks my gestures without wires
- as Gandalf will once he knows what gestures mean.
ALAN ALDA What a stupid grin.
ALAN ALDA (Narration) I was beginning to see where my computer
is going - off my desk and into the rest of my life.
SANDY PENTLAND In one sense, what we're trying to do here is build
a virtual butler, right, somebody who'd be around with a towel
and say, oh, master is getting up now, I'd better start the
coffee. So if we can duplicate those sort of abilities, I
think that'll be fun. It'll be nice, it'll be convenient.
ALAN ALDA You think that'll be like an early step, an actual thing
that'll walk around and do things for us?
SANDY PENTLAND Oh,
I don't think it'll walk around. I think what you'll have
is little sensors in the corners of the rooms that know when
you're asleep and know when you're getting ready to get out
ALAN ALDA And then parts of the house will just do it?
That's right. And you know another application for something
like this is if you have young children it might warn you
when the kids are playing near the stairs so you can go and
shush them away from the stairs. So we're building eyes and
ears to be helpful.
CLAUDIO PINHANEZ Welcome to the Media
Lab cooking show.
ALAN ALDA (Narration) You never know what's around the corner in
the Media Lab, and before I know it I'm guesting on a cooking
ALAN ALDA So, we're going to do artichokes on top of pasta, right?
The artichoke is a sauce. The artichoke's been soaking in...
ALAN ALDA (Narration) It's actually another of those Media Lab
demos, this time of what one day might be a TV studio where
the cameras themselves get the shots, with no one operating
ALAN ALDA See, it's starting to get yellow...
ALAN ALDA (Narration) I'd been asked by Claudio Pinhanez to prepare
one of my favorite recipes, complete with the sort of details
that require tight close-ups.
ALAN ALDA The choke is that fuzzy stuff that's in the... By the
way, please don't try to follow the recipe. It's one of my
favorites, but since we didn't want to shut down the Media
Lab by setting off the sprinkler system, we weren't actually
able to cook anything. Then you take this and pour it over...
that's the sauce. You pour it over the pasta. This doesn't
look it, but it's delicious. It's really much better than
this when it's chopped up and it looks nice.
ALAN ALDA (Narration) Supervising the project is lab professor
ALAN ALDA Why were we doing this? Why was this, how will this turn
into something in the real world?
AARON BOBICK Our goal here
is to build what we call Smart Cams, which is basically a
camerman-less camera. Now a cooking show is particularly nice
for us because we have a basic notion of a script - that is,
the sequence of steps, a recipe, in fact, of what the actions
are, so we can make use of knowing something about those actions
to actually make the computer able to do the vision that it
needs to finally give a close up of those delicious looking...
ALAN ALDA (Narration) Claudio has already starred in a solo demonstration
of the Smart Cam system. As with all the Media Lab demos,
it's meant to show how something can be done, more than how
it will be. But the key point is that the system finds each
shot from a simple written description, rather than having
a camera operator interpreting the director's instructions.
AARON BOBICK Well, what you have here is a switcher, the kind
of thing a director would normally see where he's got the
input from the three different cameras and the director gets
to decide which of these cameras he's going to send to the
viewers. But this is a Smart Cam system where the director
can call whatever shot he wants and the cameras respond.
ALAN ALDA So the director can type in directions. Like I could
say on this camera, give me a close up of the cook? Can I
AARON BOBICK Absolutely.
ALAN ALDA (Narration) I'm proud to say that my guest appearance
with Claudio - if it produced nothing edible - will be useful
in testing the system.
AARON BOBICK A question that's been
asked to us is how specific was it to him and the way he did
things, and it'll be very nice for us to take a different
recipe to show how much or how little work we need to do to
make this exact system deal with the ultimate Alan Alda cooking
ALAN ALDA OK, now you cut it in half...
ALAN ALDA (Narration) What's remarkable about the system is that
it's able to recognize actions like mixing or cutting, as
well as find objects like people or bowls. And as with everything
at the Media Lab, who knows what it could lead to - perhaps
even home videos that people will actually want to watch!
Meanwhile, it's one more demonstration of how computers can
tell what we are doing or intending without our spelling it
ALAN ALDA Then you take this and pour it over...
ALAN ALDA (Narration) Of course, computers still have no sense
ALAN ALDA This doesn't look it, but it's delicious.
ALAN ALDA (Narration) After smart rooms, smart desks and smart
cameras - you guessed it, smart cars.
ALAN ALDA I seem to be doing OK now, why would I need a smarter
SANDY PENTLAND Well, you know, 40,000 to 50,000 people
a year die in this country alone in accidents. Most of those
accidents are bad judgment, or people didn't see things.
ALAN ALDA Where'd that guy come from?
SANDY PENTLAND Yeah, exactly.
ALAN ALDA Wait a minute, I mean it!
SANDY PENTLAND This is Boston, you have to remember that.
They come at you from all angles...
ALAN ALDA How would a Smart Car have helped me there?
Well, it probably would have warned you that someone was coming
up through your blind spot.
ALAN ALDA (Narration) Car makers are already working on inexpensive
radar systems that will warn you if a vehicle is lurking in
your blind spot.
SANDY PENTLAND The trouble is of course is
if it's beeping all the time, you're going to rip it out first
chance you get, and then of course it's not so smart.
ALAN ALDA So I don't want something that's always telling me about
the blind spot. I want it to know when I need to know it.
SANDY PENTLAND That's right.
ALAN ALDA It's got to be pretty smart to know when I need to know
SANDY PENTLAND Yeah, it's got to know what you're up to
and what it is you're trying to do and only when you're going
to do something that's going to get you in trouble does it
say, oh, watch out. OK, well, this is the half-car we that
we use to develop our Smart Cars. It's a little bit safer
than driving around on Memorial Drive, so why don't we get
in and we'll start it up and go for a drive.
ALAN ALDA (Narration) Media Lab professor Sandy Pentland is collaborating
with Nissan on a project to make cars smarter. This driving
simulator allows people's driving habits to be closely monitored
by computers checking every movement of the steering wheel
or touch of the pedals.
SANDY PENTLAND Don't hit anything.
ALAN ALDA When I put my foot on the accelerator...
ALAN ALDA (Narration) But the simulator isn't exactly like the
SANDY PENTLAND You got a speedometer, it works,
and, everything works here. The one thing that's different
in a simulator is that you can't, first of all you can't look
around and see somebody coming from the side. The other thing
is, you don't actually get the feeling of motion the way you
do when you really move. Oh, you see the red light there?
ALAN ALDA (Narration) And until you get used to it, the steering
is alarmingly light to the touch.
SANDY PENTLAND Turn left...
ALAN ALDA Oh, oh...
ALAN ALDA (Narration) It's OK, it's only the movies! Sandy's goal
is to make a car smart enough to know what its driver is about
to do before you do it - like changing lanes.
ALAN ALDA How can you tell how people are going to change lanes?
How can a computer tell?
SANDY PENTLAND So what we did is,
we had a number of different people drive around in this world
for a fairly long period of time and we told them to do things,
like we told them to pass, like once you... there we go.
ALAN ALDA I think I clipped the front end of that truck.
SANDY PENTLAND Ah, it's OK, it's covered by insurance. And we looked
at what they did just before they did something like pass,
or just before they turned.
ALAN ALDA Now, what did they do?
SANDY PENTLAND Well, it turned
out that everybody had more or less the same pattern of behavior
just before they did something, so for instance, when you
pass, you pull over, sort of look around the guy, slow down
to match speed and only then do you go to pass.
ALAN ALDA I'm a little nauseated here.
SANDY PENTLAND That happens.
ALAN ALDA That happens, huh?
SANDY PENTLAND Yeah. You get used
to it if you do it, but it happens.
ALAN ALDA So what does the computer look for? How does the computer
know I'm going to turn or pass?
SANDY PENTLAND It's looking
for the little motions you do with a steering wheel and with
a brake and accelerator that indicate that you're getting
ready to pass or turn.
ALAN ALDA Now these nutty cowboys on the roads who seem to shoot
over to that lane apparently out of nowhere... are they giving
signals too that a computer could read, or do they just go
like a nut and go to the right?
SANDY PENTLAND Even they will
have a pattern that you can recognize. But probably for cases
like that, the car would have to learn to specialize itself
to the particular person, so there might be a generic way
it works for most people, but that for the cowboys, the car
would have to learn that this is a cowboy, we have to look
for these things and not those things.
ALAN ALDA (Narration) So a smart car would check its blind spot
and warn the driver only if it knew you were about to change
lanes. But it could also keep a watchful eye on you for other
signs of impending trouble.
ALAN ALDA Is this a camera here?
SANDY PENTLAND Yeah, there's a
camera right near where the speedometer is looking back at
you to see if you're looking sleepy or not.
ALAN ALDA How can it tell if I'm sleepy?
SANDY PENTLAND Well, it
watches your face and looks at your blinking. If you're blinking
a lot and your blinking gets longer and longer, that's an
indication that you're getting sleepier and sleepier.
ALAN ALDA So then what? What does it do about that?
Well, there's a couple of things it can do. One thing it can
do is turn on the air conditioning and wake you up or give
you a warning that says get some coffee. Another thing it
can do is actually put a little peppermint spray into the
air conditioning, because peppermint's been shown to wake
ALAN ALDA (Narration) By now I was over my nausea and beginning
to enjoy the ride.
SANDY PENTLAND Want to pass this guy?
ALAN ALDA How fast can you go in this thing?
SANDY PENTLAND It
goes up to about 120.
ALAN ALDA Does it?
SANDY PENTLAND Yeah.
ALAN ALDA Oh, let's see.
SANDY PENTLAND A little scary at that
ALAN ALDA (Narration) But soon the thrill of simulated speed gave
way to curiosity about that camera in the steering wheel.
So it was back to Sandy's lab to learn how a computer can
not only recognize you but even tell how you're feeling, all
by simply gazing at your face.
SANDY PENTLAND So here's a
camera up here, right on top of the computer, and the first
thing we need to do is we need to record a picture of you.
There we are. Got you. Now we need to learn a little bit what
you look like in different situations, so, why don't you look
ALAN ALDA Wait a minute, I have to get into this.
Got to get in... acting comes in here, right? OK. Let's record
ALAN ALDA Tired.
SANDY PENTLAND So now we know when to say go home.
And now we try and look confused.
ALAN ALDA Try and look confused. Try not looking confused.
SANDY PENTLAND OK, what's going on here? OK, good. So now we have
you normal, we have you laughing actually also, we have you
tired and we have you confused.
ALAN ALDA Right.
SANDY PENTLAND So this is the picture we just
took of you to try and learn what your face looks like. So
what the computer does, is it has a model of what a human
face looks like that it's learned by looking at hundreds and
hundreds of faces and using that, it's able to find where
your eye and nose are. It then scales your head and rotates
it into a position that's a standard position so that we can
compare you against other people. So we get this ghostly "chip"
we call it, face chip, which is just the center if your face,
no hair, so if you combed your hair different, it's not going
to get confused or anything like that. And now we can actually
compute the description of what your face looks like as opposed
to anybody else.
ALAN ALDA (Narration) It does this by comparing my face with a
set of standard faces - these are just eight out of the full
panel of one hundred - and assigning a value to each match.
This set of values then describes my face. What's more, these
values can be used to construct a sort of synthetic me from
all those other faces - that's the real me on the left, the
artificial me on the right. Another program maps the movement
of key parts of your face as your expression changes, and
can put a label on how you're feeling. And having your computer
know how you feel is the goal of another Media Lab professor,
ALAN ALDA This brain is really fun to see sitting here on your
desk. Do you refer to that in your work?
ROZ PICARD Well,
we refer to parts of it, we don't refer to the duct tape and
hooks that are actually in that particular model.
ALAN ALDA I'm missing a little duct tape in mine. What... how do
you relate that to what you do?
ROZ PICARD Actually, it's
become very important to notice that, well, most of the work
people are focused on, especially in computer science, focuses
on the cortex, the outer part of the brain. People have largely
ignored the inside of the brain, which includes a bunch of
parts that are called the limbic system. One of the main parts,
for example is this amygdala right here, that's removed for
the moment. This is, functions kind of like the fire alarm
in your house would. This fires and it can hijack the cortical
part of the brain, your reasoning processes, and basically
take over and cause you, for example, to be scared and to
ALAN ALDA And go woo, woo, woo...
ROZ PICARD Well, or whatever,
yeah. And so what we're finding, actually is not only is this
important for just fear and traditional emotion, but it's
also important for decision making and for rational processes.
ALAN ALDA What you're saying is that this little part of the brain
reacts to something on a way that's prior to thought and affects
the thought process. Something that's sort of emotional in
some way. It's an immediate response. So do you get that into
the interaction between me and the computer?
ROZ PICARD Well,
that's what we're trying to do right now. We're trying to
give the computer the ability that humans have to pick up
on emotional cues, and use those on its interaction with you.
JENNIFER HEALY Clench your jaw for a second for me so I can
find the muscle.
ALAN ALDA (Narration) Here we go with the wires again, this time
administered by graduate student Jennifer Healy.
JENNIFER HEALY OK, now you can see this on the blue here, so go ahead
and clench your jaw. See, so that's how I know it's on a good
place. Yeah, great. Good job.
ALAN ALDA Wow, look at that.
ALAN ALDA (Narration) As well as the tension in my jaw, she measures
the clamminess of my hands.
JENNIFER HEALY Yes, if you've
ever noticed when you're nervous, your palms get all sweaty,
and that sweat helps the electrical conductivity.
ALAN ALDA I see.
ALAN ALDA (Narration) Another sensor picks up my heart rate. The
idea of all this is to let the computer know my emotional
state - which unknown to me was about to be aroused.
JENNIFER HEALY Look at that, isn't that cool?
ROZ PICARD He went off
the scale there again.
ALAN ALDA (Narration) Now as anyone who knows me can tell you,
the secret to manipulating my emotions is food.
ALAN ALDA All right, all right. I'm just going to think of a saltine.
I bet you a saltine gets me right back... it went down, see?
ROZ PICARD Now you went down. You just dropped all the way
ALAN ALDA Saltines are not very exciting to me. Now watch this.
Pasta. Oh, red sauce, a little ricotta on the side. Some hot
ROZ PICARD The peppers went up.
ALAN ALDA Did they? I could have told you that.
ROZ PICARD I don't
know about the ricotta though.
ALAN ALDA (Narration) The point of having my computer know my emotions
- through my facial expression or my vital signs - is that
it could adjust itself to my mood, running faster when I'm
bored, trying another tactic if I'm frustrated. It's all part
of making our interactions with computers more like interacting
with other people.
ALAN ALDA Does this meant that at the next stage of this, somebody
will have to be hooked-up like this in order for the computer
to recognize what they're going through?
ROZ PICARD No, eventually
we hope that these sensors will disappear into your clothing
or into the twiddler itself, into the devices that you're
naturally in physical contact with. In fact, people are physically
in contact with computers more than with other human beings,
frequently, so there's a lot more opportunity than you realize
for these sensors to be collecting information.
ALAN ALDA (Narration) And thereby hangs another tale. Stay tuned.
ALAN ALDA This cafe is just around the corner from MIT in Cambridge.
And as you can see, computers are everywhere here. I can plug
my laptop into the Internet while I drink my tea. But if you
think this is the ultimate in computers being everywhere,
this Thad Starmer, who doesn't just take his computer everywhere
he goes, he wears it wherever he goes. You wear it everywhere?
THAD STARNER Well, if I'm at a conference or out on the street,
pretty much so, yes. You get information at the strangest
times, and it helps you manage it during your day.
ALAN ALDA Well, explain that to me. Let's say you're passing somebody
in the hall and they say something to you, interesting. What
happens to that information? It just goes into this little
thing in your eye here?
THAD STARNER Basically. What we have
here is a keyboard...
ALAN ALDA Oh, what's that?
THAD STARNER This is my keyboard. It
uses more than one button at a time to get letters or words.
ALAN ALDA Oh, it's like playing the clarinet or something.
THAD STARNER Yeah, almost exactly like that.
ALAN ALDA When you click those words in, do you see them up on
your eye there?
Yeah, this is a private eye, and what it does...
ALAN ALDA Doesn't it bother you to just see out of one eye all
THAD STARNER No, what's nice about this device is
that it looks like I see words on top of your nose right now,
because I'm sitting here with you. I've typed in like the
dates and who this is for and who you are and where we are
and that then starts pulling up files from my past to remind
me about what I want to talk about.
ALAN ALDA It would be really useful taking a test.
Yeah, actually they let me use it on tests now. If I have
it with me...
ALAN ALDA Get out! What do you mean, they let you use it on tests?
THAD STARNER On tests. I have all my books on line....
ALAN ALDA Naturally.
THAD STARNER But the point that the professors
make, say if I have this with me all the time, then it's part
of me. And I'll have it out in the field, so there's no reason
why I shouldn't be...
ALAN ALDA In a way, your professors already accepted this as an
ALAN ALDA (Narration) And already in local MIT hang-outs, Media
Lab students with their auxiliary brains slung in pouches
across their bodies no longer raise neighborhood eyebrows.
Most wear their private eyes like Thad Starmer, mounted on
eyeglasses. Brad Rhodes is setting his own, jauntier, fashion
trend. Right now, the computers worn by this group of "cyborgs"
as they call themselves, aren't networked together, but already
they're experimenting with wireless links both within the
group and to the Internet. And in fact one of the Media Lab
cyborgs is already online most of the time. Steve Mann's wearable
computer is even equipped with a head-mounted camera, and
he is continually transmitting images from his everyday life
to his very own Web page.
ROZ PICARD Steve is evidently over
at the food trucks wearing the Wear Cam on his head, and we
can see what's he's doing. All right, Steve's in line right
now, he's looking at the menu here and it looks like there's
a few people ahead of him and it's not too long of a wait.
We can click on the next image he sent most recently to see
the menu up closer, and see what our choices are at this particular
ALAN ALDA (Narration) I was about to receive what to the best of
my knowledge is my first e-mail from a cyborg.
So here's a piece of mail from Steve, and he's asking if you
ALAN ALDA You know what's great about this? If all I saw was his
e-mail, all I'd see was I had a choice of whether I wanted
falafel or not. Falafel is nice, I like falafel, but when
I look at the picture and I see this picture of the whole
menu, I can see I can get tabouli if I want. So tell him I
want some tabouli.
ROZ PICARD OK. Some tabouli it is.
ALAN ALDA (Narration) A few weeks later I got to meet Steve Mann
ALAN ALDA Steve, hi.
STEVE MANN Hi.
ALAN ALDA What are you working on?
STEVE MANN It's Alan.
ALAN ALDA Yes, Alan. How can you tell?
STEVE MANN Well, my computer
recognized you. How are you doing?
ALAN ALDA (Narration) Captured by Steve's video camera, my face
was sent back to the main computer at the Media Lab, run through
the face recognition program, and the result flashed back
to Steve's computer. Steve, by the way, has a ham radio license,
since as well as everything else, he's a walking radio transmitter.
STEVE MANN Yeah, this thing on my head is an antenna. It's
just a short whip antenna. There's a copper screen here...
ALAN ALDA Oh, there's a whole lot of screen in there. What is it,
I take it that's not to keep out mosquitoes? What does that
STEVE MANN That's the ground plane for the antenna.
ALAN ALDA (Narration) Steve Mann's been experimenting with personal,
wearable sensing devices for almost a decade now... and most
of them are still stashed away in his dorm room at MIT. This
is his first helmet-mounted rig, dating from the mid-1980's,
and already possessing a period charm all of its own.
STEVE MANN ... over my head like this, and then I could see my screen
here. These are the shoes that I was experimenting with...
ALAN ALDA (Narration) The shoes were able to monitor his footsteps.
STEVE MANN So inside here I had various sensors that produce
an electrical signal...
ALAN ALDA (Narration) But for sheer inventiveness it would be hard
to beat these thermostat-equipped briefs, able to wirelessly
control his dorm-room air conditioner.
ALAN ALDA You never had to get out of bed to change the thermostat
because your underwear was changing the thermostat for you?
STEVE MANN Yeah, yeah.
ALAN ALDA That's unbelievable.
ALAN ALDA (Narration) But if I thought I'd just seen the ultimate
in wearable computers...
NEIL GERSHENFELD Let me ask you to
put that on the ground and rest your foot on it.
ALAN ALDA (Narration) I'm in the lab of MIT professor Neil Gershenfeld.
NEIL GERSHENFELD Let me ask you to shake hands with me. Now
look at the screen here. That's data in your body. Let go,
it'll stop. Shake hands... our bodies are data networked right
ALAN ALDA You mean this thing, the thing under my shoe...
NEIL GERSHENFELD ...is putting data into your body.
ALAN ALDA Into my foot, up my leg, through my arm, down...
NEIL GERSHENFELD Yes. And then when you touch me, it goes through
my body. And it goes up to my shoe to my shoe computer.
ALAN ALDA Oh, come on. You've turned me into a computer here.
NEIL GERSHENFELD A data network.
ALAN ALDA I am the Internet.
NEIL GERSHENFELD You're a bodynet.
It's your... actually, it's very important. If you let me
have that back, I'll show you how it works.
ALAN ALDA I can't believe this.
NEIL GERSHENFELD It's actually
very important. You're not the Internet. You're a bodynet.
ALAN ALDA The worldwide Alan.
NEIL GERSHENFELD You're the person-wide
ALAN ALDA Because it never goes, it doesn't go to anybody else
but who I touch.
NEIL GERSHENFELD It's your bits.
ALAN ALDA My bits, yeah. What's in here, this is...
Let's think about your shoes. First of all, there's plenty
of space there. Nobody's fighting over getting access to your
shoe right now. That'll change, but right now nobody is. Second,
there's power. When you walk, you make watts of power. And
in a couple minutes I'll show you a shoe power source. With
this mechanism, your shoe can talk to your body, turning your
body into a data network. Suddenly, a foot bottom makes more
sense than a laptop. You have your shoes when you get dressed,
rather than carrying technological clutter around, just get
dressed and let your shoes do all this work.
ALAN ALDA I don't even think about it, right? I just put my shoes
on and all of a sudden I'm wired up.
NEIL GERSHENFELD That's
right. Now, for example, when we shake hands you can give
me your business card. If you pick up a telephone, you can
give it phone numbers. If you touch a doorknob you'll authenticate...
ALAN ALDA This can have a really large amount of data stored in
NEIL GERSHENFELD It's a computer. Your body's a network
and it's a computer. This is where we're working on your shoe
power source. When I walk, I'm making up to maybe fifty watts
- a lot of power. And we think we can get at least a watt
out from that. So what we're developing here are the shoe
inserts, so this is... there are a couple of different types.
This is a piezo-polymer laminate, and when you step in this,
the spring stores energy, puts it into stretching this, which
makes a charge this way.
ALAN ALDA (Narration) And sure enough, with each simulated footstep,
the power builds up.
NEIL GERSHENFELD Here we're starting
to become a sneaker manufacturer, working with Nike we made
our own shoe mold to start molding soles, where we're going
to start taking these things, putting them into the soles,
connecting then to the shoe computer I showed you there to
make it something that lives in your shoe and you can forget.
And also one of the other nice implications is, with a shoe
power thing like this, eventually you'll run down the power
if you're sitting, so you need to go for a little walk and
clear your head and power up your shoe.
ALAN ALDA I love this... all this technology comes down to taking
a few steps with your feet.
ALAN ALDA (Narration) OK, it was finally time to step into the
future myself, and see what the cyborgs see. I was about to
experience "augmented reality". All around the lab are little
ALAN ALDA This way to the egress.
ALAN ALDA (Narration) When the camera on my head mount sees one,
it triggers my on-board computer which then displays in my
video goggles a pre-determined message.
ALAN ALDA Do not throw away any of the rolls from this printer.
ALAN ALDA (Narration) That message needn't be just text. In this
case, an electrical panel, I'm treated to a computer animation
of what's inside. And when I look at a video screen, my on-board
computer plays me a video. Eventually all sorts of things
in my environment- say a painting in a museum or a copier
machine in my office - might trigger my wearable computer
to provide me with useful background information. Surrounded
by most of the cyborgs presently on the planet, I wondered
about the next generation of wearable computers.
ALAN ALDA Do you have like, fashion conversations about... Do you
give much thought to different ways that this stuff looks?
JENNIFER HEALY People give it a lot of thought, actually.
There's a lot of debate whether it's good to cover the eye
at all, whether you might want it on your arm, a screen. We
have an undergraduate working on an even better private eye
which would only take up a fraction of this space. It's a
little prism that'll directly show you the screen and yet
be here, and allow your eyes to be seen by the person you're
having a conversation with.
STEVE MANN Hopefully be completely
invisible. One won't be able to tell that we're wearing special
glasses. We won't have these, this display's about six years
old with a big cathode ray tube up on top of it. You know,
they're getting smaller and smaller and eventually we'll have
them built into a regular pair of spectacles and the only
evidence that somebody is wired will be the fact that their
eyes sort of go across in a sort of reading motion. We'll
think that they're possessed by the devil or on the Internet.
LENNY FONER And there's some precedent for this in that it
used to be that when Walkman's first came out, everybody said,
you know, who'd want to wear this thing on their head, these
headphones all the time and then all of a sudden it became
stylish and nobody gives it a second thought, I mean, you
see somebody wearing a Walkman, big deal...
ALAN ALDA (Narration) In fact, fashion designers have already started
to play with the idea of incorporating computers into clothing
and accessories. These sketches were submitted by students
of the French fashion designer Jean Claude Gaultier for a
contest held in his Paris studio late spring. But for today's
cyborgs - that's already the boring stuff. For them, what's
really interesting isn't what's already possible - it's what's
out there just beyond.
back to top
ALAN ALDA (Narration) At the heart of the Media Lab is a large
dark space called the Cube - on my last day there filled with
TOD MACHOVER This is what we think of as a music
videogame, actually. It's called harmonic driving. The idea
here is that it's a piece of music that you actually drive
through, and by the way you drive and the decisions you make
while driving, the piece actually changes. You want to give
it a try?
ALAN ALDA Yeah, if I crash, this piece is over, or what?
Oh, it's a terrible thing that'll happen...
ALAN ALDA Oh all right, let me see. Let me do it...
ALAN ALDA (Narration) This is the prototype of one of several music-making
machines, called hyperinstruments, designed by a team led
by the Media Lab's Tod Machover.
TOD MACHOVER Hey, all right.
We start by building special instruments for the world's best
musicians, like Yo Yo Ma, and Peter Gabriel and orchestras.
For the last five years or so, we've been trying to build
instruments for audiences.
ALAN ALDA Where the audience makes the music.
TOD MACHOVER Where
the audience in the very least experiments with the music,
sometimes performs the music and makes the music. These are
sensors, one, two, three, four, and then under each of your
feet, those are measuring how much electrical current is being
picked up from your body. So as you move, it knows where you
are in this field.
ALAN ALDA This, can I make the same... if I go to the same place,
it does the same thing every time?
TOD MACHOVER The exactly...
ALAN ALDA So it's really like playing an instrument. There's no
TOD MACHOVER No randomization, plus
right in the middle there's supposed to be a sort of cymbal
ALAN ALDA (Narration) This sensor chair, like the other hyperinstruments
here, was being readied for an appearance at New York's prestigious
Lincoln Center Festival, just a few weeks after my visit.
Luckily, it was still just a prototype...
TOD MACHOVER You
broke it! If you want to come over for a second to what we
call the rhythm tree, that's a different idea. That's one
that's designed to have something physical that you hit that
gives you a sound exactly when you hit it.
ALAN ALDA OK.
ALAN ALDA (Narration) The performance Tod Machover and his team
are preparing is called The Brain Opera, and several of his
hyperinstruments - designed to be played by the audience -
look like bits of brain.
TOD MACHOVER This one is, you really
have to work to get the kind of regular rhythm that you got
with the sensor chair. Why don't you give it a try?
ALAN ALDA Right. Hello? Ladies and gentlemen, my fingers never
left my hands.
ALAN ALDA (Narration) Teresa Marrin has invented a hyperinstrument
called the digital baton, which makes different orchestral
sounds depending on where it's pointed.
TERESA MARRIN Electric
guitars, trumpets down here.
ALAN ALDA What's this again?
TERESA MARRIN That adds string sounds.
ALAN ALDA (Narration) An infrared detector follows the tip of the
baton in space.
TERESA MARRIN I like that. Now take it down
there, you'll get some strobe bass.
ALAN ALDA (Narration) Now on this instrument, supposedly the better
I sing, the louder my angelic accompaniment.
That was really good.
ALAN ALDA I think I just wrote Chariots of Fire.
ALAN ALDA (Narration) Three months after my visit to the Media
Lab and Tod Machover and his band of hypermusicians and hyperinstruments
were ready to meet the public. By now the heavenly choir was
accompanied by angelic video images. The idea of The Brain
Opera is that audience member first get to generate their
own hypermusic on what were now fully-tested hyperinstruments.
ALAN ALDA The original was like being inside a skeleton and this
is like being inside the real thing. It really does change
the quality of the music.
ALAN ALDA (Narration) Tod Machover's vision for the Brain Opera
is that the sounds generated here by the audience will be
incorporated into the performance that follows.
ALAN ALDA It's hard to believe that this is going to become a symphony.
In just ten minutes.
ALAN ALDA (Narration) Despite the plan to employ some of the audience's
input in the performance, I can't say I noticed any input
from my own earlier efforts in the lobby. I listened in vain
for my perfectly sustained E-flat. But at one point in the
performance, an even wider audience had their say when Teresa
Marrin used her digital baton to select contributions from
people listening in over the Internet. The Brain Opera, by
stretching our conventional ideas about musical instruments,
about music, about the whole relationship between audience
and performers, struck me as a an interesting metaphor for
the Media Lab itself - exploring a future in which computers
change the very nature of everyday experience.
ALAN ALDA That's all for this edition of Scientific American Frontiers.
See you next time.