Visit Your Local PBS Station PBS Home PBS Home Programs A-Z TV Schedules Watch Video Donate Shop PBS Search PBS
SAF Archives  search ask the scientists in the classroom cool science
scientists from previous shows
cool careers in science
ask the scientists

Photo of Prof. Cassell Brain Music

The Brain Opera, created by Tod Machover, allows an audience to generate hypermusic on hyperinstruments and then incorporate the sounds into a live performance. Following the premier of the Scientific American Frontiers Special Inventing the Future, Tod told viewers more about this collaborative event by answering viewers' questions as part of an Ask the Scientists panel. Here are viewers' questions and his answers.

Q What inspired you to invent this marvelous Brain Opera?

A It is always hard to say exactly where inspiration comes from for a particular project. In fact, I always find that one of the signs of a project that has real promise to be interesting is when I find that I can't trace the exact point of inspiration: that usually means that the motivation comes from someplace deep and complex within oneself. Of course that can also mean that one is just plain confused! And it takes a lot of experience to know the difference between the two feelings.

As for the Brain Opera, I do know that I was kind of obstinately obsessed with doing this for 7 or 8 years, often in the face of some pretty weird looks that people gave me when I tried to describe it! I think that one of the deepest inspirations came from a desire and belief I've had -- ever since I was a kid -- that it is possible and important to produce serious, deep, and stimulating artistic experiences that are at the same time fun, entertaining, and interesting for as many people as possible. In some ways, this comes from a desire to "synthesize" the interests and backgrounds of my two parents: my mom is a piano teacher who has spent her career inventing new techniques to teach music creativity to kids; my dad is one of the pioneers in computer graphics. My mom comes from a European high-art, intellectual tradition (she couldn't understand why I liked the Beatles in High School!), and my dad comes from a very popular-culture-oriented family in the Middle West. So part of the inspiration for the Brain Opera came from this deep desire to make a very involving music experience that everyone could take part in.

Another inspiration was to give people a sense of what this new "culture" that is emerging at the MIT Media Lab really feels like. There is something new, I believe, in the way that we -- and especially the youngest students -- are creating totally new disciplines that combine humanism and science, design and theory, thinking and feeling. I wanted the Brain Opera to give the general public a sense of stepping right into the middle of that environment.

Lastly, and most importantly, probably, was that I wanted to write a piece of music that would be beautiful, rich, and would touch people, especially in the way that underneath its complex and ever-changing surface there is a core of music that speaks in a simple, straightforward, and expressive way. I guess I feel that the world is like that....

Q Are you guys gonna' record a CD for release soon?

A It is a bit of a complicated question as to what it means to release a CD of something like the Brain Opera, since every performance is different, and the piece's mutation and evolution is part of its interest. But I am exploring various options right now. One would be, in fact, to put out a well-produced and recording "official version" of the Brain Opera, that would give people my favorite take of it. Another idea would be to put it out as a CD-ROM, with produced music but also the chance to modify this music and even experiment with "virtual" hyperinstruments. A final version might be to put something especially well designed and produced on the Internet, that could be updated and added to continually. It is very possible that we will do some combination of these three, for instance a CD-ROM that has "hooks" to the Internet for updates, interactivity, group play, etc. As usual with my projects, none of this has really been tried before, so I usually end up having to invent the production and distribution medium along with the piece itself. This has its advantages (i.e. artistic control), but BOY does it take time! Another new CD of mine, called "Angels" will be released in February on the Erato/Warner label, so you might want to look for that.

Q Don't hyperinstruments make it pointless to go out to a concert and things of that nature? Don't they require less skill? Does it away from the excitement felt when you can play a real instrument. Does it make it pointless to even try to play a real instrument?

A To be clear, there are two kinds of hyperinstruments. A lot of the hyperinstruments that I've designed are specifically conceived for extremely skilled performers, like Yo-Yo Ma or Peter Gabriel, to give two extremes. These necessitate playing a traditional instrument very well, and use the measurement technology to add to what can normally be done with that instrument. The kinds of hyperinstruments found in the Brain Opera were specifically designed to take away the physical skill needed to play the instrument. I wanted to do that so that anyone could make some kind of music in the Brain Opera, and I think we did a pretty good job of it.

But if you look at the way the three performers play the same instruments (Gesture Wall, Rhythm Tree, Digital Baton, Sensor Chair, etc.), you will notice that practice on these instruments allows you to do some very amazing things with them; i.e., they are not just toys. I do believe that computer technology does and can redefine how we use our physical skills. I would much rather see someone invest twenty years of disciplined and concentrated practice in training their musical imagination and intelligence rather than physically training the fingers. Having been through Juilliard and having seen the number of musicians who become nothing more than glamorized athletes, I know that there IS indeed a difference.

Q What is the sensor chair designed to measure?

A The sensor chair uses a technique called RF-field capacitive sensing to measure the position and movement of the body. An inexpensive circuit is set up to ground the body by touching a piece of metal. With the sensor chair, the metal is on the seat of the chair, and the grounding is done by sitting on the chair. With the Gesture Wall, you stand on a metallic plate, grounding yourself through your feet. Once grounded, your body conducts electricity, which flows out of your body. Metallic sensors are placed around the body - for the Sensor Chair, they are contained in four plastic canisters placed in front of you, plus in two foot positions. These sensors measure the amount of electricity around them. Thus by moving your body closer or farther from the sensors, a very precise measurement is given of where you are. This position measurement can be translated - using clever software - into a measurement of the speed of movement, and also the "kind" of movement, such as smooth, jagged, energetic, etc. A final layer of software decides which movements will produce which sounds, and this software can be changed at any moment during the Brain Opera, depending on the effect that we need.

Q How do the sensors pick up the motion of people to play the music?

A This question is pretty similar to the one about the sensor chair. The trick is that we have to be very clever about what aspect of the body movement to measure (you don't want to collect ALL the data -- that is too much, and not very interesting -- but just the relevant data to indicate what "feeling" the movement has) and how to define in software which movements will produce what music. This is indeed the core of hyperinstruments, and I could write a book about all the different ways that such translation is done. Unfortunately, I have to get to my Tokyo rehearsal this morning, so can't do that right now!

Q How does the digital baton work? How do you know where which instrument is?

A The digital baton is an attempt to combine the physical feeling of touching and squeezing a normal instrument, with the freedom of invisible movement in the air which we tried to achieve with instruments like the Sensor Chair. The Digital Baton has an infrared sensor in the tip which measures where it is pointing. Pressure sensors mounted in its rubber handle measure what each finger is doing individually, and 3 accelerometers also mounted in the handle measure 3D movement of the arm. These many features are combined to measure just about anything people can do by moving their hands and fingers, from large gestures, to delicate finger ones.

Teresa Marrin, who was the main designer of the Digital Baton and plays it in the Brain Opera, did have to practice hard to remember the "feeling" of where in the air different sounds are that she needs to play. We do lots of clever tricks, though: when it is not so very important which sound she plays at a particular moment, we can put hundreds of sounds in the air and organize them into clusters...then she can freely pick a "kind" of sound without worrying about exactly which one she will get. If it is essential that she play a particular sound at a given moment, we can "cheat" and have the computer put ONLY that sound in the air at that time! That's the advantage of building the instruments and writing the software ourselves....

Q Can your music be written out in notes, like sheet music, so that you or other people can play the same piece later?

A Some of the music of the Brain Opera is precisely notated, some is slightly improvised, and others -- especially that involves the incorporation of unpredictable music created by audiences -- is completely improvised. It is not easy to notate HOW one plays instruments such as the Sensor Chair or Digital Baton, so for the time being the "score" of the Brain Opera is a combination of written down material, verbal instructions, and the necessity of our team to directly train any new people who would want to perform the piece. Not ideal, but it will probably stay this way for the foreseeable future.

Q Do think this [the Brain Opera hyperinstruments] will replace our school band one day?

A I don't know....How good is your school band??!! No, honestly, I think that acoustic music is here to stay, but I also believe that sophisticated technology will make its way into our everyday music-making and listening, just as it will into just about everything we do. One of my life goals is to try and help make sure that when this happens, it happens in a way that makes instruments more fun and "human" to play, not mechanical and silly. Not easy, but that's why I try to build these hyperinstruments.

Q Will hyperinstruments eventually be available to the public?

A In the short term, you will see "features" from hyperinstruments popping up in the next generation of electronic instruments, like keyboards or percussion instruments, usually to give these instruments a more intuitive "feel" or improvising ability. Next, special places around the world will provide set-ups where people can go and play with hyperinstruments in a museum or theater setting. A few of these are in the air right now, and you'll start seeing them before the year 2000. I am involved in building an interactive "House of Music" in Vienna, Austria, and there is talk of doing similar things in Seattle and Los Angeles, among other places. Finally, I do hope that instruments such as those we invented for the Brain Opera, will be sold at reasonable prices so that anyone can buy them, practice them, and get really good at them. This is a bit harder to do, since music instrument manufacturers are pretty (no, very!) conservative, and most general tech companies see developing music instruments as a bit flaky. But this is changing, and I predict that within 10 years, hyperinstruments will be available and a pretty common phenomenon.


Scientific American Frontiers
Fall 1990 to Spring 2000
Sponsored by GTE Corporation,
now a part of Verizon Communications Inc.