But first, Kevin Abbott sheds some light on how virtual imaging technology can be used to create stunning visuals, to support theatrical performances, and be an art piece all on its own.

(soft upbeat music) (upbeat music) Kevin Abbott, I normally don't say this, but I'm kind of blinded by the light.

This is wonderful.

You are in this world of virtual, so I'm gonna have you basically tell us what you do, according to your title, Director of the Virtual Imaging Technology Lab and more.

- This is our new space here at Western Michigan University that we are just opening up now.

And so this space was part of the former Music and Dance Library, which, with all the changes that happened during COVID, the space became available and my dean had the idea to turn this into a technology space.

And so the idea was to bring together a number of different virtual technologies that can be used to work together.

So we have the LED wall, and then the blue cameras above are the motion capture system.

And then we often pair that with technology like game engines, so that they're able to actually create imagery in real time, meaning that we can have imagery on the screen that reacts in the moment to performers, to patrons, and what they're doing to have a more immersive experience.

- There are cameras in this room all around us.

What about this?

What does that mean and what are they doing?

- [Kevin] So those are our motion capture cameras.

So we have a 25 camera system, so that for our footprint here is actually really nice.

We get really excellent coverage and allows us to do both, I guess I would say traditional motion capture, where you're putting a performer in a suit with the reflective markers.

Again, dance has been our most common use of that.

But then we also do things like track a camera so that we can have a virtual imagery that connects here, or also that we can do interactive things.

So we started to build a demo, which is like an underwater scene, and as you walk close to the screen, some of the creatures that are in the water actually then respond to the fact that you've come closer to them.

So you can do traditional body tracking, but that you can also set up things that you can track independently in the space or build relationships between things.

So the fact that they're in the same footprint together, I mean, it's really nice to come in and just turn everything on and it's all ready to go.

Where in the past we were always having to move stuff around, roll stuff in and roll stuff out, but then also the fact that we can just set up so many unique relationships that we couldn't before because the wall and the tracking system live together in the same space.

- What's your background?

- Our background is actually as a theater designer, came to actually to Western, to study, initially to study acting, and then I got introduced to design.

I really hadn't seen it before, and just absolutely fell head over heels.

And I'm still in love with design.

I didn't know that you could get a job where you just got to make up how things look and either put them on a stage or put them on a screen.

I still absolutely love that process.

And then towards my late years in undergrad, you started to be able to get a desktop computer that would actually run graphics well.

And so started to be able to work more with 3D computer software.

I was making set designs, because I didn't draw as well as I wanted to back then, but it, that world just totally made sense to me.

And then I left theater for a little while.

I spent some time working in video games, and a lot of the work that I had done in the theater, both as a set designer, but particularly as a lighting designer, really carried over into the 3D design world.

- So what is happening on the wall here?

- Well, this is a frozen frame from a dance project that I did in 2019, 2020.

And it's a blending, here you're seeing all virtual characters.

So the whole piece was a blend of video of the dancers, but then also motion captured the dancer and it was expressing the ideas of, as we entered COVID, a lot of us had to, we had this duality where we were communicating in person, but we were then often communicating digitally, virtually, which was different.

And so it was the idea of that sort of virtual persona sort of breaking out.

But then coming this alternate sort of part of yourself.

It was called "Nine Boxes", was the working title because the dancers at the time had to be separated.

The floor was taped out into nine individual squares where they worked.

And so we ended up using that idea to help drive the visual ideas for the piece.

- [Shelley] How is something like this used?

- Um, really used to make art.

So the tool, the tool that's largely behind this is called Unity.

It's primarily used to make video games, but because it's able to generate its imagery on the fly, like in a video game, when you right press the forward button, the WASD, your character immediately reacts.

And so the computer is rendering that scene for you one frame at a time right in front of you.

And so we really like to use it for our production work because it allows us to not have to wait for the computer to render.

We get to see our changes immediately.

When we're doing live work, often a dance piece is just one piece that's part of a larger concert.

And so having this tool allows us to really take advantage of the shorter rehearsal chunks that we have by making changes almost immediately and have them show up and not wait.

So it's a very responsive environment to work in as an artist.

Then the next piece we did after this, which was called Stranger, that one was part of a live show and it was simply projected on the back wall of the theater - [Shelley] 3D modeling and animation.

You're an expert in this.

How does this work?

- [Kevin] That's been around actually for quite a while.

So that was where I started, was creating 3D models, again mostly for video games.

And so one of the reasons that we wanted to put the studio together was to try to give students more experience in those areas.

There's so many industries now that are using 3D modeling, right?

Not only game development, architecture, automotive engineering, it goes on and on and on.

Same with the game engine technology.

A lot of other things other than games are being made with it.

And so we really think it's important for our students to have some experience with that because they can take that experience and go in so many different directions with it.

- You use what's called real time rendering technology.

Talk to me like a fifth grader.

What's that mean?

- So if you watch a movie like let's say Toy Story, right?

All the material in Toy Story, the characters, the animation, the lighting, the textures, all that is prepared.

And then that gets sent to a big floor or building of computers called a render farm.

And because of the high quality, it takes a long time to generate one frame, right?

And a film is 24 frames per second times however long the film is.

So it's thousands and thousands and thousands of frames.

And it just takes a long time to render high quality.

A video game does the same thing.

It takes the lighting of the textures and the models and it renders a frame, but they are hyper optimized to happen immediately so that when you move, it responds in the moment.

You give up quality, so you can't quite have the visual fidelity, the perfect lighting, the realistic lighting that you have in a film.

But boy, every year the game engines get better and better and better at that realism.

And for me, like here, this is not, I'm not trying to make it look realistic, but the play of light on the figures has a realistic feel to it.

It gives them a sense of dimension.

And so that ability to assemble your work and then immediately get to see it in motion is amazing.

It means for me with art, I like surprise.

I like to see things that I haven't seen before.

I like things that challenge me.

And so I think that that's one of the reasons why I tend to want to look to the future.

And these tools to me are very interesting because they've so far proven to be very good at helping me create things that I hope are surprising to people, that pull people through a work, that show them something on stage that they maybe never have seen on stage before.

So I'm really, really interested in new experiences and surprising experiences.

- I'm picturing my computer, I'm picturing this, what's the scale difference, similarity?

How does this even happen?

- I mean the computers are, they're kinda like dogs, right?

Like, you have chihuahuas.

You have great names, right?

And computing can be very much the same, where there's just a wide range of things that those computers serve.

So for us, generally our work is pretty demanding, in terms of computers.

A lot of what we use would fall under the term of even gaming computers, right?

Gaming computers, particularly ones that run Windows are really good at playing video games.

They're really good at rendering.

- [Shelley] There's a niche there, right?

- [Kevin] Yep.

And so they come with faster processors, they come with more expensive video cards.

And these days, the video cards are the parts that really does most of the... - [Shelley] It's a literal card?

- Well, no, these days they're really quite, they're really quite large and can be quite powerful.

So our LED wall isn't super big.

And so we actually run ours off of one pretty powerful computer, but this is a computer that's $15,000, for the single computer to run it with a very, very high end graphics card.

And so it is one of the things that's a challenge for a university, it always has been, and also the arts as well, is that when you start to wade into stuff that's normally reserved for Hollywood, that's normally reserved for the video game industry, those are industries that generate a lot of revenue.

And so the companies that serve them charge quite a bit for their technology.

So for us, that's always a balance that we have to try to meet to what can we afford, how much are we wading into that?

Because, you know, the pieces and parts can all be expensive.

I can remember stories, our first motion capture system was one that had been used on Polar Express, the Tom Hanks movie.

And we bought it used, you know, because that was what we could afford at the time.

And in talking to some of the people who were working with it, they were explaining that when they did their capture sessions with Tom Hanks, they would put all the little, right, little markers, the reflective markers, like a hundred of them on his face.

- On his physical face?

- [Kevin] Right.

And those, at the time, they were like, they were like $5 each, these little reflective markers.

And at the end of the day, they'd washed them all down the sink.

So that was $500 every time they did it, because you really can't put used markers back on Tom Hanks' face because he is supposed to be doing another movie two weeks from now.

And if he breaks out from those, then there's gonna be a lawsuit available.

So that's just sort of a, you know, an example of the range of cost, right?

So for us, we're always trying to make smart choices, hopefully stuff that's enough, like what's in the industry, that it's a good experience for our students, giving us really good quality as we work, but then trying to balance it so that we're not over investing too much and have the ability to take in a number of different technologies.

- Keep on doing your thing and keep on paying it forward.

Appreciate this conversation.

- Yeah, thank you.

(upbeat music)