A computer is an incredibly powerful means
of creative expression,
but for the most part,
that expression is confined to the screens
of our laptops and mobile phones.
And I'd like to tell you a story about
bringing this power of the computer
to move things around and interact with us
off of the screen and into the physical world
in which we live.
A few years ago, I got a call from
a luxury fashion store called Barneys New York,
and the next thing I knew,
I was designing storefront kinetic sculptures
for their window displays.
This one's called "The Chase."
There are two pairs of shoes,
a man's pair and a woman's pair,
and they play out this slow, tense chase
around the window
in which the man scoots up behind the woman
and gets in her personal space,
and then she moves away.
Each of the shoes has magnets in it,
and there are magnets underneath the table
that move the shoes around.
My friend Andy Cavatorta was building
a robotic harp for Bjork's Biophilia tour
and I wound up building the electronics
and motion control software
to make the harps move and play music.
The harp has four separate pendulums,
and each pendulum has 11 strings,
so the harp swings on its axis and also rotates
in order to play different musical notes,
and the harps are all networked together
so that they can play the right notes
at the right time in the music.
I built an interactive chemistry exhibit
at the Museum of Science and Industry in Chicago,
and this exhibit lets people use physical objects
to grab chemical elements off of the periodic table
and bring them together to cause
chemical reactions to happen.
And the museum noticed that people
were spending a lot of time with this exhibit,
and a researcher from a science education center
in Australia decided to study this exhibit
and try to figure out what was going on.
And she found that the physical objects
that people were using were helping people
understand how to use the exhibit,
and were helping people learn in a social way.
And when you think about
it, this makes a lot of sense,
that using specialized physical objects
would help people use an interface more easily.
I mean, our hands and our minds are optimized
to think about and interact with tangible objects.
Think about which you find easier to use,
a physical keyboard or an onscreen keyboard
like on a phone?
But the thing that struck me
about all of these different projects
is that they really had to be built from scratch,
down to the level of the electronics
and the printed circuit boards and
all the mechanisms all the way up to the software.
I wanted to create something
where we could move objects
under computer control
and create interactions around that idea
without having to go through this process
of building something from scratch
every single time.
So my first attempt at this
was at the MIT Media Lab
with Professor Hiroshi Ishii,
and we built this array of
512 different electromagnets,
and together they were able to move objects around
on top of their surface.
But the problem with this
was that these magnets
cost over 10,000 dollars.
Although each one was pretty small,
altogether they weighed so much
that the table that they were on
started to sag.
So I wanted to build something
where you could have this kind of interaction
on any tabletop surface.
So to explore this idea,
I built an army of small robots,
and each of these robots has
what are called omni wheels.
They're these special wheels
that can move equally easily in all directions,
and when you couple these robots
with a video projector,
you have these physical tools
for interacting with digital information.
So here's an example of what I mean.
This is a video editing application
where all of the controls
for manipulating the video are physical.
So if we want to tweak the color,
we just enter the color mode,
and then we get three different dials
for tweaking the color,
or if we want to adjust the audio,
then we get two different dials
for that, these physical objects.
So here the left and right channel stay in sync,
but if we want to, we can override that
by grabbing both of them at the same time.
So the idea is that we get the speed
and efficiency benefits of using these physical dials
together with the flexibility and versatility
of a system that's designed in software.
And this is a mapping application
for disaster response.
So you have these physical objects
that represent police, fire and rescue,
and a dispatcher can grab them
and place them on the map
to tell those units where to go,
and then the position of the units on the map
gets synced up with the position
of those units in the real world.
This is a video chat application.
It's amazing how much emotion you can convey
with just a few simple movements
of a physical object.
With this interface, we open up
a huge array of possibilities
in between traditional board games
and arcade games,
where the physical possibilities of interaction
make so many different styles of play possible.
But one of the areas that I'm most excited
about using this platform for
is applying it to problems that are difficult
for computers or people to solve alone.
One example of those is protein folding.
So here we have an interface
where we have physical handles onto a protein,
and we can grab those handles
and try to move the protein and
try to fold it in different ways.
And if we move it in a way that
doesn't really make sense
with the underlying molecular simulation,
we get this physical feedback where we can
actually feel these physical handles
pulling back against us.
So feeling what's going on
inside a molecular simulation
is a whole different level of interaction.
So we're just beginning to explore
what's possible when we use software
to control the movement
of objects in our environment.
Maybe this is the computer of the future.
There's no touchscreen.
There's no technology visible at all.
But when we want to have a video chat
or play a game
or lay out the slides to our next TED Talk,
the objects on the table come alive.
Thank you.
(Applause)