How will robots and androids relate to human society? Since 2007, playwright/director Oriza Hirata’s robot and android theater productions, which feature state-of-the-art robots on stage, have been performed in Japan and abroad, attracting much attention. Robots and androids that speak and move together with live actors on stage have caused some theatergoers to say, “I thought they were human beings.”

In recent years, with the advent of generative AI and other technologies, the coexistence of technology and humans has attracted even more attention. We asked Hirata, who has depicted the symbiosis of robots and androids with humans from various perspectives, about the relationship between humans and technology and the differences between robots and humans, while looking back on robot theater.

Humanistic Perspective Missing in Robotics Research

– You have been performing with robots and androids since 2007, but could you tell us again how this project came about?

Hirata: It all started when I met Dr. Hiroshi Ishiguro and Dr. Minoru Asada, two leading experts in robotics research, at Osaka University. When I met them for the first time, I had only one question prepared.

“I can create a play with a robot that makes it look like it has abilities it doesn’t have, is that okay?” I asked. Mr. Asada replied, “That’s what I’m looking forward to.”

Mr. Ishiguro told me, “We got into robotics research through Astro Boy, Gundam, and Mazinger Z, but young robotics researchers now only think in terms of existing robot technology, so I want to show them more of the future,” and so I decided to join the project.

– What did you mean when you said, “make it look like it has abilities it doesn’t have”?

Hirata: It’s about how to make a robot seem human. That’s why I gave the robot a sense of direction.

Hirata: First, I prepared a script of about two minutes, entered the movements and dialogue into the robot, and checked its performance. Then I gave the robot a “no” on the spot. I gave instructions to the programmer, such as “Please leave 0.5 seconds between the dialog here and there,” or “Please raise its right hand 30 degrees at this point,” and ran the robot again.

Then the robot’s movements became so natural that about 20 robotics researchers there sighed.

– So the acting instruction worked for the robot.

Hirata: Yes, it did. But that’s not surprising. Roboticists thought that improving a robot’s performance would make it more human, but theater people have been thinking about “how to make people look human” for 2,500 years, so they are very good at making even non-human things look human.

My direction is very specific, and I do not tell the actors to “look happier” but to “move like this” to show their humanity. This method worked well with the programming of the robot.

– What kind of movements specifically make a robot more human-like?

Hirata: We call them “microslips,” which are unconscious and unnecessary movements. For example, when a robot picks up a cup from a desk, it puts its hand on the desk or touches something else before picking it up. Affordance, a field of cognitive psychology, has shown that good actors incorporate a moderate amount of wasteful movements into their performances.

– So you think that the more wasted movement, the more natural it looks.

Hirata: That’s right. When a robot picks up a cup, it grabs the cup in a straight line with a jerky motion, which scares children and the elderly. In robotics, what is required is how fast and accurate the robot is, which means it has good performance, but it does not take into account “how the robot can enter society”.

What Did People Who Saw Robot/Android Theater Find Realistic?

– Since your encounter with Mr. Ishiguro, various types of robot and android theater have emerged.

Hirata: For the last 10 years, Mr. Ishiguro has been creating new robots and asking me, “Would you try something with this?” So I’ve been doing theater with the idea of “What was Mr. Ishiguro’s intention in creating this robot?”

For example, when a skeleton-type android was created, I based the play on Franz Kafka’s novel The Metamorphosis. In The Metamorphosis, the main character wakes up in the morning to find that he has turned into an insect, but instead he has turned into a robot……

The man’s parents can clearly tell from the robot’s movements and words that it is their son. I was able to express this through the robot’s joint movements and dialog, and it worked really well.

– In Mr. Hirata’s plays, robots and androids sometimes seem human or alive. What exactly does the audience perceive as “real”?

Hirata: There are several reasons, but the first is context. For example, if a robot suddenly said “Good morning” in the middle of the night, you would think it was broken. But if a robot says “Good morning” at a very good time in the morning, it feels like it is speaking spontaneously.

Since the beginning of robot theater, a major discovery has been that when the context overlaps speech and movement in this way, the performance is perceived better.

The other is the pause. The structure is similar to that of a microslip, which looks more realistic when it is moderately still and then moves. This is the same for humans and robots, because when you input a lot of information, it takes time to process it and you freeze. Computers are designed to resemble the human brain, so the structure of the two is not that different.

– I see.

Hirata: When we actually toured the U.S. with the android theater, an elderly couple told us after seeing the 30-minute play, “The actor who played the android was very good.” So they thought it was human, and the fact that they didn’t notice it for 30 minutes made a lot of scientific sense.

– That was an impressive response. What were some of the other responses?

Hirata: There was also a strong backlash from the West. In Japan, robots tend to be portrayed as friends, like Astro Boy, Doraemon or Perman, but in the West, many characters fight, like The Terminator or RoboCop. I think people react differently depending on their childhood experiences.

Also, there is an order of magnitude difference in realism between seeing a robot in a movie and seeing a robot on stage talking to a live actor. It reaffirmed for me the power of theater, that people empathize with the possibility of sharing space and being touched.

Social implementation of Robots and Their Risks

– Can the technology that makes robots and androids look like humans be used in fields other than art?

Hirata: Of course. We once conducted an experiment in a shopping mall to socially implement a robot that guides people. For example, two people, a man and a woman, are standing in front of a map and talking about “What shall we eat today?” The robot quickly approaches them. The experiment was to find out from what angle and at what time the robot should speak to them so as not to frighten them.

I wrote a script of about 30 patterns, staged the timing, and spent about two full days filming with motion capture. Then we statistically determined the lines of motion that people would feel comfortable with.

This also has to do with microslips. For example, in conventional robot development, a robot would enter the field of view at the shortest distance and ask, “Are you looking for something?” But that was not enough. It is incomparably more reassuring if the robot enters the human’s field of view at a short distance, disappears for a moment, and then comes back and calls out to the human.

– I hear that very good tissue handlers study such moves to avoid alarming people.

Hirata: That’s what I mean. When this tacit knowledge is analyzed in 3D using motion capture and other methods, an optimal solution can be found. At affordance, we are constantly videotaping the movements of a lacquerware master’s brush and analyzing them in 3D. Then, it becomes clear that the lacquering “tamenuri”(finish with a coat of clear lacquer over a coat of vermilion lacquer) is applied once every three times, and again once every 15 times ……. If we let the machine memorize this information, we can clearly improve the reproducibility of the process.

From now on, I believe there will be any number of reproductions of movements that have been called “artisanal techniques” or “genius”. Dance, for example, is easy to apply. A 3D analysis of the steps of an extremely skilled dancer will show what is skilled and what is not. If we can verbalize this, even an amateur teacher can become skilled in teaching dance to children.

However, as the analysis of the so-called “first class” progresses, there is a danger that others will be weeded out.

Will Technology Rob Humans of Their Value?

– You mentioned risk, but what do you think will be the impact on humans as robots and androids become more familiar to us in the future?

Hirata: I think that, at least temporarily, the value of what only humans can do will increase, while what robots can do will become more efficient and less expensive.

In the theater and movie industry, for example, it is a substitute for actors. If robots and androids can play stuntmen in dangerous action scenes, the cost of insurance and other expenses for actors can be reduced. And in 20 or 30 years, if it goes any further, it may be of value to have all the actors played by humans, rather than the other way around.

– What about risk? People often argue that technology will take away people’s jobs. How can people deal with such concerns?

Hirata: Solidarity and empathy. Labor unions must be properly formed. Without solidarity, workers will be swept away by the convenient spiritualism of those in power, and they will not be able to protect their rights and values.

What is needed, then, is first of all education. It is not about spirituality, but about acquiring skills like literacy. It is necessary to have the ability to understand not only superficial knowledge but also ideology. It is also important to develop a legal system that protects their rights and values.

– I see.

Hirata: And finally, dialog. This is a difficult question to answer at this point. Some people say, “We all have the option of not working,” while others say, “But I want to work.” Then the only way to achieve overall harmony is to have a proper dialog.

When this happens, the role of artists, philosophers, and scientists is very important. Scientists and philosophers have the evidence, the words, and the ability to convey them as a basis for dialog. They have to properly explain “When a robot enters society, there will be these advantages, but there will also be these disadvantages” from the standpoint of natural sciences, social sciences, and humanities. And art is a genre that makes this easy to visualize.

– In Hollywood, there was a strike by the union of actors and screenwriters demanding restrictions on the use of AI in film production, wasn’t there?

Hirata: Hollywood was unprotected at first when CG came in. A lot of extras were put out of business. Given that, it is not surprising that they paid hundreds of millions of dollars to actors who were valuable, and the rest went to CG. The IT society tends to promote inequality.

Hollywood people who have experienced this are sensitive to it, so they go on strike. On the other hand, you could say that Japan is still too innocent.

Oriza Hirata’s Views on the Difference between Robots and Humans

– While some people worry about people losing their jobs to robots, your first robot theater piece, Working Me, depicted the relationship between a robot who loses the desire to do housework and a human. What do you think is the difference between robots or androids and humans?

Hirata: Ultimately, I don’t think there is one. If you cut off an arm and let it bleed, it might be artificial blood. And it is not so easy to say that a person living on artificial blood is not human. Mr. Ishiguro often says, “If you define what a human being is, I can make it just like that, because I am an engineer.”

– I guess you could say that there is no structural difference between a robot and a human. In the theater, even if a robot or an actor makes a puzzled gesture, it is impossible to tell from the outside whether they are really puzzled or thinking about something else.

Hirata: It’s basically a question of what the “heart” is. Mr. Ishiguro and I call it “what used to be called the heart”. I think there is something that used to be called the heart, just as there was something that used to be called God.

If there is one difference between robots and humans at this point, it is that robots and AI are not good at creating things that do not yet exist. Creating something that does not yet exist may be the last area left for humans.

– You have made humans coexist with robots and androids in your theater, but what is your biggest motivation to continue making robot theater?

Hirata: I think it’s trying something for the first time, something I don’t understand. Robot theater and android theater were things that no one had done before.

Next year, I will create an opera with Mr. Kaito Nakahori, a composer who is considered one of the leading figures in contemporary Japanese theater, using the latest 3D sound technology with about 30 speakers arranged not only on a flat surface but also vertically so that the sound can fly diagonally. I have no idea what it is when I look at the score, but it is the first opera of its kind in the world. I always want to do something new like this.

In theater, you never know what is going to happen. I have been doing theater for decades and I still get nervous on the first day. It is interesting because you never know. That’s why I like it so much and why I keep doing it.