XR films, utilizing technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), are on the verge of fundamentally transforming the traditional concept of cinema. These films place the audience at the center of the story, creating a new form of expression that engages all the senses. The ‘Beyond the Frame Festival’ (hereafter referred to as ‘BTFF’) offers the perfect opportunity to experience the cutting edge of this medium.

Now in its fifth year, this festival presents more than just the evolution of film—it introduces the future of entertainment. We spoke with producer Katsutoshi Machiba about the festival’s origins and the challenges of breaking cinematic conventions through this year’s event.

^{“Beyond the Frame Festival 2024 trailer”}

The Birth of the ‘Beyond the Frame Festival’: How XR Films Inspired the International Film Festival

The inspiration for establishing the “Beyond the Frame Festival” dates back to 2016, often referred to as the dawn of XR (Extended Reality). I was previously involved in the film industry, but upon moving to Samsung Electronics Japan, I took on the responsibility for GearVR, a VR device that utilizes mobile phones, marking my first encounter with XR. 

At that time, XR was largely recognized as a medium for gaming, with few exploring its potential for film-like experiences. However, my background in film and my initial experience with the XR film “CIRQUE DU SOLEIL ZARKANA” by Felix & Paul Studios inspired me to explore XR filmmaking. 

This work wasn’t merely a recording of CIRQUE DU SOLEIL’s performance with a VR camera; rather, it involved deconstructing and reassembling the live performance specifically for VR, resulting in a unique and immersive experience. This initial encounter left a profound impact on me and led me to believe that XR could revolutionize traditional filmmaking.

Since then, while researching XR, I began producing various experimental XR films in Japan. A particularly notable project was “Ultraman Zero VR,” created in collaboration with Tsuburaya Productions. This groundbreaking XR film blended XR technology with Japan’s special effects expertise and was showcased at numerous film festivals both in Japan and abroad.

^{Ultraman Zero VR}

At these festivals, including prestigious ones like the Cannes Film Festival and the Busan International Film Festival, I met many XR professionals from around the world. I discovered a global community eager to create XR films and engaged in discussions about how to collaboratively develop the XR industry. 

While major countries were establishing XR divisions at film festivals and producing many XR creators, Japan lagged in nurturing globally competitive talent. A significant gap in Japan was the lack of opportunities to experience XR content. 

To address this, I began a serialized article titled “VR Film Guide” (now “XR Film Guide”) to introduce XR films from around the world and create opportunities for engagement with XR works. Additionally, in collaboration with Tetsuya Ohashi from Cinemaleap, I sought opportunities to establish an XR film festival.

In February 2021, amid the COVID-19 pandemic and with support from the Agency for Cultural Affairs’ Culture and Arts Revenue Enhancement Project, we held the inaugural XR International Film Festival, the Beyond the Frame Festival. The festival’s primary goal was to connect Japanese creators and industry professionals with the global XR industry. 

To achieve this, we showcased highly acclaimed XR works from overseas and organized talk events with key figures from the international XR community. Initially, XR film festivals were unfamiliar in Japan, leading to low attendance. However, recognition has been growing both domestically and internationally, with the number of submissions increasing each year. 

I vividly remember the challenges of preparing and running the festival, as it was an unprecedented event organized by a small, dedicated team.

I myself have been engaging in various activities under the term “XR films” with the desire to create something cinematic using XR. Recently, however, my perspective has started to shift. I’m beginning to realize that the way traditional films, expressed within a frame, and XR films are fundamentally different. 

By using the word “film,” I may have inadvertently reinforced the fixed notion that it’s something you watch while sitting in a chair, which could be limiting the potential of XR film expressions in Japan.

To begin with, I use the term XR to refer to a collective of technologies such as VR, AR, and MR. While we may lump them together under the umbrella term XR, VR, AR, and MR each have entirely distinct characteristics. Even within VR alone, there are various technologies like head tracking, eye tracking, and hand tracking.

To express something using these different technologies in an “XR film,” we need to adopt a mindset and approach that differs from that of traditional films.

Someone once compared creating XR works to space development. While space development involves developing technologies specifically for use in space, many of these technologies have also been adapted for use on Earth. Similarly, the cutting-edge technologies born from XR works may eventually find applications in various real-world scenarios. 

As I’ve been involved in various XR projects, I’ve had more opportunities to engage with such advanced technologies. Many of these technologies seem to have potential not only within XR films but also in everyday life.

Initially, film festivals around the world took up XR as “a new form of film” or “a new possibility for film.” Just as traditional films have used screens in cinemas to show us new worlds, XR films will use XR devices to present new experiences. 

In that sense, XR films could be considered an evolution of traditional films. However, we also need to recognize that XR films are evolving into new forms of expression that go beyond just sight and sound, engaging various other senses as well.

What to Look Forward to in ‘Beyond the Frame Festival 2024’

The Beyond the Frame Festival is celebrating its fifth year this year. I’ve noticed a significant increase in submissions each year, and the festival’s prominence both domestically and internationally is growing. 

Before highlighting some noteworthy works, I’d like to mention three key points about this year’s Beyond the Frame Festival that I, as the festival director, want to share.

First, as an international film festival, we are forming alliances with film festivals across Asia and launching the first project of “XR Festival Asia.” This initiative will select creators from various Asian countries to produce XR works.

Second, we have an impressive panel of judges. Joining us are Mr. Osamu Suzuki, who has created numerous contents over 32 years and retired from broadcasting and scriptwriting on March 31 this year to establish the consumer-focused fund “Startup Factory”; Ms. Hiroko Fujioka, the development leader of WOWOW Lab, which explores new forms of entertainment within WOWOW; and Mr. Michel Reilhac, the founder and curator of the XR section at the Venice International Film Festival.

Third, the XR works themselves. The number of submissions is increasing yearly, with entries from over 20 countries around the world. This year, there is a notable trend towards MR (Mixed Reality) and multiplayer works. MR technology combines the immersive experience of VR with the perception of the real world from AR, allowing users to sense the presence of elements not found in the real environment. 

While VR has traditionally been experienced through VR devices and AR through AR devices, this year’s MR usage is becoming very sophisticated, with seamless transitions between AR and VR expressions, removing the boundaries between reality and otherworldly spaces. MR may well represent the ultimate form of XR.

Additionally, multiplayer works allow multiple users to experience the same space simultaneously. Previously, it was common for a single person to experience one work at a time, making it challenging to accommodate many participants.

However, with the multiplayer approach, multiple people can share the experience of a single work simultaneously, which has the potential to address the primary challenges of participant numbers and profitability in XR films. Expect an increase in multiplayer works from this year to next.

“XR films” differ from “traditional films” in that they are interactive experiences. Unlike traditional films, where you sit in a chair and watch a screen while the story unfolds, XR films require active participation. 

To fully enjoy an XR film, immerse yourself in the world created by the creator and pay attention to various cues and details. Make sure to explore the 360-degree space in every scene. If you encounter a door, open it actively—you’ll likely discover a new narrative world. 

Don’t worry about others’ opinions; find your own way to enjoy the experience. By doing so, you’re likely to find a favorite work that resonates with you.

I will do my best to visit the festival venue every day. Please feel free to discuss XR films with me. I look forward to meeting you all.

I believe that digital twins represent one possibility within XR. It may be that human creativity and technology have not yet fully reached this potential.

Is a digital twin simply an expression of an “actual existence” using digital means? Where does this “actual existence” come from? Do humans all perceive “actual existence” in the same way? Perhaps by creating some form of representation, we are fixing our imagery.

When I produced the work Sen last year, I realized that creating digital representations of things that do not actually exist could also be considered a form of digital twin. 

Many films have historically depicted non-existent objects or phenomena, often predicting the future. What has been depicted in traditional films has often come to pass in reality. This too could be seen as a form of digital twin.

If we consider that the real world exists because humans have reproduced their creativity, it suggests that humans have been recreating their creations in the real world even before the advent of digital technology. 

What is the significance of transforming this into a digital form and recreating it? I would like to discuss this at Gemini Laboratory. 

Is digital twin merely a supplementary method?

Personally, I use the expression method of “XR films” to digitally recreate the combination of cutting-edge technology and human creativity. Recently, at film festivals, it has become more common to express this world view not only within XR devices but also in the actual festival booths. 

As MR technology and direction continue to evolve, the boundaries between reality and digital realms are blurring. When reality and digital converge, I feel that the true meaning of digital twin will become clearer.

When dreams and imagination become reality through technology, entirely new forms and expressions that do not conform to past concepts may emerge.

I look forward to discovering such unprecedented possibilities with the people at Gemini Laboratory.