The Mill's Rama Allen Describes the Potential Behind New Forms of Visual Storytelling

Rama Allen, Executive Creative Director at The Mill

Rama Allen, Executive Creative Director at The Mill, a creative technology and content studio, explores the potential for — and the creative processes behind — a new form of visual storytelling that merges computer gaming and cinematic technologies to create movies where the story line and imagery could change in response to audience input..

As immersive experiences evolve, there’ve been many developments recently in the way we tell stories and communicate experiences. I wanted to ask you about the concept of presence and some of the recent technology developments that are opening up new creative opportunities. What have you seen evolve, and what creative effect does that have on your art?

Allen: First, the really exciting thing about agency -- or the ability for the audience to participate -- is collaboration and empowerment. As a traditional filmmaker, you have dictatorship over the power of attention. You tell the story and bring your audience along. But all their perception and attention is pretty much exactly where you need it to be.

I find agency more attractive and exciting because you’re effectively building a world and creating a set of conditions to elicit a particular emotional response, while leaving a lot to serendipity for the audience to find their own way through, or to manipulate the story however they see fit.

I think that’s very appropriate in this day and age because a lot of our experiences in the physical environment — from automotive to the applications on our little pocket supercomputers — have a responsive nature to them: They bend to our whim.

The one medium where that hasn’t happened yet is cinema. Passive cinema is fantastic, and of course it’s not going to go anywhere. It is one of the truly great art forms.

However, I’m interested in investigating what happens when you begin to introduce agency to the live-action cinema format.

The tools we have access to now, and some of the trends we’re working with, are fusing real-time images with rendered film. That ability allows us to fuse game engine and cinema. When you start to hybridize the language of video games and film, you get a very exciting place for artists to tinker and create.

Can you give me an example of how agency might manifest itself? Who is the target consumer of these experiences: an audience, a gamer, a video user?

Allen: That is a really good question. This is brand new. It’s an interesting thought experiment right now. But at the easiest level, the concept has been proven by my first experiment with our partner, Epic Games, and our client General Motors.

We created a short film called, “The Human Race.” It uses a real-time game engine computer graphics package to create the vehicles in the race to end all races: a human driver in a Camaro against artificial intelligence in an autonomous vehicle.

Right now the level of agency is such that you can customize the vehicle the human is using to race. It is photo-real, it appears grounded and composited, and it operates just like a sophisticated computer graphics (CG) vehicle, but because it is created in real time, you can change variables on the fly, or change them beforehand if you wish.

Do you want to change the color of the car? Do you want to change the year of the Camaro?

These are very simple ways to dip your toe into agency, but it gives the audience collaborative direction.

You can imagine, if the narrative of the story warrants it, giving the audience the opportunity to affect change on objects, characters or environments in the film to reflect their desires or change the story.

That is agency at its simplest, and it works very well. The next thing is to develop stories that use objects created in real time to generate a collaborative moment with the audience.

For example, you can imagine a treasure hunter deep in a mystical temple cave. The treasure hunter is in front of a sacred relic on a plinth. The adventurer must do something — for example hum in tune with the crystal harmonics in the cave — and the audience must hum along. Meanwhile, the software is listening for the audience to be in pitch and harmonizing. When they are, it unlocks and the film moves forward.

So you could have this hybridization of passive cinema going all the way up to a decisive moment where a specific story object requires collaboration from the audience. That inspires a sense of play, of audience involvement. It is exciting. It hasn’t been done yet, but I think that’s a really interesting use case.

You can add things like story branching, which causes the film to become wildly more complicated, but interesting in that the story lines can change.

Tell me about the “The Human Race” project you created. What was involved in bringing this to fruition? I believe you used a few different genres of communication in this.

Allen: We worked with Epic Games, the company that created the Unreal game engine. It is easily the most popular and the most sophisticated game engine there is. It runs most of the triple A games.

Video games have been on a collision course with cinema for 20 years, in language and sophistication and also in the fidelity of the image.

Epic asked us if we could leverage The Mill’s traditional computer graphics prowess to make a live-action film that would use the Unreal engine to create all the compositing and all the visual effects on the fly.

This means that every single frame playing in front of you is being rendered at that moment. Each frame takes 41 milliseconds to render, so it is really fascinating. It is not baked. It is not cooked. You can affect change.

We wrote a script around a car race so we could use The Mill’s Blackbird, a vehicle that can change its wheelbase on the fly to emulate any existing vehicle. It can scan the terrain with LiDAR and is equipped with 360-degree cameras.

It collects an enormous amount of information and gives me as a director something to frame with when I’m shooting. We used it to shoot the car race in the mountains outside Los Angeles.

We also used something we had invented called The Mill Cyclops, which is — in effect — an augmented reality viewer for the director.

As the Blackbird is hurtling down the road, I am in the chase camera car but what I see on my monitor is a live feed of any of vehicles that will be in the finished movie: a 2018 Camaro. The monitor does not show the Blackbird.

Cyclops is a really profound and powerful tool. It even responds to lights and reflections in real time giving me an accurate idea of the beauty of the shot through shadow and reflection. We used these tools and went back to the studio and started to build the film using the Unreal game engine as the platform.

Then we built a configurator, a mini-application that allows you to pick different colors and even change the year of the Camaro in real time on the film.

Many things are going on in that one scenario, but from a production and workflow perspective, it seems as if it is collapsing the concept of preproduction, production and postproduction. Artistically, that must be an extremely powerful development. Is that an accurate observation?

Allen: Absolutely. What it means — from the point of view of writing all the way to delivery — is that everything will be affected by this kind of approach. For one thing, you have to write in anticipation of the technology.

You have to write knowing full well what opportunities and pitfalls you might encounter. If it’s real time, how can I affect change? What limitations do I have? How do I write appropriately to those?

From a production standpoint, we can build a lot of these objects and items well before they are physically available. The Camaro we “filmed” was built in computer graphics before we even started shooting, and that is what I used on set.

How do you collaborate with what must be a non-traditional team of technologists, artists, craftsmen and technicians who have probably worked in separate silos prior to this development?

Allen: That is one of the most exciting parts of this job. A lot of the projects I work on involve these non-traditional intersections of skill sets. With some projects I’ve worked on, there’ve been architects, sculptors and lighting designers.

When these different skillsets come together, you get the serendipity of creative development because people come at a problem based on their background. Those novel interactions are where all these great ideas can really come to fruition.

It’s very much a team sport, just like traditional filmmaking. The creative team is at the table; the coding and technology team is at the table.

Everybody sits together while you are writing to make the best use of the technology while avoiding the “shiny object” syndrome. We’re not making something because it’s a fad; we’re trying to use the technology appropriately.

This happens a lot with virtual reality, too. It is writing in anticipation of the platform. You can’t just do things the way they used to be done. You need to be flexible and nimble. You have to evolve. To make good stuff, you have to think tangentially. You have to work with lots of different kinds of people, accept that the process is going to be rather messy and look for the bits of gold as you go along.

Since you mentioned it, let’s talk about virtual reality. How do you see this merging affecting some of these new forms of storytelling or experience delivery?

Allen: I think creating immersive experiences is an enormously exciting prospect for filmmakers and storytellers, because an essential truth of cinema and story is the desire to have your audience lose themselves in the world you have built.

We want to wander through Gothic hallways and castles or stroll across an alien beach.

The opportunities to help your audiences get lost, as well as eliciting a stronger emotional response, are far too attractive to ignore, and that’s part of the VR medium’s allure for a lot of storytellers.

If you really achieve presence, then the sense of engagement and the emotional responses you get are so much stronger because people feel as if it is their story.

My favorite film of all time is “Blade Runner,” and I’ve seen it probably 50 or 60 times, but when I recall anything about it, I recall watching “Blade Runner.” However, when I recall a successful VR experience, I think about having done it. It was my story. I was in it. VR stimulates a different part of the brain, and it feels richer and more personal.