Scott Gershin, the Director of Editorial and Creative Director of the Sound Lab, along with Viktor Phoenix, Sound Supervisor and Senior Technical Sound Designer, joined us to discuss the challenges presented by incorporating sound into traditional entertainment as well as VR and AR experiences;
New Sound Lab at Technicolor Opens to Explore New Frontiers in Audio Entertainment
Sound will be an essential component of future VR and AR experiences and its inclusion will demand new skills and new techniques. To develop these capabilities, Technicolor has opened The Sound Lab at Technicolor, staffed with experts drawn from multiple disciplines across the entertainment industry – including film, music, gaming, advertising, and beyond.
Scott Gershin, the Director of Editorial and Creative Director of the Sound Lab, along with Viktor Phoenix, Sound Supervisor and Senior Technical Sound Designer, joined us to discuss the challenges presented by incorporating sound into traditional entertainment as well as VR and AR experiences; they provide a unique perspective on the exciting possibilities these technologies present.
Gershin: In addition to the film and game projects the crew works on, The Sound Lab was put together to also focus on the new immersive audio industries and technologies, such as 360 videos, VR, AR, and mixed media, etc.
What sets The Sound Lab apart is that we have been able to recruit people who have worked in many different mediums. Our designers and composers come from film, games, commercials, television, and the technology sector. Sound Designer Masanobu Tomita, who spent 15 years on all the Star Trek TV franchises, Supervising Sound Designer Adam Boyd, who was head of central audio at Activision, plus Sound Designer Chris Hegstrom, who spent years at Microsoft working on the Xbox and Halolens projects, are all now members of The Sound Lab. Additionally, Debbie Gonzalez came from the music industry, bringing her musical abilities and experiences to The Sound Lab.
With this team of audio artists and technologists, our team will be able to adapt and grow within these new immersive mediums.
Phoenix: It’s all about how these new forms of entertainment are experienced. Generally they are experienced with headsets and headphones, and that means we can use some of the more advanced audio concepts like 3D audio.
This allows us to put the user into the world we have created. It allows us to move people from having only a passive role in the experience to having an active role. It is really exciting to be able to move sounds around in ways that we were not previously able to do.
Gershin: When film first came out the sound was mono, coming out of a central speaker. Then film started using left and right speakers to enhance the center speaker, enhancing what was happening spatially on the screen.
Then sound evolved to include surround speakers, first from a mono source, to create an immersive experience which gave way to 5.1, 7.1, and Atmos formats (as well as others). Combined with 3D picture, the goal was to make the experience more immersive, to teleport the audience inside the world that the film maker had created.
We are on the brink of experiencing and being entertained in entirely new ways. While it has similarities to film and television, it also has a lot in common with theatre. Instead of being way back in the audience or looking through the “window” – you now have the perspective of being on stage or in the space with the characters.
Sound has always played a leading role in enabling experiences to be more immersive. Now that picture is also becoming immersive, you can put the goggles on and be transported to whole new environments filled with visuals, sounds, and tactile information.
Phoenix: It has been the dream of many science fiction writers of the past century to immerse their reader completely in a story. As a sound designer, I am now able to merge things to advance the narrative aspects of an experience. VR is merging creative aspects and approaches with technical workflows from games and films.
By immersing us in these worlds, 3-D sound allows us to really sense these narrative elements above and behind us in a way that we cannot do with 2-D speaker-based surround sound. It expands our whole field of audio and enhances the visual experience. It allows us to have new levels of spatial storytelling.
Gershin: We are breaking new ground and that’s incredibly exciting. We are creating new workflows, finding new challenges. There are a lot of things that are non-standard, so new approaches need to be created and understood.
We often discuss workflow and approach, how we want to support the story and the experience. Do we want it to be realistic to the nth degree or surreal? How do we want to address spatial opportunities? How will the sound support the story and not become a distraction? Having the mastery to decide what is enough or what needs more. Will the technologies support those artistic ideas?
One of the great things about the illusions in which we participate is that we can break away from reality and go into fantasy or into hyper-reality. We can take you underwater; we can take you to distant lands, or alien places introducing you to an array of characters and technologies. Or we can take you back in time to experience what those before us experienced.
We can indulge in those fantasies and the audio needs to be able to support them so we can stretch reality and stretch physics to take audiences to places they have never been, both visually and audibly. It is a giant blank canvas for us to create on, and that is incredibly exciting.
But now instead of looking through “the looking glass,” we are now on the brink of being there – tricking the brain through sights, sounds, and haptic responses to teleport you into the story book.
Phoenix: It’s a blank canvas on which audio plays such a big role in creating presence. Audio is one of the critical factors for immersion, especially in VR. It enables us to create the feeling of being in a virtual environment. It is a super important pillar of that.
But we have to manage how that affects our workflow. We are not just merging everything from games and films. We are picking the most important elements and workflows that are best for this new medium.
We are pulling in non-linear sound design from games, systemic sound design from interactive media, and great-sounding, heart-pounding audio from films. We are bringing all those together and that affects how we plan, design, and implement experiences.
Gershin: In this new medium, audio starts to play a strategic role. Because you – as the experiencer – are in a spherical environment, you are not sure where to look, or how we are guiding you through the experience.
Audio provides the ability to assist. If somebody on your right says, “Hey, look over here,” or “Hey, come over here,” the natural thing to do is to look at whatever is making that sound. So audio becomes a navigational tool to help you explore that environment.
This means audio has to be involved in pre-production. With our clients, we start discussing how the viewer is going to navigate around, how the pace of the experience is going to ebb and flow, emotional peaks and valleys, what role the different characters play, and how audio is going to be used to support those ideas.
Phoenix: Absolutely. We are finding it increasingly important to be involved early on every project, to understand the creative vision, and to be able to give our counsel and advise on how to use sound in ways that people had not thought of. That is growing more and more important every day.
Gershin: Every industry has its own language. Film has a language, television has a language, commercial/advertising has a language, and gaming has a language. Education is key. Everybody is trying to participate in this new medium and they all come with a different perspective on workflow and every experience is a little different.
VR 360 has many similarities to linear formats such as film and television.
Then there is the stuff in the middle that is computer graphics-based, where the workflow is very similar to animation. This means we can draw on all the work we have done in animation for both film and television.
If we start to have a much more active role, we can start using the languages we have for gaming. If we say “NSF mod” or “Unity and Unreal Engine,” everybody in gaming knows exactly what we are talking about. If we talk to the film people they will understand terms like “Pro Tools,” “Atmos,” and “Dolby.”
We try to draw upon things that are familiar and then teach people how they apply to this new medium. Then, a new language starts to emerge: words like “presence,” “diegetic,” and “non-diegetic.”
New words are emerging in the world of VR. It will have its own vocabulary, but to make our clients feel comfortable we try to go to areas they know. We are trying to guide them and to educate them on how we should work, and on the creative possibilities.
Gershin: For any medium in its infancy there will be a lot of new material that is generated. Some of it will not be very good, but some of it will be amazing.
There will be a lot of stuff that I would call low-hanging fruit. These will be quick to produce and people will go, “Okay, that’s interesting” – but little by little a new generation of storytellers will emerge. Those storytellers will use sound in very new and interesting ways.
Phoenix: I have been dreaming about the future of storytelling ever since I saw 2001: A Space Odyssey as a child. How we can travel through space and time figuratively and literally. We are doing this now. That is what these technologies represent.
I see us using technologies like semantic speech recognition, and AI-driven speech synthesis to create natural input and natural storytelling; so that we get the sense not only of being physically immersed in a story, but of our words and actions having a real impact on the stories we are experiencing, and being able to do that with friends and family no matter where we and they are.
As a dreamer, I see this breaking down so many boundaries.