This is the fifth and final post in my introductory series on Virtual Reality Cinema.
Here I outline possible futures.
4. Stereo, Pano, Ciné + camera & software
5. environments, events & processes
10. recording depth maps
11. empathy & uncanny valley
VR Cinema involves cameras, viewing hardware and distribution channels.
As we saw in my previous posts, 2-D camera systems are becoming widely available, while for now stereo shooting is often done with proprietary systems that involve custom software with some form of secret stitching magic.
I’m hoping that this will change, and that cheap stereo systems will soon be available, with perhaps the possibility of doing the complex processing needed to create good 3-D in the cloud, something that Google might possibly offer with the Jump system developed with GoPro.
Note that VR videos can be created without cameras; I call these software-generated or CGI movies.
As discussed in part 1, the Oculus Rift prototype started the current VR wave, but it was the cheap Google Cardboard — with one million units in the first year — that made VR viewing a mass medium. Cardboard’s brilliant idea is to use the millions of advanced smartphones out there as image processors and screens.
There are many possible business models for VR distribution. We’re seeing independents like Jaunt and vrse creating their own channels of VR content that is headset-agnostic. Samsung, on the other hand, is offering content that can only be accessed through its own Gear headset, and Oculus prototypes.
We can surmise that Facebook acquired Oculus to further its business goal of keeping its customer base involved in its software, and its advertisement. We will learn more about Facebook’s goals next year, with the introduction of the consumer Oculus headset. Facebook might well be getting into the hardware business.
I’m hoping the hardware-agnostic model will win out, so that hundreds of VR movies can all be seen without having to switch headsets. It appears that Google is seeding VR cinema with the Cardboard viewing device, and the Jump camera system, with an eye to help create VR content for YouTube, which is presently not tied to any headset hardware. The YouTube 360 channel is also ensuring a common VR video standard of equirectangular images discussed in part 2.
4. Stereo, Pano, Ciné + camera & software
My friend Michael Naimark is currently a resident artist at Google VR. He recently gave a tech talk at Jaunt entitled Stereo, Pano, Ciné. Michael’s title offers three intersecting categories of VR filmmaking, involving stereo 3-D, 360-degrees and moving images. This taxonomy defines, in rough order of complexity:
Stereo-Ciné – “normal” stereo 3-D movies
Pano-Ciné – 360-degree movie images in 2-D
Pano-Stereo – 360-degree still images in stereo 3-D
Stereo-pano-ciné – 360-degree movies in stereo 3-D
Michael agrees with me that there is one more essential category: Camera vs. Software movies. Camera movies are shot in the real world with cameras, whereas software movies are made with software that creates 3-D virtual models. As we saw in part 3 of this series, 3-D is more complex to render in camera movies than in software movies. Documentaries are, naturally, based on camera movies.
5. environments, events & processes
We looked at the content of VR movies in part 4. I believe that documentary has a special place in the development of VR Cinema. A hundred years ago, the Lumière cameramen traveled across the globe, shooting and projecting documentary images that introduced the public to cinema. It strikes me that cinema history may well repeat itself, and that documentary footage will initially dazzle viewers with a feeling of you-are-there and introduce VR movies to the public.
These documentaries will include “ambient” environments, like Chris Milk’s Clouds Over Sidra; events, like Paul McCartney’s performance of Live and Let Die; and, no doubt, “processes,” like making a meal or building a boat.
I believe VR fiction movies will take more time to mature than documentaries, because those filmmakers have to invent a new storytelling language that incorporates editing and interactivity.
VR movies put you in a space. The power of many VR videos is simply the authentic feeling of being transported to a different place. Michael Naimark was a pioneer of movie maps, and imagines that VR will put “dots” on Google Earth, placing you in a living room, barber shop or meadow recorded in a remote location.
Cicero and other orators used the method of loci -- also called the memory palace-- to remember lengthy speeches. The path through the speech was imagined as a path through a palace, with each room acting as a mnemonic device for a section of the discourse. Part of the narrative challenge of VR Cinema may be to find analogous ways of using space as a metaphor for storytelling.
The viewer’s perception of himself is key to the definition of spatial stories. As discussed in part 4, the default role for the viewer is a disembodied voyeur, which feels both safe and somewhat disquieting. Future VR movies will no doubt imagine the viewer’s POV in different ways.
The importance of audio is often overlooked in discussing VR. Audio is even more immersive than video because we hear the entire environment at once; our ears create a 360-degree perception of the world.
I spoke about audio with Jean-Pascal Beaudoin, the co-founder of Headspace Studio and a frequent collaborator of VR filmmakers Felix and Paul. Jean-Pascal captures 360 degree sound using various techniques. One approach, pictured below, uses a quad stereo binaural microphone comprised of four pairs of ears. Another approach uses a tetrahedral microphone which has four capsules (three for the x, y and z dimensions, and an omni for the sphere).
Jean-Pascal worked on New Wave, a VR movie by Aron Hjartarson and Samir Mallal about a couple drifting apart, wherein the audio is explicitly interactive. As you shift your viewpoint, you shift the audio between the woman’s and the man’s interior monologues.
Jean-Pascal also worked on a VR promo for the movie Wild wherein Reese Whitherspoon’s character sits next to the viewer in a forest without seeing him. Sound, says Jean-Pascal, can be “less literal” than images, and also serve to prompt the viewer to follow a sound — for example, a bird passing overhead. Jean-Pascal quotes Pixar's John Lasseter and states that the challenge of VR is “to make the technology invisible.”
In my first post in this series, I compared VR headsets to Edison’s 1894 Kinetoscopes because both deliver a personal cinematic experience. However, immersive movies can also be projected for groups of people on hemispheres, similar to planetariums, and in other public environments.
The SAT dome in Montreal, a projection of modelized buildings, Luc Courchesne modelized by a Kinect camera
My friend Luc Courchesne is co-director of the Metalab at the Society for Technological Arts in Montreal, and has been working with dome images for the past 20 years. Luc helped design and create a 20-meter-diameter hemisphere in the SAT building that is used for giant projections of camera and software imagery. Luc tells me that he and his SAT colleagues have decided to make all future projects bi-media: designed for both personal headsets and public hemispheres.
Interactivity is a key unsolved problem for public VR movies. A VR ambient environment, like Luc’s beautiful video series of waves lapping on beaches, works effectively in a dome. However, there is no easy crowd counterpart to an individual’s body movements or joystick control.
Full-blown interactivity is an unsolved problem for camera-based VR movies. In many movies, the viewer is stuck in the center of a sphere and can move his head around but cannot change his body’s position. Some movies take the viewer on a pre-defined “ ride,” allowing him to look around as he moves along one or several defined routes. Other movies allow the viewer to travel from the center of one sphere to that of another via a “wormhole.”
The ability to move around freely inside a sphere via a joystick or body-position sensor is easy to achieve in software-based movies, because it involves calculating different trajectories through a pre-defined world of computer-created objects.
On the other hand, it’s highly impractical to shoot the countless possible paths through a space with a camera. This means that for complete freedom of VR movement, the real environment has to be transformed into a computer model of depth maps. In other words, camera-based worlds have to be transformed into software-based worlds to make them fully interactive.
10. recording depth maps
Luc Courchesne cites three current techniques for creating depth maps of the real world that can be used for software-based movies: the Lidar, the projected grid, and conversion of photos to model.
Lidars are currently used by VFX artists to create depth maps of an environment or a set; it works like a radar, calculating distances as proportional to the time before you receive an echo. Projection devices, like Microsoft’s Kinect, involve calculating depths by doing an image analysis of how a projected infrared grid is distorted by nearby objects. Autodesk offers a free Web app, 123D Catch, to simply turn a series of photographs of something into a 3-D model via cloud processing. (Autodesk’s Recap 360 is a more professional tool.)
The Kinect camera, image with infra-red grid, resulting color-coded depth map
11. empathy & the uncanny valley
The future of VR camera movies may therefore involve expanding the definition of a camera. For example, Microsoft’s Kinect device includes both a normal video camera as well as infrared projection and sensing to create 3-D models of the scene shot. This idea of a hybrid video/software camera would allow filmmakers to shoot video and model 3-D space simultaneously. Naturally, this would profoundly change the cinematographer's role.
However, whereas 3-D modeling of objects in the environment may work for VR movies, 3-D modeling of people is another matter entirely. Let’s remember that one of the wonderful aspects of 360 VR movies is their ability to create empathy with faraway people.
By creating 3-D models of people, filmmakers risk placing viewers in the “uncanny valley,” the area where software-created images of people look creepy because they reproduce them without conveying their presence. Paradoxically, a person sometimes feels more present when their image is less realistic ….
The future of VR Cinema is so promising precisely because it is so full of questions:
— How quickly will filmmakers be able to access non-proprietary 3-D camera systems and stitching software?
— Will distribution of VR movies be divided among different headset platforms?
— How quickly will we see a new storytelling language evolve for VR fiction?
— How will viewer's POV be represented in the future?
— How will audio evolve in VR fiction? When will 360-degree audio tools be available?
— Will VR movies be bi-media, distributed for both private and public experiences?
— What forms will interactivity take in future VR projects?
— Will we see the emergence of hybrid cameras and VR movies combining real and 3-D modeled images?
— What will the cinematographer's role be in making these new kind of movies with new kinds of cameras?
— How will future VR movies build on the medium's potential for presence, intimacy and empathy?
Michael Naimark web site
Ocean Beach immersive video
YouTube Creator blog: 360 videos
wikipedia: Method of loci
wikipedia: Microsoft's Kinect camera
123dapp.com: Autodesk's free 123d app
YouTube: 123D Catch for iOS 7
VR on thefilmbook
VR Cinema 1: Google Cardboard
VR Cinema 2: Image Spheres
VR Cinema 3: Cameras
VR Cinema 4: Content
VR Cinema 5: Futures
Carne y Arena PART 1 - VR by Alejandro Iñarritu with Emmanuel Lubezki
Carne y Arena PART 2 - Notes on Design of VR Cinema