Greetings from Normandie. This is the third post in my series on Virtual Reality Cinema.
My first post introduced the Google Cardboard headset.
My second post presented the Cardboard app, and showed the mapping of a sphere on to a flat equi-rectangular image. I also detailed the making of a 360-degree dance movie, Blanca Li 360, shot with two camera viewpoints.
This post offers some examples of Camera Systems for shooting VR movies. Note that this is not an exhaustive catalogue.
The first prototype of Jaunt's 360 stereo 3-D camera had 32 lenses and 4 microphones
1. Theta camera
2. Freedom360 system
3. stitching software
4. latest smartphones
5. stereo 360
7. Google Jump
Let's start with a brief Virtual Reality camera glossary:
360 - a VR environment that completely surrounds you
spherical - a 360 environment that places the viewer at the center of an image sphere
cylindrical - a 360 environment that places the viewer at the center of a cylinder -- similar to spherical VR but without the north and south poles. Another variation is a sphere with the south pole missing.
equi-rectangular mapping - a method of creating a flat map from a spherical image, with distortion at the north and south poles.
stereoscopic or stereo or 3-D - a three-dimensional environment, many VR professionals prefer to reserve the term "3-D" for computer graphic imagery
2-D or monoscopic or mono - a two-dimensional VR environment
1. Theta camera
In my previous post, we looked at a continuous 360-degree 2-D movie, Blanca Li 360, composed of two overlapping fish-eye shots facing opposite directions. The two shots were filmed at different times with a Red Dragon camera and 6mm lens, which made it difficult to match actions between the two different takes, but made it simple to rotate the sensor 180 degrees in the same position to get matching viewpoints. This would have been physically impossible with two Dragons shooting simultaneously.
In a video tutorial, the eleVR group demonstrates a similar recording approach with two simultaneous fish-eye views, accomplished with the humble $300 Ricoh Theta M15 camera. eleVR is a group of VR researchers whose members include Vi Hart, Andrea Hawksley, Emily Eifler and Elijah Butterfield. The Ricoh Theta is a small, flat camera with a fish-eye lens on either side, and can record 3 minutes of video at 15 fps with a composite output image of 1920 x 960 pixels. The Theta seems like it could serve as a simple 2-D sketching tool for VR moviemakers.
eleVR members demonstrate the Theta camera and workflow for 2-D 360 movies
2. Freedom360 system
Joergen Geerds, founder and CEO of Freedom360, proposes a professional camera system with six GoPro 4 cameras. This cubic configuration allows you to create 360-degree 2-D movies with a compact, movable camera system. Joergen recommends setting each camera at 1440p resolution in a 4:3 aspect ratio at 60 fps. This will result in a 6K output file after video stitching. He states that doing a master at 6K is a form of "future-proofing," even if it has to be distributed at lower resolutions today.
The Freedom360 VR camera system uses 6 GoPro cameras to create 2-D movies
Joergen is a strong advocate for moving cameras. "Our philosophy is to have a fast, flexible portable that you shoot with on top of a mountain or hanging out of a helicopter. I am a deep believer in moving the camera, so we have a special gyroscope tool for moving the camera; it's like a Steadicam for 360."
Joergen's advice to beginning VR filmmakers is to start with 2-D and concentrate on story. He says that stereo systems require considerable experience and expertise, citing the work of Montreal group Felix and Paul: "They apply a ton of preplanning, lots of effort and plenty of magic sauce!"
My friend Moe Shore at AbelCine quotes a price of $5,400 for a complete Freedom360 system, including six GoPro Hero 4 cameras and required accessories. Moe adds that AbelCine will soon host training sessions to initiate filmmakers in VR.
3. stitching software
Joergen stresses the importance of video-stitching software to create a seamless image sphere composed of the six video streams. He recommends using either Kolor's Autopano Video program or VideoStitch. Kolor and VideoStitch, says Joergen, "both accomplish the same thing with different philosophies." It's worth noting that GoPro recently bought Kolor, indicating the camera manufacturer's resolve to engage with the emerging VR-movie market.
Stitching six video streams together requires a computer with a powerful graphic card. (VideoStitch is specifically designed for a nVidia graphics card.) Powerful graphic cards will combine the images more quickly, which is especially important if you work at higher resolutions. Video stitching is a complex, multiple-step process that involves synchronizing, correcting geometry, blending stitches, and color correcting to match the six camera exposures and hues. A final stitched 6K master is in a 2:1 aspect ratio.
Screenshots from Kolor's video demo of its Autopano Video stitching program.
Images from 4 of the 6 cameras are highlighted in frames 2 to 5.
Joergen notes that a stitched equi-rectangular 6K master can be edited in Adobe Premiere or After Effects. Once finished, the file can be compressed to 4K or lower via handbrake to match the capability of today's smartphones.
4. latest smartphones
The reality is that only the most recent smartphones have the graphic-processing power to deliver the most sophisticated apps and their VR movies. For example, the pioneering vrse app is optimized for the Apple iPhone 5s, 6 and 6 Plus, and Samsung Note 4 and Galaxy S6.
Aron Hjartarson, executive artistic director of Framestore's VR Studio in California, explains the VR smartphone technical requirements. Aron says that "the threshold for an acceptable VR experience, in my opinion, is 1K across for your eyes at 60 fps." Because a smartphone image displays only part of the 360 VR environment, if that partial view is, say, 90 degrees, and you're seeing 1K, that implies the entire 360 image is 4K, or four times greater.
Aron adds: "The smartphone needs video-decoding hardware that can work at 60 fps and also has the GPU [graphics processor unit] power to render that decoded video onto a sphere fast enough so there is no lag. So, the only acceptable experience is with high-end phones that have the GPU power to decode 4K videos."
5. stereo 360
Stereoscopic "3-D" VR is significantly more difficult than monoscopic "2-D." Traditional cinema stereo is done with two cameras positioned like two eyes, often with the help of a half-silvered mirror. In a multi-camera 360 rig, one can imagine doing this when your eyes are in the same position as any two of the cameras. The difficult problem is how to create stereo for the countless other eye positions. Many VR companies have developed their own proprietary workflows and software to create a stereo sphere from a cluster of cameras. There is a veil of secrecy about the different solutions.
At Framestore VR, Aron Hjartarson, has helped to develop a range of solutions for many different camera configurations, including Red Dragons and Epics, Arri Alexas, Freedom360, and GoPro arrays. He acknowledges the difficulty of the 3-D VR problem: "Writing software for rendering stereo panoramas is a tricky proposition," and evokes having to create a stereo pair of virtual cameras for every vertical scan line.
New Wave by Aron Hjartarson and Samir Mallal, available on vrse.com
Aron Hjartarson worked with filmmaker Samir Mallal to create a short VR movie entitled New Wave, which was shot with a custom cluster of GoPro cameras. The three-minute VR movie features a young couple in separate parts of the frame; as you shift your viewpoint, you shift from the woman's to the man's interior monologue.
The beautiful sunset imagery of New Wave has considerable dynamic range, which Aron achieved by having different exposures on different GoPros. He notes that the entire crew had to hide inside or behind the boat to stay out of the frame.
Patrick Meegan is a content producer with Jaunt, which has established itself as one of the leading producers of VR content, and have just opened an office in Los Angeles. Patrick explains that Jaunt has a vertical business model: the company builds cameras and produces movies with a variety of partners, and also distributes the content via apps for Google Cardboard and Oculus Rift headsets.
Jaunt's first prototype camera (pictured at the top of this post) was a sphere with 32 cameras. The second prototype is more boxish, and has been used to make most of Jaunt's VR movies. Patrick says that the prototype went through several iterations, starting with "12 + 2" -- 12 GoPro cameras for "the equator" and two GoPro cameras for the top of the sphere. "The camera housing is a 3-D printed shell."
Left: Jaunt's second prototype 360 stereo camera system - Right: Jaunt camera system on a Steadicam. Note the small camera size.
Jaunt's current camera system adds two cameras for the bottom to create an entire sphere, for a total of 16 cameras. The system is about 10 inches in diameter and weighs 4 pounds. Patrick states that Jaunt currently has "30 plus camera systems and counting. We're constantly building more as we bring in more partners."
Patrick explains that Jaunt creates spherical stereo by using information from any three overlapping cameras. The approach is to first make a depth map of the scene, and then fabricate virtual stereo pairs. This proprietary process involves a lot of computation: "We're using image disparity, optical flow and different computer vision techniques to triangulate the distance of the pixel, and then see the relationship of the pixels around it to understand the depth of the imagery." The stitching and rendering work is done "in the cloud" with a custom front-end interface.
Patrick says "anyone working cameras understands that GoPros have a lot of limitations," noting post issues with synchronization and exposure matching. Jaunt has been working on a new camera system, Neo. "We designed our own camera from the ground up. We use a large sensor, so we have more resolution and also a lot more dynamic range, as well as a global shutter. The cameras are smart and speak to each other, so you have perfect sync and full manual control of exposure, fps, et cetera." Jaunt will release the Neo "at the end of the summer."
7. Google Jump
Google brought VR to the masses with the inexpensive Google Cardboard design. The mammoth company is continuing its effort to grow VR moviemaking with the recently announced Google Jump, a cylindrical stereo camera system with 16 GoPros, which was developed in conjunction with GoPro.
Google Jump system with 16 GoPros
Google will make the system available only to selected VR moviemakers for YouTube distribution, as part of an "early access program," after which it will be sold by GoPro to the general public at an undisclosed price.
vihart.com: Introducing eleVR
elevr.com: The Problem of 3D Spherical Video
Ricoh Theta camera
YouTube: elevr's Complete Theta Tutorial
-- A detailed video about making 360 movies with the Theta
Freedom360 camera system
abelcine.com: Freedom360 system
YouTube: Kolor Autopano Video - My First 360 Video -- A detailed demonstration of stitching a 360 ski movie
YouTube: VideoStitch channel
theverge.com: GoPro made a crazy 16-camera rig for Google Jump
Vi Hart's diagram evoking the difficulty of shooting stereo 360
I thank the following people for sharing their knowledge for this post:
-- Joergen Geerds
founder & CEO
Freedom360, New York
-- Aron Hjartarson
executive creative director
Framestore VR Studio, California
-- Patrick Meegan
Jaunt VR, California
-- Moe Shore
director of advanced imaging
AbelCine, New York
VR on thefilmbook
VR Cinema 1: Google Cardboard
VR Cinema 2: Image Spheres
VR Cinema 3: Cameras
VR Cinema 4: Content
VR Cinema 5: Futures
Carne y Arena PART 1 - VR by Alejandro Iñarritu with Emmanuel Lubezki
Carne y Arena PART 2 - Notes on Design of VR Cinema