Experimenting with Light Fields
We’ve always believed in the power of virtual reality to take you places. That’s why we created Expeditions, to transport people around the world to hundreds of amazing, hard-to-reach or impossible-to-visit places. It’s why we launched Jump, which lets professional creators film beautiful scenes in stereoscopic 360 VR video, and it’s why we’re introducing VR180, a new format for anyone—even those unfamiliar with VR technology—to capture life’s special moments.
But to create the most realistic sense of presence, what we show in VR needs to be as close as possible to what you’d see if you were really there. When you’re actually in a place, the world reacts to you as you move your head around: light bounces off surfaces in different ways and you see things from different perspectives. To help create this more realistic sense of presence in VR, we’ve been experimenting with Light fields.
Light fields are a set of advanced capture, stitching, and rendering algorithms. Much more work needs to be done, but they create still captures that give you an extremely high-quality sense of presence by producing motion parallax and extremely realistic textures and lighting. To demonstrate the potential of this technology, we’re releasing “Welcome to Light Fields,” a free app available on Steam VR for HTC Vive, Oculus Rift, and Windows Mixed Reality headsets. Let’s take a look at how it works.
Capturing and processing a light field
With light fields, nearby objects seem near to you—as you move your head, they appear to shift a lot. Far-away objects shift less and light reflects off objects differently, so you get a strong cue that you’re in a 3D space. And when viewed through a VR headset that supports positional tracking, light fields can enable some truly amazing VR experiences based on footage captured in the real world.This is possible because a light field records all the different rays of light coming into a volume of space. To record them, we modified a GoPro Odyssey Jump camera, bending it into a vertical arc of 16 cameras mounted on a rotating platform.
Right: Light field rendering allows us to synthesize new views of the scene anywhere within the spherical volume by sampling and interpolating the rays of light recorded by the cameras on the rig.
It takes about a minute for the camera rig to swing around and record about a thousand outward-facing viewpoints on a 70cm sphere. This gives us a two-foot wide diameter volume of light rays, which determines the size of the headspace that users have to lean around in to explore the scenes once they are processed. To render views for the headset, rays of light are sampled from the camera positions on the surface of the sphere to construct novel views as seen from inside the sphere to match how the user moves their head. They’re aligned and compressed in a custom dataset file that’s read by special rendering software we’ve implemented as a plug-in for the Unity game engine.
