Peek Behind the Sneaks: Total Immersion in Video Editing

According to legend, near the dawn of filmmaking in 1896, the Lumière brothers’ short film — The Arrival of a Train at La Ciotat — had Paris audiences running from the theater in terror.  The technology was new, and audiences were unaccustomed to the lifelike quality of film.  It seemed to them the train would break through the screen and plow into the audience.

Today, if you watch a person experience virtual reality (or VR) for the first time, you’ll get a similar set of reactions.  People reach out to touch things that aren’t really there.  They laugh and squeal at delightful images or scream in fright, falling out of their seats at the unexpected.

So, it’s clear that 360° cameras and VR display technologies offer filmmakers an unprecedented tool set for creating engaging, immersive experiences.  It also comes with a new set of challenges. Standard conventions in editing — or even something as simple as framing a shot — no longer apply. That’s why Adobe created CloverVR, a technology concept for editing 360° video from inside a virtual reality interface.

“360° video is a new artistic medium, and the rules are pretty different from traditional video,” says Aaron Hertzmann, a principal research scientist from Adobe Research Labs. “It’s hard to appreciate without experiencing it, but when you have a VR experience with 360° video, you really do feel like you’re in a different place.”

While the immersive quality of the medium wows viewers, it can make it cumbersome for editors used to working with today’s tools. “Traditional video workflows are really mature, so having to reinvent the wheel for editing 360° video can be frustrating for editors,” explains Stephen DiVerdi, research scientist at Adobe.

Stephen’s first experience with virtual reality was through a 1992 arcade game called Dactyl Nightmare.  It was a rudimentary shooting game with blocky graphics, and required gaming equipment the size of a small car.  Although the experience was unique, it was also cumbersome and the content quality was lacking.  It’s part of what fuels his interest in making it easier to create compelling VR content today.

“I’ve been waiting a long time for VR to take off, so I could not be more excited,” he adds.  “But the original promise of VR was that it would be more than a novel content experience.  It has the potential to change the very way we interact with computers.  So what we wanted to do with CloverVR was take a specific workflow — in this case editing 360° video — where we could put the entire interface into VR and give you a what-you-see-is-what-you-get experience.”

As part of the Sneaks presentations at Adobe MAX 2016, Stephen showed how CloverVR tackles that problem by allowing editors to browse, edit, preview, rotate and align their footage without ever having to remove their VR headset.

Editing within the VR environment itself is critical because it’s easy to miss things in a conventional video editor if you are limited to looking at a two-dimensional projection of three-dimensional content. Think of it like looking at a world map. The view in the middle of the map gets smaller and it’s easy to miss details, while the north and south poles get distorted and stretched by the projection.

An example of how 360° video gets distorted in a traditional two dimensional projection.

 “A lot of the things you might do to a 360° video — brightness, color correction, cutting from one scene to another — are just easer to perceive from inside the virtual environment,” Stephen notes.  “It’s easier to understand how big an object feels, how quickly something is moving, or what might cause the viewer discomfort.”

CloverVR also provides tools for editors to quickly browse content. With 360 degrees of action, it can be hard to browse all the raw footage to locate the exact scene or action the editor wants to include.

“Although we included a version of the conventional timeline to assist with the edit, we quickly realized that simply having the timeline wouldn’t work,” notes Cuong Nguyen, Adobe research intern and core contributor to the project.  “It’s too easy to get lost within the timeline, so VR editing requires a unique visualization.”

To assist with this, the CloverVR team included what they call “Little Planet” view — it’s a spherical projection of the content that pops-up in what looks like a globe.  It makes it easy to take in all the action at once, to find the right scene and include it.

“Little Planet” view shows a 360 representation of the video at a glance, making it easier to find the action.

Clover VR also includes “adaptive vignette” to narrow the field of view when scrubbing through content to reduce eyestrain and nausea that can be caused by excessive movement at the periphery of view.

Another common challenge with 360° video is directing the attention of the audience to the desired action in the video. Unlike traditional film, the editor can’t force a particular view on the audience, because you never know where they might be looking; if the action is happening behind them — they might miss it.

Of course, an editor wants their audience to get lost in the story metaphorically, not literally.  That’s why CloverVR has a rotational alignment tool that gives editors the ability to quickly rotate and align the action in footage from two different shots across a cut.  That makes it easy to create visual continuity and allows the viewer to follow the action without suffering from disorientation in a shift from one scene to another.

CloverVR contains other features as well, such as notation and captioning, and was developed in collaboration with real-world editors who are already struggling with the challenges of editing 360° video. This collaboration informed the initial feature set, and continues to influence future development.

“We’ve had some surprises. Things that are trivial for regular video can be hard for VR. For example, if you’re sitting down with your team to review the daily edit, you can’t just point at something on the screen if everybody is wearing a headset,” adds Stephen.

“That’s why we’re thinking about how to create a more social editing experience, so that teams can work collaboratively and share progress with their clients more easily,” Cuong notes.

By placing editors inside the environment of their 360° video content, CloverVR is expanding the boundaries of immersive technologies like virtual reality today, as well as expanding the creative canvas — making 360° video accessible to anyone with a story to tell in the future.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Watch other Sneaks from this year’s MAX here and read other Peek Behind the Sneaks stories here.