Rotoscoping in After Effects
Rotoscoping (or just roto if you’re one of the cool kids hip to the lingo) is the drawing or painting on frames of a movie, using visual elements in the movie as a reference. The most common kind of rotoscoping these days is tracing a path around an object in a movie and using that path as a mask to separate the object from its background. This allows you to work with the object and the background separately, so you can do things like apply different effects to the object than to its background or replace the background entirely.
Rotoscoping in After Effects is mostly a matter of drawing masks, animating the mask path, and then using these masks to define a matte. Many additional tasks and techniques make this job easier, such as using motion tracking on the object before you begin drawing masks, and then using the motion tracking data to make a mask automatically follow the object.
Pete O’Connell has done a terrific job in his Advanced Rotoscoping Techniques for Adobe After Effects training DVD of laying out a highly efficient workflow for rotoscoping in After Effects that is built around motion tracking, decomposition of an object into simple pieces that are easier to draw masks around, and a few other tricks. He’s made a couple of excerpts available on his website, and from these you can get a general sense of the overall workflow and of the high quality of the training on the DVD.
Here are some examples of workflow tips that I picked up from Pete:
- Immediately after beginning to draw a mask, press Alt+Shift+M to turn on keyframing for that mask and set a keyframe. This way, you’ll never again do that thing that all of us have done at least once: edit a mask frame-by-frame for several minutes (or longer!) and then realize that you lost all of your work on previous frames because you forgot to click the stopwatch to make the mask shape animated.
- Draw your masks on a white solid layer with its Video (eyeball) switch off, above the (locked!) footage layer. This way, you run no risk of accidentally moving the footage layer when you manipulate the mask, and you can also much more easily apply tracking data to the mask. (You apply the tracking data to the invisible solid layer that holds the mask.) This also means that you don’t lose your cached RAM preview frames each time you fiddle with the mask, which is a big advantage.
- Turn on the Preserve Constant Vertex Count preference.
- When possible, transform (rotate, scale, move) the whole mask or a subset of the mask vertices instead of moving the vertices individually. This is both for efficiency and to avoid the “chatter” that comes from inconsistent movement across frames.
- Manual motion tracking beats manual rotoscoping. The more effort you spend getting good tracking data for various parts of your scene and object, the less time you’ll spend drawing and tweaking and retweaking masks. (To drive this point home, consider that Pete has an old tutorial still up on Creative COW where he uses these same rotoscoping techniques… but from before After Effects had built-in motion tracking, so his motion tracking was entirely manual. And still the techniques work very well.)
We have a copy of Pete’s DVD here, and several members of the After Effects team have been passing it around and recommending it to each other. Great work, Pete!
On a related note…
Sean Kennedy provides several good tutorials on the SimplyCG website, including some for rotoscoping in After Effects. Sean maintains an index of these tutorials on his website. One of the more useful things that Sean has done is to provide a free script, TrackerViz, which makes tracking motion and applying tracking data to masks a lot easier. You can get TrackerViz and a series of detailed instructions on the SimplyCG website.