Refine Edge tool, Refine Soft Matte effect, and related new features in After Effects CC (12.0)
(For a complete list of what’s new and changed in After Effects CC (12.0), see this page.)
For the basics of the Roto Brush tool in After Effects CS5 and later, see these resources.
Chris and Trish Meyer provide an excellent overview and series of video tutorials about Refine Edge and related features on the Pro Video Coalition website.
introduction to the Refine Edge tool and improved Roto Brush & Refine Edge effect
The Roto Brush effect has been replaced by the new Roto Brush & Refine Edge effect, and a new Refine Edge tool has been added under the Roto Brush tool.
The Refine Edge tool and the associated properties in the Roto Brush & Refine Edge effect provide the ability to improve an existing matte by creating partial transparency for areas of fine detail at the edges of a matte, such as for hair. These new features and their user interface elements are closely related to the Refine Edge features in Photoshop.
basics of using the Refine Edge tool
As with the Roto Brush tool, you use the Refine Edge tool in the Layer panel.
To use the Refine Edge tool, you must already have an alpha channel defined for the layer. You are not restricted to using the Roto Brush tool to define the initial transparency for the layer; you can also use any other method, such as color keying or conventional rotoscoping by drawing masks.
If you are using the Roto Brush tool or conventional rotoscoping to define your initial matte, you should complete this phase of matte creation before using the Refine Edge tool. For example, you should typically not create a Roto Brush stroke and then immediately create a Refine Edge stroke on that same frame; rather, you should make your initial Roto Brush strokes and corrective strokes on your entire clip before going back and refining the edge with the Refine Edge tool.
When you draw strokes with the Refine Edge tool, you draw strokes across or along the edges of a matte. If you draw your first stroke in an area of complete opacity or complete transparency, After Effects will tell you to instead draw your first stroke across or along the edge of a matte. If you draw a stroke with Refine Edge tool on a layer that does not yet have the Roto Brush & Refine Edge effect, the effect is added with Fine-tune Roto Brush Matte disabled, since it is presumed that you are using the tool to refine another kind of matte, such as one created with masks. If you draw a stroke with the Refine Edge tool on a layer on which the Roto Brush tool has already been used, then Fine-tune Roto Brush Matte is enabled.
The edge refinement algorithm relies on the color immediately outside of the stroke on one side being the background color and the color immediately on the other side of the stroke being the foreground color. If your stroke is too broad, such that it covers a range of colors, make it narrower.
You can draw subtractive Refine Edge strokes (named Erase Edge Refinement in the Timeline panel) by Alt-dragging (Windows) or Option-dragging (Mac OS) in the Layer panel; these strokes tell the Roto Brush & Refine Edge effect what regions to not apply additional edge refinement to.
properties of Roto Brush & Refine Edge effect
The Refine Edge features are very closely related to the Refine Edge features in Photoshop, so we tried to make the user interface match that of Photoshop as closely as possible. One outcome of this is that the names and user interface strings for various pieces of the Roto Brush and Refine Matte features needed to change to match Photoshop.
Here is a summary of the changes to match Photoshop, from After Effects CS6 (11.0) to After Effects CC (12.0):
- Smooth became Feather.
- Feather became Contrast (values inverted; different default range).
- Choke became Shift Edge (values inverted).
There are a few other changes in property names and user interface strings that distinguish Roto Brush properties from the new Refine Edge properties, such as the change from Propagation to Roto Brush Propagation and the change from Matte to Roto Brush Matte as the name of the property group in the Roto Brush & Refine Edge effect.
Here are the new properties for Roto Brush & Refine Edge effect:
- Reduce Chatter: Note that this property defaults to 50% in the standalone Refine Hard Matte effect, 0% in the Roto Brush Matte portion of the Roto Brush & Refine Edge effect, 50% in the Fine-tune Refine Edge Matte portion of the Roto Brush & Refine Edge effect, and 50% in the Refine Soft Matte effect.
- Render Refine Edge: Enable this to see the result of the entire effect, or disable it to see the intermediate result from the Roto Brush portion of the effect without the Refine Edge portion.
- Base Refine Edge Radius: Setting this value to something other than 0 adds a uniform band along the entire segmentation boundary that behaves as if you had drawn a stroke with the Refine Edge tool along the entire edge on the base frame. This region can be propagated, corrected, and erased just as if it had been drawn with the Refine Edge tool.
- Fine-tune Roto Brush Matte and Fine-tune Refine Edge Matte: These options enable or disable the property groups for refining the Roto Brush matte and Refine Edge matte. You may, for example, only want to fine-tune the Refine Edge matte if you are using the Refine Edge tool on an edge that was created using masks or color keying, not the Roto Brush tool.
- Smooth: This new property in the Refine Edge Matte group reduces detail & noise in the matte edge to create a smoother outline.
In the standalone Refine Soft Matte effect and in the Refine Edge Matte property group of the Roto Brush & Refine Edge effect, you can choose one of three Chatter Reduction options: Off (the default), More Detailed, or Smoother (Slower).
Keep in mind that the Roto Brush Matte property group and Refine Edge Matte property group affect different regions of the matte on the layer to which the effect is applied. The Refine Edge Matte refinements affect only the area defined by Refine Edge strokes; the Roto Brush Matte refinements affect only the area not defined by Refine Edge strokes.
Refine Soft Matte and Refine Hard Matte effects
The new Refine Soft Matte effect performs the same edge refinements as performed by the new Refine Edge Matte properties in the Roto Brush & Refine Edge effect. The difference is that the Refine Soft Matte effect applies the edge refinements to the entire matte, at a constant width around the entire matte, whereas the edge refinements performed by the Roto Brush & Refine Edge effect are performed within the regions defined by the Refine Edge strokes, which may be of varying widths and along only some parts of the matte edge. Also, since the Refine Soft Matte effect is simply drawing a uniform-width region around the alpha boundary (matte edge) rather than tracking and propagating strokes across time, it can be much faster than using the Refine Edge tool.
The effect that has been called Refine Matte in previous versions of After Effects is now called Refine Hard Matte. The Refine Hard Matte effect performs the same edge refinements as performed by the Fine-tune Roto Brush properties in the Roto Brush & Refine Edge effect.
The properties of the Refine Soft Matte effect are very similar to the properties of the Refine Hard Matte effect, though the underlying algorithms for some identically named things are actually different between the two effects. Specifically, the Reduce Chatter property functions differently between these two effects.
The Refine Soft Matte effect has a few additional properties beyond those in the Refine Hard Matte effect:
- Calculate Edge Details: Computes semi-transparent edges, pulling out fine details within the edge region.
- Additional Edge Radius: The distance in pixels on either side of the matte edge to include in the edge region for finding fine detail and creating semi-transparency.
- View Edge Region: Shows edge region as a yellow band.
behavior with Refine Edge, Roto Brush, and Refine Matte features across versions of After Effects
When using the Save A Copy As CS6 command, the following occurs:
- Instances of the Roto Brush & Refine Edge effect are dropped; i.e., they aren’t saved in the project.
- Instances of the new Refine Hard Matte effect show as missing when the project is opened in After Effects CS6.
- Instances of the Refine Soft Matte effect show as missing when the project is opened in After Effects CS6.
Projects created in previous versions of After Effects (CS5-CS6) with the Roto Brush effect are upgraded to use the new effect when opened in After Effects 12.0.
Projects created in previous versions of After Effects (CS5-CS6) with the Refine Matte effect will retain the old version of the Refine Matte effect, with the old property names and values, when opened in After Effects 12.0. If you apply a new instance of the Refine Hard Matte effect in After Effects 12.0, that instance of the effect will use the new property names and values.
When a project saved from After Effects CS5-CS6 is opened in After Effects 12.0, expressions referring to an instance of the Roto Brush effect are broken. Animation presets and scripts may also need to be updated.
changes in behavior of Feather properties
Refine Hard Matte: For existing After Effects CS6 and earlier instances, no change (because it’s still using v1.0). For existing After Effects 12.0 instances (i.e., projects saved in earlier prerelease builds), it gets reset to 2.0; if you want to use an equivalent feather, multiply the previous value by 2.71. new instances still default to 2.0.
Refine Soft Matte: For existing After Effects 12.0 instances, it gets reset to 0.0; if you want to use an equivalent feather, multiply the previous value by 2.71. New instances still default to 0.0.
Roto Brush & Refine Edge: For existing After Effects CS6 or After Effects 12.0 instances, the existing value gets upgraded automatically (multiplied by 2.71). For new instances, the Roto Brush Matte effect’s Feather property has a new default of 5.0 (approx. the old default of 2.0 times 2.71), and the Refine Matte effect’s Feather property still defaults to 0.0.
Aside from the standalone Refine Hard Matte and Refine Soft Matte effects (that get reset), you shouldn’t see any visually identifiable change in the result. If so, please submit a bug report.
tips and notes for using the Refine Edge tool and related features
When you draw a Refine Edge stroke, the Layer panel automatically switches to Refine Edge X-ray view mode. To turn Refine Edge X-ray view mode on or off, press Alt+X (Windows) or Option+X (Mac OS). This view mode is mutually exclusive with other Show Channels choices. You can also choose Refine Edge X-ray in the Show Channel and Color Management Settings menu at the bottom of the Layer panel.
Note: The Adjust Exposure control in a viewer panel is ignored when using the Refine Edge X-ray view mode.
To avoid unexpected propagation when resolution decreases temporarily, drawing Roto Brush or Refine Edge strokes will turn off Fast Previews adaptive resolution mode.
Drawing Roto Brush or Refine Edge strokes on time-reversed layers (time-reversed or negatively stretched) now draws a banner that says “Roto Brush & Refine Edge strokes can’t be added to a time-reversed layer.”
Note that the edge refinement features of the Roto Brush & Refine Edge effect and Refine Edge effect rely on color information being present even in areas of complete transparency. Most methods of creating transparency in After Effects preserve the color values even when the alpha channel is 0. One exception is the Keylight effect, which obliterates color information in areas of complete transparency. In such cases, you can use the Set Channels effect after the Keylight effect (or similarly behaving effect) to bring the color information back; just use the current layer as the source layer within the effect and set each color source to itself. You could also use Keylight alone if you set its View property to Intermediate Result instead of Final Result.