Solving Different Problems

The most common apples ‘n oranges-style tech comparison I hear these days takes place between the Adobe Flash Player and pick-your-favorite-dedicated-video-player program. “Why does Flash Player take more CPU to play a video than my favorite video player?” It’s a reasonable question and the answer can be distilled quite simply as, “The Flash Player solves a different problem than your favorite video player.”

This is the core thesis of coworker Tinic’s “On Performance” blog post from some time ago which addressed complaints comparing the performance of AJAX, SVG, and even static HTML to Flash. I’ve wanted to write this post for awhile but always hesitated since previous attempts at technical exposition largely fell on deaf ears. However, with the recent developments that minority browsers have made with supporting the HTML5 video tag, coupled with the recent announcements of such video sites as YouTube and Vimeo offering streams via these technologies, more people are recognizing that the issues surrounding video playback in the browser are in no way unique to the Flash Player. For those who wish to learn, here goes (for those who don’t, feel free to skip straight to the comment form and get your flame on).

Video flow

Here is the general flow that a dedicated media player has to worry about:

video-flow.png

The compressed video data traverses through the video decoder which outputs YUV data to be splatted on the screen.

Here is what the Flash Player needs to do in order to play back video:

flash-flow.png

The Flash Player has to do a little bit more. In addition to decoding the data, it has to convert YUV data to the RGB colorspace and combine the image with other Flash elements. Then it has to cooperate with another application (web browser) to present the video to the user.

So the dedicated media player solves a problem: Generally, it plays linear media files from start to finish while allowing user interaction in the form of random seeking along the timeline. That’s the most basic, trained monkey-type of labor in video playback. At most, the player might handle DVD menus.

Flash Player solves a different problem: It plays linear media files from start to finish while combining the video with a wide array of graphical and interactive elements (buttons, bitmaps, vector graphics, filters), as well as providing network, webcam, and microphone facilities, all programmable via a full-featured scripting language, and all easily accessible via a web browser using a plugin that most of the browsing population already has installed.

“But if I download the media file and play it in my dedicated media player, it takes less CPU.”

Perhaps, but that’s solving a different problem (playing a file using a dedicated, single-purpose application). The Flash Player works to solve the problem of making video accessible via the web browsing environment.

“But minority browsers don’t use as much CPU while playing HTML5 video tag data.”

Are you sure about that? As of this writing, browsers that support the video tag are likely to use a tremendous amount of CPU in order to play video. Why? It’s likely because they have to deal with the exact same performance penalties that Flash does; specifically, that YUV video data needs to be transformed into RGB data to be plopped in the browser window. This is why I thought this might be a good time to write this article– I have been seeing more and more comments in online forums observing that the video tag seems quite likely to peg CPUs.

“But at least with HTML5 video tag data, I can download the video and play it offline using a dedicated player.”

Try to keep up here. That’s not the problem that Flash Player is chartered to solve (same goes for HTML5 video). I know you’re smart enough to download videos and watch them offline, but try to think on a macro level. Try to think of non-tech-savvy people of your acquaintence. (Please tell me you know people who aren’t as tech savvy as you. If you don’t, get to know some in order to maintain proper perspective.) Have you ever tried to tutor such an individual (remotely) in downloading and installing a separate video player? I have and it rarely goes well.

But I digress…

What can be done to improve performance?

Flash Player has an impressive software H.264 decoder, competitive with anything else you’re likely to find out there. But there’s going to be that YUV->RGB conversion penalty, plus blending the other SWF elements on the video, and possibly scaling to a much larger rectangle. We’re pushing the software algorithms as hard as they will go. So what are the hardware options?

Dedicated decoding coprocessor

We’re beginning to see video decoding coprocessors packaged in computers, particularly low-end netbooks. In this situation, the green box labeled [Video decoder] in the diagrams above is replaced by a piece of hardware. Everything else about those diagrams remains the same (including Flash’s need to convert YUV to RGB). Flash Player 10.1 has support for Broadcom’s video decoding coprocessors (according to the release notes, the BCM70012 “LINK” chip). This is currently supported in Windows. Support for these chips in Mac and Linux would be nice but the drivers aren’t ready yet.

Video card acceleration

Assorted video cards from Intel, ATI and nVidia are also able to assist in video decoding as of Flash Player 10.1. Again, see the release notes for more detail on which models and drivers are required. Again, this is presently only available in Windows.

“Why aren’t these card supported in Linux? My free software media player support VA-API, VDPAU, and XvBA (the various hardware acceleration APIs).”

Again, say it with me: Flash Player solves a different problem.

In the above depiction of Flash Player’s workflow, the video card stands in for the green [Video decoder] box. The key point here is that the decoded video frames need to be accessible by the Player which needs to do its thing before the data can be presented to the user. As of this writing, none of these drivers in Linux allow retrieval of the decoded video data. Their counterpart Windows drivers do allow this which is why this feature is supported in Windows.

That’s for Linux. What about Mac? I’m not sure but my Mac colleagues have mentioned something about Apple not making their hardware decoding APIs available to applications (if the APIs exist at all, which I’m not sure they do).

Video card scaling

After Flash Player has decoded a video frame, overlaid everything on top and is ready to present it in fullscreen mode, the hardware can assist in doing the scaling to fullscreen. This is a feature that Flash Player has had for a few releases now. Having to scale an RGB bitmap from 400×300 -> 1920×1200 is something that can easily be pawned off on hardware assuming that the right APIs are present.

“Why doesn’t the Linux Flash Player use Xv for fullscreen scaling?”

We’re not really going through this again, are we? I hope I have cogently presented the reasons between this post and this other one.

Summary

In case it’s unclear, let me re-iterate: Flash Player is not the same as your standalone multimedia program. Flash Player solves a different set of problems than your standalone multimedia program. You might claim that your favorite multimedia program solves video playback problems to your satisfaction. However, the internet at large has unanimously declared that it appreciates the Flash Player’s solution to the video problem.

Regarding the myriad video acceleration APIs in Linux, we are watching developments closely and look forward to implementing support when the drivers support the features conducive to Flash Player’s operational model.