Welcome to the Thicket

It seems that my Welcome to the Jungle post illustrating the mess of audio APIs in Linux has shaped up to be among the most popular on this blog. A picture is worth a thousand rants and all that. I sometimes consider updating that graph to reflect the current (surely messier) state of things. But I think it drives home its point nicely as is.

Another topic that has arisen is hardware-accelerated video decoding, H.264 video in particular since that is the standard video format. There aren’t nearly as many ways to do this as there are there are to, e.g., push time domain samples out to a kernel-abstracted DAC, but it still amounts to at least a thicket.

linux-video-accel.png

Unfortunately, all of these options have problems right now which make them unsuitable for use in the Linux Flash Player. I plan to expound on this fact in a later post.

42 Responses to Welcome to the Thicket

  1. Apopas says:

    Gnash already supports VA API and you can’t really make your flash work with VDPAU?Always excuses, that’s what Adobe is good for all these years… The community has full 64bit system almost 7 years ago and you’ve only made a almost workable version of flash…

  2. Steven Walter says:

    Whaa whaa whaa, it’s not my fault that Flash Player sucks. Clearly this is Linux’s fault, even though Mozilla and Chrome have far superior HTML5 video players on Linux.

  3. Jeffrey W. Baker says:

    You keep pushing this point but the larger picture is that EVERY media player on Linux has superior performance to Flash Player. VLC is better, mplayer is better, every player Linux has is WAY WAY WAY better than Flash Player. So obviously there exists some way to play video — even h.264 video — which is sufficient for mplayer and vlc but not for Flash.What you really mean to say is that there’s no way for your closed, proprietary application to leverage Linux video APIs because you can’t let the distributions configure and package it. All the open source players work fine and your player barely works, so the split between open and closed is really the only thing that explains the performance gap.

  4. Please ignore the trolls. Some of the more pragmatic people appreciate the work you do. Additionally, we appreciate that you let us know. Yes the current stuff sucks but look at it compared to 2 years ago… 5 years ago… We’re working on it 🙂

  5. Josh Yost says:

    I find all too often that instead of directly addressing the problem we tend to blame whatever we dont like. This thoughtful post does not seem to be of that nature, however responses that closed vs open source cause the issue is the case in point.

  6. Penguis says:

    Too little too late. Adobe has neglected users of non Windows operating systems for far too long. Major video sites such as YouTube, Vimeo and Dailymotion are already gradually switching to html5 video because of this. Soon we will be free at last.

  7. greg says:

    Yes, this feels a bit like the audio API chaos again. I’ve been trying to push VDPAU — it’s a nice API, well specified and documented, has good application support already and at least one well-working implementation.VA-API existence only seems to be a NIH phenomenon.CrystalHD’s odd /dev interface should be ignored, just like XvBA, which doesn’t have any documentation (!) is considered unsupported by AMD…I have *no idea* why Broadcom didn’t use VDPAU or VA-API. It doesn’t make sense, both APIs are suitable.

  8. Sean Upton says:

    Slightly OT, but: Adobe had MPEG-LA license for h.264 for Flash Player, but… I think it would help the case of simplification of the stack to abstract and access h.264 decoding if the folks developing the layers below yours had a royalty-free patent license to h.264. If MPEG-LA was smart, they would recognize the ubiquity and revenue value of offering all free/open-source applications and libraries a royalty-free license, and charged everyone else as they do now. Google, Adobe, and Apple would be smart to collectively try to pressure such a result.

  9. graph analyzer says:

    Looking at it the, VA API seems the most reasonable approach. CrystalHD is in staging right now for a reason, before it can go to mainline, it should become a VA-API backend.This might kill the ‘same API on all plattforms’ idea of CrystalHD, but those chips are far too uncommon to optimize applications for them anyway.With VA-API you’ll get them for free in the future.(I know that VDPAU was intended to be what VA-API is now, but it would be too much for the linux community to agree on a vendor proposed interface 😉 )

  10. mike russell says:

    just use vdpau and be done with it. more excuses from adobe again. mplayer works fine with vdpau, vlc works fine with vdpau…. and its made by one of your closed source mates, NVIDIA.

  11. Chris Lees says:

    Wow, VDPAU causes problems? That’s the first I’ve heard of it, considering that almost the entire Linux world has moved to support it.Wouldn’t it be better to support VDPAU which works with more than 50% of the Linux install-base (it’s not just Nvidia, remember), and then fall back to the current situation with no acceleration if VDPAU isn’t present?

  12. VoltageX says:

    Chris, VPDAU is great unless you’re using a chipset that’s only supported by VAAPI.

  13. Steve Antal says:

    Stop the FUD, stop blaming Linux APIs for everything, you already have an unaccelerated method, so pick an accelerated one that is most used, document it and stop picking lame excuses for your own inability to develop a flash player that actually works on most[linux] systems. But do this fast cause HTML5 is already crushing flash, top 10 requests on Youtube was to have an alternative HTML5 video player.So either you actually write some code instead of drawing misleading diagrams and writing blog posts or flash will go down the drain, which of course it would deserve cause this should have been resolved years ago.

  14. So basically your myriad of choices is VDPAU or VA-API? I’m not trying to troll, but this post seems pretty content free.

  15. xxx says:

    @Penguis:VA-API existence only seems to be a NIH phenomenon.You got your facts wrong, it’s the other way around: VA-API was first, then Nvidia came up with VDPAU, a classic NIH. It took Linux players over in a storm thanks to the large user base with proprietary nvidia drivers.

  16. xxx says:

    Of course, you could also use a higher-level library designed to abstract these things from you, such as GStreamer. But then you wouldn’t have anything to bitch about, would you?

  17. wds says:

    So, extrapolating from how long it took you to come up with this post since the last video hardware decoding post I assume it’ll be around April until you tell us why VPDAU isn’t good enough? ;-)Kinda sucks not having the punchline really.

  18. Anonymous says:

    I imagine it has more to do with being able to ensure encrypted video paths for protected content than the effectiveness or widespreadness of any of the apis… the fact remains that they are all accessible APIs and hence could be used as points to ‘steal’ content by placing a recorder that pretends to be a decoder/renderer using a vid decode API in place of the actual vid card.

  19. Matt says:

    Too little too late.By this time next year, no one’s going to care about Flash anymore because we’ll all be using HTML5 video.How much do you want to bet Flash will still be 32bit, with no video acceleration on linux, and 5 times slower than under IE/Windows?

  20. AnonymousCoward says:

    The issue here is not with Linux, it’s with your implementation of Flash.Gnash has proven that you can efficiently run Flash, on all sorts of different platforms.Let’s hope HTML 5 + h264 is the end of Flash. Good riddance.

  21. I wouldn’t take much notice of the ‘Adobe sucks’ people because Adobe clearly do plan to support Flash and AIR on this platform, the first to get 64bit support.I wouldn’t take much notice of the ‘HTML5 ftw’ people either, as FireFox, the most popular FOSS browser isn’t going to support h.264, making the VIDEO tag a bit of a codec nightmare, and YouTube wont transcode their entire library.I do appreciate the insights into the problems the Linux team at Adobe has though. I’m sure you’re working with the right people to get things sorted out and cleared up where possible – after all, full screen video is basically a solved problem 🙂

  22. LGB says:

    Hmmm I still think that virtually every player on Linux has superior performance compared to flash. So I usually download a video even from youtube then play it from command line since flash player simply lags all the time and uses all my CPU. By contract mplayer for example plays them often with almost zero CPU usage (or at least no more than 10). So it’s a proof that it can be done right even on Linux, so I can’t see that Linux can be blamed for it: other players can do it just well!

  23. Marcus says:

    It’s weird how projects like mplayer and xbmc has hardware acceleration working just fine, but a large company like adobe can’t make it happen?

  24. Tuxie says:

    Use VA-API and get both VDPAU and XvBA support for free! Forget about CrystalHD, it will be supported by VA-API also, eventually.Both VDPAU and XvBA are vendor-specific APIs and shouldn’t be used directly.VA-API is the future. Directly coding to VDPAU/XvBA was a hack to have *something* while VA-API matured.

  25. Artem S. Tashkinov says:

    I don’t heard of a single problem of VDPAU in Linux.So, again, what’s your post about?

  26. RealNC says:

    I’ve written this before somewhere in your blog, and I’ll write it again:VLC/MPlayer/whatever_except_flash can play fullscreen 720p video smoothly without any form of acceleration using the plain X11 renderer. So why can’t Flash? If you claim you’re dealing with Linux multimedia since 1999 then you’re doing something wrong for sure.You don’t believe me? Get the mp4 of any 720p YouTube video and play it in mplayer using the X11 back-end. Then try it with Flash and see the suckage.The lack of video acceleration is *not* the problem. Flash is. Video accel is there to reduce CPU load, it is *not* there to allow for smooth fullscreen playback on otherwise fast systems. Flash can’t play a video smoothly on my dual core 3.3GHz machine while every other application can do so without problems and without any form of video acceleration. This is an embarrassment really.

  27. blabber says:

    you can admit it, adobe: you simply 5uck, there’s just no excuse :)p.s. fortunately we won’t need your flash crap anymore soon, thanks html5!

  28. thinkliberty says:

    So build a better one, if you don’t like what’s currently available.Or better yet, improve the existing open source API so it plays nice with your flash player.Doing something is always harding than crying like a little bitch, that you don’t like the work that other people have done.

  29. Fabio says:

    Please add at least support for Xv. While it can’t fully accelerate H.264 decoding it’s supported on most drivers (and often there are no alternatives), at least on:all intel cards since i810 (with open source driver); since i830 it offers tear free rendering. Other decoding APIs will maybe be supported (if and when a gallium driver will be ready). Intel only has open source drivers;all radeon cards since R100 (with open source driver), with tear free rendering (see Feature Matrix for Free Radeon Drivers). Other decoding APIs will not be supported on R100 and R200 (since gallium3D won’t work here), maybe they will be supported on R300 and newer (if and when a gallium driver will be ready). ATI only has open source drivers for cards older than R600;the nvidia nouveau driver (see Feature Matrix for Nouveau Drivers). I don’t know if it supports tear free rendering (I don’t own nvidia cards);It could at least give the following advantages:faster decoding;tear free rendering (watching video with tear is really annoying for me, I always end up downloading the video and watching it with VLC);Supporting more advanced APIs would be nice, but al least please support Xv since many cards only support this.

  30. Eric G says:

    And what is your excuse for Mac OS X? Watching a Flash AD (which I don’t want in the first place) pins 2 cores at 50%…Instead of complaining, why not contribute?

  31. Mika says:

    Mike,I’d appreciate any further insight you may have into the APIs you mentioned and the feasibility of using them for Flash. Looking at the chart itself doesn’t remind me of a thicket — above all the software stack appears to offer a couple of levels of abstraction with the possibility of skipping past some abstraction levels if you wish. That kind of abstraction is present in a lot of different software stacks and is not generally considered a problem.In fact, the chart offers an obvious solution: Use VA-API. Assuming the abstraction doesn’t hurt performance significantly, using VA-API brings support for VDPAU and XvBA automatically. This doesn’t help with CrystalHD, at least for now, but as others have pointed out, it just might become another implementation below the VA-API abstraction in the future. Also, as far as I know, CrystalHD isn’t so widely used that the difficulty of supporting it should defer support for all the other implementations that are available through a common abstraction API.Or is there something I don’t know? Is performance inadequate if you use VA-API and the actual implementation under that is VDPAU or XvBA? Are VA-API or its VDPAU or XvBA wrappers too immature? I imagine there must be a reason why this seemingly obvious solution doesn’t strike you as a good one.I’d be keen to hear your insight into that.

  32. Ugly American says:

    It must be frustrating not to have the sourcecode to the ATI & nVidia proprietary drivers to see what’s wrong.:)Flash is going to be run over by HTML5 anyway.When you won’t share your ball, people go and get their own.

  33. someone says:

    I think you forgot OpenMAX (all 3 variants of it). Sure it’s not really popular, but it does exist.

  34. After reading these comments I am really glad not having to deal with the (largely anonymously posting) Linux community…Hat’s off to you Mike – damned if you do and damned if you don’t.

  35. n says:

    v4l2 has VID_TYPE_MPEG_DECODER but i don’t know this is usable.

  36. Branislav Kirilov says:

    Don’t worry guys, flash player sucks hard in windows too. I haven’t seen any other software less optimized for performance than flash. I wish it is gone from the web for good. It hogs CPU like hell. And that is on ANY computer. They are doing directx version. After so many years and in version 10? WTF!?!? were you sleeping all these years. Flash always was a CPU killer, and that’s way I avoid if i can…

  37. What’s wrong with using FFmpeg to access all the lower layers? That’s what all other players are doing.There’s also the OpenMAX IL alternative, that’s already used by all major codec vendors in the embedded world, but would require extra development in the Linux desktop.

  38. Kevin Newman says:

    Can you do one of these for all the versions of Windows (XP, Vista, 7 – x86 and x64 – nvidia/ati/Broadcom/Intel/everyone else)?Also, can you do one for a single Linux platform, rather than lumping all of them together – maybe Android alone, or Ubuntu.

  39. Anonymoose says:

    Your thicket looks like a few blades of grass to me.. and yet you act like you need a scythe handed down from the gods to chop through it.At least your audio diagram was jumbled-looking (although one could have made a similarly complex diagram for Windows audio API’s where Flash conveniently works).

  40. Nigel Jones says:

    I appreciate you explaining what’s happening around acceleration on linux.However it’s basically unusable today. It needs serious focus if that platform is to be supported.I’ll echo the other comment above that performance is poor on linux too. Every time I watch a flash video on home pc (a few years old intel 820d running windows) one core spins, the cooling fans rev up and it’s generally an amazingly inefficient experience.Quite simply flash is not appropriate for sd or hd video streaming. end of.

  41. Chris Brunhaver says:

    Now that 10.1 it out, with all of the work on hardware acceleration on the Windows, Mac and Adroid platforms, do you see Adobe taking a second look at VA-API and re-evaluating it’s stance on true 64 bit linux support? If the business case is still in doubt, how about making this a paid-for download (using a vehicle like the Ubuntu Software Center)?

  42. Tobin Davis says:

    Here’s an interesting factoid I learned while doing Poulsbo testing at Intel (2007-8). Running flash in Windows XP in a virtual environment with Linux as the host OS was much faster than native flash. This is still true today. And this is without any special video drivers for the VM (stock Windows Vesa driver). So why is it the API’s fault? If the Linux API’s are so bad, why not work with the developers to fix them. Complaining will not help.