Video playback on Linux especially with the open-source drivers have long struggled considering the workload and there being no surplus of X.Org / Mesa developers. Open-source Linux GPU drivers are still living with OpenGL 2.1 or less where as the rest of the world is living with OpenGL 3.x and even OpenGL 4.x.
With open-source video playback acceleration there's also the struggle with the GPU vendors providing documentation or code in this area that could potentially compromise their GPU's DRM (Digital Rights Management) capabilities on other platforms. AMD, for example, will not publish code or documentation concerning their UVD (Unified Video Decoder) found on R600 ASICs and later as their DRM is closely tied-in. It also looks like things haven't changed with the third-generation UVD processor found on the just-released Radeon HD 6800 series. AMD and open-source video playback continues to be a matter talked about within our forums in ATI, please release an Open UVD API and recently within ATI Evergreen 3D Code May Soon Go Into Gallium3D, among other threads.
With the open-source ATI Radeon driver there is basically X-Video support and that's it. Within the Gallium3D world for all drivers there was work going on two summers ago of generic GPU video decoding using shaders, but still it's not widely usable and the work was primarily focused with the Nouveau Gallium3D driver. There's also been some work on XvMC within the X.Org state tracker for Gallium3D, but again that's unfinished and it's still only X-Video Motion Compensation, which is only a step above X-Video.
At the beginning of this year there was an AMD developer looking into XvMC for the Radeon R300 Gallium3D driver, but that too is unfinished and the developer (Cooper Yuan) ended up being moved within AMD to work on other unrelated work.
To much surprise, it looks like some of this work may be revived. Christian König has written to mesa-dev talking about his planned adventure in bringing XvMC and VDPAU to the R600 Gallium3D driver, which is the newer Gallium3D driver supporting the ATI Radeon HD 2000/3000/4000/5000 series graphics cards.
In doing so, Christian would need to merge Mesa's pipe-video branch to master, which contains the previous R300 XvMC work, and then hacking away at R600g. It may end up being shader-based rather than taking proper advantage of the ATI UVD, but at least it's a step in the right direction. "And now I simply have to say that I'm deeply impressed by the cool design of your XvMC driver and Galium3D in general. It simply works great!!!"
Not too much information was provided at this point by Christian König, but it sounds promising. For those who don't recognize the name, Christian is the independent developer who was the one that was first to write HDMI audio support to the RadeonHD driver and later ported his HDMI audio to the ATI KMS driver, before any AMD or Novell employee managed to write the code for this feature.
Hopefully this work will come to fruition and be finished off unlike earlier attempts. For those not familiar with the Linux video APIs and extensions, XvMC is capable of offloading the video decoding of motion compensation and iDCT work to the GPU for MPEG-2 video files. VDPAU supports motion compensation, iDCT, VLD, and deblocking, but for many more formats. The supported VDPAU formats include up through MPEG-1, MPEG-2, MPEG-4 ASP, H.264 / AVC, VC-1, and WMV9, but some video hardware is not capable of supporting all formats.