I'm currently setting up a gentoo htpc, based on a 780G/4850e combo.
I did get xvba to work (libva/mplayer), XBMC is finally mostly working...
But so far I've failed all attempts at playing DTS/AC3/DD Audio using the hdmi out as passthrough.
Music, menu sounds and other movies play well.
XBMC recognized that it should use passthrough, but sends a 2-channel-48kHz PCM signal to the receiver instead, which probably contains the AC3/DTS data, as the resulting noise is just a wild stuttering.
I'm quite (but never 100%) positive, that I've tried all possible/reasonable audio output deviecs (hwplug:1,3, and the pretty printed HDMI hdmi, as well as all the others that fail to open), with consistent results.
All my google-fu so far has failed me, as hits vary in theme and never really match my situation closely.
Another issue that I observe is that I am unable to use iecset to manually unset the audioflag on the device - with this unset the output might be recognized as what it is, and not a generic 48kHz PCM.
But then I am not entirely sure as to what really causes the breakage. It could stem from XBMC not setting modes corectly, the wrong device selected for passthrough duties, some other wrong setting, driver limitations, alsa-misconfiguration...the only thing I know is that AC3 at least worked on windows (DTS didn't work, but this may be due to the TV not speaking dts..)
So, I'd be glad and thankful if anyone has a checklist or pointer in the right direction!
Addendum 2: For some unknown reason I managed to use iecset to switch from audio data-mode (PCM) to non-audio data-mode (passthrough). No idea why that wasn't working the last time I tried. I managed to succeed at playing DD soundtrack and use dts to DD re-encoding, and even switch back into PCM mode.
sadly the switching via iecset isn't exactly HTPC-compatible. Is there a way to use asound.conf to create virtual devices that set this flag? ...and why the hell doesn't XBMC set the flag by itself... :-(