The problem is that because Linux insists on caching too much to improve throughput, it likes to swap stuff than give up on caches. Which immediately introduces a big bottleneck. Swapping path is extremely slow and has probably not been looked into for years.
Note that even if you swapoff all your devices, kswapd will still hunt down pages which are on disk and read-only (excutable pages e.g.) and evict them thinking that it can read them back at any time. Of course, that time comes when I move my mouse and X/kwin/plasma-desktop needs that page.
My issue was seen with 2.6.36-rc7, so some of the desktop responsiveness patches are in there.
It was said here: http://www.phoronix.com/forums/showthread.php?t=25084 that the 2.6.36 will provide RS690 hdmi audio support. I tried it and it is still not working for me. Does anybody know which git changeset provided the fix ? Is a new version of the xorg driver needed ? Everything seems ok for me, the device is listed in aplay -l and enabled in the mixer but still no sound...