GLSL is only used in color space conversion and scaling I believe. No one had written anything on h.264 decode at this moment. Only MPEG2 in vdpau code in Gallium3d driver which nouveau support.
If you're running ion install the nvidia binary blob for hardware decoding.
Using GLSL for color space conversion is bound to be less efficient than XV (even if it doesn't use the dedicated video layer of the card, it's likely not going through the entire GL pipeline), which I guess XBMC doesn't support anymore.
Use a better player that still supports XV, like mplayer ;)
I use XBMC with xvba support, it works amazingly well. Even better than using VLC under Windows, which chokes a bit with subtitled movies on my AMD E-350 system. XBMC under Linux handles it just fine.
I really wish Gstreamer and Flash had support for xvba. That and some performance issues with gnome-shell are my only remaining problems with this system, which is a remarkable improvement since I bought it some 8 months ago.