There's some documentation out there on how videos and the 3D pipeline work, and there's definitely room for improvement in the drivers out there.
One general link that describes the overall 3D pipeline and software stack is here:
http://www.x.org/wiki/Development/Do...VideoCardsWork
You will probably want to familiarize yourself with Gallium3D, as that's what a lot of future development will probably use
http://en.wikipedia.org/wiki/Gallium3D (also check out the external links there)
I've often heard that the best way to get involved is to pick an extension that you would like to see supported in Mesa and actually create an implementation of it.