It has 3d support today for 5xx, although it's pretty recent and there are problems using 2d and 3d together (which are being dealt with now).
It's useful to understand that there are actually 3 drivers working together :
- the x driver (radeonhd or radeon) which handles display setup, modestting, 2d and video acceleration, in the freedesktop.org tree at xorg/driver/xf86-vide-radeonhd or xf86-video-ati (radeon)
- the drm driver (mesa/drm) which is essentially the kernel driver, required for 3d acceleration, optional for 2d & video acceleration but tends to make them faster and more reliable
- the 3d driver (mesa/mesa), which executes the actual OpenGL 3d drawing commands
re: fglrx vs radeonhd/radeon, depends what you mean by "better". The fglrx 3d is faster and supports more OpenGL extensions, but the radeon/radeonhd 3d is open so if it has problems with your specific system it's easier to get help making it work, and it's easier for application developers to work out compatibility issues
We have released full 3d documentation for 5xx, and started releaseing 6xx 3d docs (only about 30% so far). Look in www.x.org/docs/AMD/