You really start to hate kms when you want to be able to unload the drivers. Would be nice if somebody could explain me how to switch to a default text console and unload the driver in a script which is not executed via ssh. That's a basic requirement to install binary drivers later. There is basically a hack that can be used to unload, but that kills the vt. For reference you can try this:
Of course X needs to be stopped while you do that.
for x in $(ls /sys/class/vtconsole/*/bind 2>/dev/null); do echo 0 > $x; done
rmmod radeon ttm drm_kms_helper drm