I got a strange mode at one point also, the display said it was 1680x1050@75, it only gave a black screen and a xorg lockup. It wasn't however when I was playing around with that nativemodes setting, there's a huge amount of settings for the DAL, you can find them by "strings fglrx_drv.so | grep DALRULE -i". (I was trying to get widescreen resolutions to work, didn't work).

NOTE: Playing around with those settings might be dangerous to your videocard or monitor.

This only shows a part of the settings, not all of them start with DALRULE. I tried several ones there that I thought might allow custom modelines, but none of them did. Albeit I can't be sure of the format they use.

Also note that if you add modelines, it'll add a corresponding line to that file, which I found curious.