Do you have the problem of blank screen when change between the frame buffer and X window also. I think they might be the same problem.
That's a known bug for nvidia driver. I heard that it can be fixed by using the 169.04 driver.
I've tried the 169.04 for my 7600GT and it does work. Hopefully that should fix your problem.