Basically you built a new system from scratch and the first card that you got a little bit of output on leads up to a spontaneous reboot?
In my book that could be anything hardware related, including, but certainly not exclusively, the fglrx driver.
Did you bench / test the memory and/or CPU at all before you started swapping the VGA cards?
Think broken (new DOA) hardware, badly seated hardware, misconfigured or accidentally overclocked BIOS CPU, voltage or DRAM settings, lacking driver support in the kernel for an Intel quad-core (I'm a bit guessing here but I hope you get the train of thought), etc. etc. etc.
AFAIK the PCIe version of that card ought to more or less work with the bog-standard Catalyst 8-3 Linux driver installer completed and maybe a reboot thrown in.
That's the problem with shiny new systems: they tend to be rather complex beasts if you think it through and one really ought to take it step-by-step adding one piece of hardware at a time (okay, mobo, vga card, cpu and memory is the bare minimum but maybe start with one memory stick at a time?), testing it properly (the memory test included on many Linux distro install / Live CD´s is a good place to start) as you move along.
-edit- One other thing I thought of: the behaviour you describe is that of a PC with a power-supply that is lacking in capacity.
E.g. it has enough Watts to keep the quad-core, hdd and memory running all the way to where it loads gdm or whatever window manager, but collapses as soon as the graphics card start to suck power.
Did you think of upgrading the Power Supply Unit along with the mobo, memory and most of all the quad-core?
Those puppies suck watts by the hundreds and the 38xx GPU isn't far behind either.
it's a complete new system, with a new seasonic s12 650W PSU which should be good enough for this system.
I've tested the computer using KDE and apps using VGA with no accel (standard xorg conf after the linux install) for hours, the system is stable.
I've tested it overclocked (Q6600 email@example.comGHz with the FSB@402 MHz) but since I've installed the ati drivers, I've put the fsb back to it's standard frequency (233 MHz) to check if that was the cause of this behaviour ...
I'm planning to lend my new gfx card to a friend using windows to know if it's working or not ...
Well, I just tried to remove 2 gigs of rams using the latest ati drivers.
Finally, I achieved to get xorg to start ... BUT, as soon as I put back the 4 gigs the computer freezes on x start-up ...
I'll try to update my mobo bios but I don't think that will help me ... I really don't wana be limitated to only 2 gigs of rams just cause I'm using unsuported linux hardware (atm I really consider ati to be unsuporting linux, since there's no support for 3870x2 yet and their 3870 support is like buggy ...)
I keep hope (at least that's free ...) to get a solution to this issue ...