GCC 4.6, LLVM/Clang 2.9, DragonEgg Five-System Benchmarks

[phoronix]

Phoronix: GCC 4.6, LLVM/Clang 2.9, DragonEgg Five-System Benchmarks

Version 4.6 of GCC was released over the weekend with a multitude of improvements and version 2.9 of the Low-Level Virtual Machine is due out in early April with its share of improvements. How though do these two leading open-source compilers compare? In this article we are providing benchmarks of GCC 4.5.2, GCC 4.6.0, DragonEgg with LLVM 2.9, and Clang with LLVM 2.9 across five distinct AMD / Intel systems to see how the compiler performance compares.

http://www.phoronix.com/vr.php?view=15844

[Jonno]

My reading of this is that the LLVM's x86_64 backend is slightly better than GCCs x86_64 backend, but that the C(++) frontend (eg Clang) still sucks. Would be great to know if this holds for other frontends and/or backends available in both compilers as well.

Still, beating even parts of the long optimized GCC is a great step forward for LLVM. Here's hoping to Clang being brought into shape as well!

[XorEaxEax]

My reading of this is that the LLVM's x86_64 backend is slightly better than GCCs x86_64 backend, but that the C(++) frontend (eg Clang) still sucks.

Yes the combination of GCC + DragonEgg(llvm) did very well, although it was beaten on many occasions by a pure GCC setup. However the standout for GCC+Dragonegg was the Himeno test, where it was like ~40% better than all other compilers. Actually I find that a bit suspect, 40% is HUGE. Is there anywhere that I can find the exact compiler flags used for this benchmark? In fact, it would be great if Phoronix could just put the compiler flags used in the test right next to the results.

As a core i5,i7 owner it's great seeing that Gcc 4.6 has some nice optimization improvements for them.

[loonyphoenix]

Yes the combination of GCC + DragonEgg(llvm) did very well, although it was beaten on many occasions by a pure GCC setup. However the standout for GCC+Dragonegg was the Himeno test, where it was like ~40% better than all other compilers. Actually I find that a bit suspect, 40% is HUGE. Is there anywhere that I can find the exact compiler flags used for this benchmark? In fact, it would be great if Phoronix could just put the compiler flags used in the test right next to the results.

As a core i5,i7 owner it's great seeing that Gcc 4.6 has some nice optimization improvements for them.

I'm wondering what's stopping DragonEgg from becoming an offical part of GCC (as an experimental, optional plug-in, of course). The performance differences seem to warrant it.

[BenderRodriguez]

I think it's pretty useless showing the results without knowing what which compiler flags have been used. It should be at least noted if gcc was using -O2 or -O3. Another thing is testing the compiler for compilation speed. It is pretty pointless benchmark since it doesn't show the quality of the binary. I'd love to see the compilation time appended to every other test there AND compilation flags.

[Michael]

For anyone that wants to know compilation flags you're free to look at the test profiles and dig through the test's configure scripts default options and/or write patches against phoronix-test-suite/pts_Graph to have it report and print to the graph the flags, which I'll gladly accept upstream if someone does so.

[kevstev]

For anyone that wants to know compilation flags you're free to look at the test profiles and dig through the test's configure scripts default options and/or write patches against phoronix-test-suite/pts_Graph to have it report and print to the graph the flags, which I'll gladly accept upstream if someone does so.

Michael, where exactly would you find these options? I spent a few minutes digging through the test suite, and could not find anything relevant. I have a few Sandy Bridge core i7's on order and we plan on benchmarking how gcc optimizations flags will affect our software, particularly if we use march=native. Perhaps I will use the phoronix suite to do so. It seems like a good product, though it could use some features like the above.

Compiler performance benchmark articles without even stating whether optimization was turned on really diminishes the usefulness of the article, which is a real shame, because its clear a lot of work went into producing it.

-K

[Michael]

Michael, where exactly would you find these options? I spent a few minutes digging through the test suite, and could not find anything relevant. I have a few Sandy Bridge core i7's on order and we plan on benchmarking how gcc optimizations flags will affect our software, particularly if we use march=native. Perhaps I will use the phoronix suite to do so. It seems like a good product, though it could use some features like the above.

Compiler performance benchmark articles without even stating whether optimization was turned on really diminishes the usefulness of the article, which is a real shame, because its clear a lot of work went into producing it.

-K

If you run a command like: phoronix-test-suite install c-ray then go check ~/.phoronix-test-suite/installed-tests/pts/c-ray-*/ in its configure script for example I know you can find their default flags there, etc.

[BenderRodriguez]

I don't know the inner workings of PTS but is it possible to make the tests run from ram disk ? (at least not those that test disk performance). It would eliminate the latency that might have an impact on some tests.

[hoytak]

I think it's pretty useless showing the results without knowing what which compiler flags have been used. It should be at least noted if gcc was using -O2 or -O3.

Another aspect of this enters here: GCC tends to not enable a lot of the more aggressive optimizations by default, even at -O3. This is because they aren't helpful in every case, aren't 100% safe on some code structures, do transformations on the code structure that may not be intended or violate standards, etc. I've recently been playing around with intel's compiler, which does enable many of these by default, and yeah, they break some code. Here, though, I'm curious how gcc does with -O3 and the following options, none of which are enabled by default (see option notes for the reasons): -march=native -funsafe-loop-optimizations -fgcse-sm -fgcse-las -floop-interchange -floop-strip-mine -floop-block -fgraphite-identity -ftree-loop-im -fivopts -fmodulo-sched -fmodulo-sched-allow-regmoves -ftracer -fvariable-expansion-in-unroller -ffast-math -funsafe-math-optimizations -ffinite-math-only. Again, note that these may not be optimal for all programs.

In general, it seems that gcc's philosophy is to generate predictable and standards-compliant code by default, allowing "unsafe" options only by specific choice. After fighting code breaks caused by intel's compiler and having to figure out which options to disable, I can appreciate this :-).