GCC vs. LLVM-GCC Benchmarks

[phoronix]

Phoronix: GCC vs. LLVM-GCC Benchmarks

Last Friday we published Mac OS X 10.6 benchmarks and then on Monday they were joined by Ubuntu 9.10 vs. Mac OS X 10.6 benchmarks. One of the requests that has come up since publishing those articles are to carry out a set of tests comparing the performance of LLVM and LLVM-GCC. With Apple's Snow Leopard release, some parts of the operating system were built using LLVM-GCC for optimized performance, although this compiler is not yet matured. In this article we have a set of 12 benchmarks comparing GCC to LLVM-GCC.

http://www.phoronix.com/vr.php?view=14164

[Mano]

Phoronix: GCC vs. LLVM-GCC Benchmarks

Last Friday we published Mac OS X 10.6 benchmarks and then on Monday they were joined by Ubuntu 9.10 vs. Mac OS X 10.6 benchmarks. One of the requests that has come up since publishing those articles are to carry out a set of tests comparing the performance of LLVM and LLVM-GCC. With Apple's Snow Leopard release, some parts of the operating system were built using LLVM-GCC for optimized performance, although this compiler is not yet matured. In this article we have a set of 12 benchmarks comparing GCC to LLVM-GCC.

http://www.phoronix.com/vr.php?view=14164

This is one, I think, important note from the release notes (for LLVM 2.5 and the appropriate LLVM-GCC):

* In this release, the GCC inliner is completely disabled. Previously the GCC inliner was used to handle always-inline functions and other cases. This caused problems with code size growth, and it is completely disabled in this release.

I think this may have a huge impact on your tested benchmark.

[BlackStar]

Ouch, LLVM-GCC was beaten solidly here. Any guesses as to why? Wasn't LLVM supposed to be much faster?

[deanjo]

Ouch, LLVM-GCC was beaten solidly here. Any guesses as to why? Wasn't LLVM supposed to be much faster?

Couple of factors I would guess comes into play. LLVM is fairly young in development and the biggest improvements so far as performance concerning LLVM is the time to compile. Unfortunately we haven't seen any of those compile times in the tests.

[Rip-Rip]

Couple of factors I would guess comes into play. LLVM is fairly young in development and the biggest improvements so far as performance concerning LLVM is the time to compile. Unfortunately we haven't seen any of those compile times in the tests.

Plus you won't see any speed advantage because phoronix is using the gcc frontend.

edit : s/backend/frontend/

[tettamanti]

next up we will be testing LLVM/Clang to see how that performs.

Shouldn't make any difference, Clang is just the frontend, right?

[tettamanti]

Plus you won't see any speed advantage because phoronix is using the gcc backend.

Hum, LLVM-GCC combines GCC frontend with LLVM backend (optimizer and code generator).

[Rip-Rip]

Hum, LLVM-GCC combines GCC frontend with LLVM backend (optimizer and code generator).

Thanks, I've edited (in fact, I wanted to write frontend...).

[kraftman]

Couple of factors I would guess comes into play. LLVM is fairly young in development and the biggest improvements so far as performance concerning LLVM is the time to compile. Unfortunately we haven't seen any of those compile times in the tests.

Someone mentioned there could be only one core used when comes to GCC and Ubuntu vs OS X benchmarks. Btw. I heard GCC is optimized to be good at some benchmarks, but I wonder if its better results have reflection in reality.

[srg_13]

I've been playing with Clang, and it really is very cool. Compile times are a lot better than GCC, and error messages are far more clear. I'm really glad Apple are funding development for it!

[[Knuckles]]

Couple of factors I would guess comes into play. LLVM is fairly young in development and the biggest improvements so far as performance concerning LLVM is the time to compile. Unfortunately we haven't seen any of those compile times in the tests.

But do normal users really care?

I think compile times should only be used as a test workload when comparing computers; using compile times to benchmark compilers is useless to most people (99%? more?).

[deanjo]

Someone mentioned there could be only one core used when comes to GCC and Ubuntu vs OS X benchmarks. Btw. I heard GCC is optimized to be good at some benchmarks, but I wonder if its better results have reflection in reality.

The way timed compilation tests are written in PTS gcc should take full advantage of the available cores.

For example the apache compile test:


\$TIMER_START
make -s -j \$NUM_CPU_JOBS 2>&1
\$TIMER_STOP" > time-compile-apache

[deanjo]

;90555']But do normal users really care?

I think compile times should only be used as a test workload when comparing computers; using compile times to benchmark compilers is useless to most people (99%? more?).

Probably not, but for the most part 99% of users don't care about the delta difference on most of the benchmarks. If it runs, that's good enough for them. The exception being maybe the game benches where even then once your maxing out your monitor's capability everything else is purely for bragging rights.

[kraftman]

The way timed compilation tests are written in PTS gcc should take full advantage of the available cores.

For example the apache compile test:


\$TIMER_START
make -s -j \$NUM_CPU_JOBS 2>&1
\$TIMER_STOP" > time-compile-apache


Yes, I just looked at this :)

[EnderWiggin]

When building these tests, is there a way to also build and affect the C runtime library. While I can appreciate that many of these tests have "hard stuff" in their code, the performance of an app is always so dependent on the performance of it's underlying runtime library (for things like memcpy, strcmp, etc.) that I think it's worth a look.

[tettamanti]

When building these tests, is there a way to also build and affect the C runtime library.
Hum, probably you can manage to override the system library with LD_PRELOAD (or LD_LIBRARY_PATH).

While I can appreciate that many of these tests have "hard stuff" in their code, the performance of an app is always so dependent on the performance of it's underlying runtime library (for things like memcpy, strcmp, etc.) that I think it's worth a look.
For critical paths (like memcpy, strcmp, ...) the library is probably picking up the most appropriate hand-coded ASM version of the function ;)

[hdante]

I think the description of llvm-gcc is incorrect. llvm-gcc back end is not a modified version of gcc, it's llvm. llvm-gcc front end is a modified version of gcc front end. Please confirm.

And the results were really insightful.

[b15hop]

Do a test of GCC vs Intel compiler and/or any other compiler. Compile time speed, run time speed, code size ... hmm..

[nanonyme]

I wonder how long it takes until Gentoo migrates to clang... Faster compile speed should be crucial there.

[ioannis]

Can you guys do a comparison of modern gcc and llvm with Visual Studio and Intel's compiler? It would be interesting to see where the open source compilers are against the proprietary

[kraftman]

I wonder how long it takes until Gentoo migrates to clang... Faster compile speed should be crucial there.

Gentoo maybe, but not Linux kernel. At least not now and probably not in the near future (if ever):

http://lkml.org/lkml/2009/5/8/388

the most interesting response IMO:

http://lkml.org/lkml/2009/5/9/59

[smitty3268]

It's a long, long way away from being supported on Gentoo. They are actually quite conservative when it comes to changing compilers, which makes sense when you think about it. They have to support all that code being compiled by users with different flags, and switching to a new compiler means going through every single emerge script and fixing every possible combination. Not an easy task to do, even for a gcc upgrade.

Also, clang itself doesn't support tons of code (think the entire KDE desktop) that's not written in C. They'd almost certainly have to start out using the GCC front-end to get most things working.

And judging by the benchmarks here, there might be a revolt among those "elite" users who stick on 50 different optimization flags to their CFLAGS in an effort to get any extra performance possible. I think the majority of Gentoo users are quite comfortable with letting the compiler work in the background for half an hour while they surf the web. Cutting that time to 20 minutes in exchange for reducing performance might not make much sense to a lot of them.

[nanonyme]

lso, clang itself doesn't support tons of code (think the entire KDE desktop) that's not written in C. They'd almost certainly have to start out using the GCC front-end to get most things working.Errr, what on Earth does that have to do with anything? That's just like complaining Sun's Java compiler doesn't compile C. :P

[smitty3268]

Errr, what on Earth does that have to do with anything? That's just like complaining Sun's Java compiler doesn't compile C. :P

Gentoo is hardly going to switch to a compiler that 95% of it's users can't use. That's all I meant. (95% because there are a lot of C++ packages most people use, even in GNOME).

[kernelOfTruth]

I wonder how long it takes until Gentoo migrates to clang... Faster compile speed should be crucial there.

I think most of the other Gentoo users agree that the speed (and correctness) of the final code counts more than the time it needs for compilation

for me it's a bonus if it compiles faster than gcc or {insert other preferred compiler}

if e.g. llvm-gcc was 4 times faster during compilation of code but the final code/compiled binary with gcc (without llvm) was 3 times faster during execution than llvm-gcc I'd prefer gcc over llvm-gcc any time :)

[RealNC]

Nope, Gentoo users aren't interested in giving up runtime performance for build speed. Who wants software that runs slower? Only developers might find it useful.

[yesterday]

Errr, what on Earth does that have to do with anything? That's just like complaining Sun's Java compiler doesn't compile C. :P

GCC is not just a C-compiler...

[nanonyme]

GCC is not just a C-compiler... This also has no relevance to anything whatsoever. I didn't say anyone would be giving up GCC. Mostly whether they'll set CC="clang" or whatever the binary is.

[lwatcdr]

I would love to see LLVM, GCC, and the Intel compiler go head to head. It is an interesting discussion as to what optimization is more important to an end users.
My opinion is that for many applications code size is more important than code speed. If the program is faster than the user then the code size will come into play. Quicker to load, uses less ram and so on. For other applications it is code speed.
That is one of the reasons I am not all that fond of 100% 64bit environments. ls does not need to use 64 bit pointers!

[deanjo]

I would love to see LLVM, GCC, and the Intel compiler go head to head. It is an interesting discussion as to what optimization is more important to an end users.
My opinion is that for many applications code size is more important than code speed. If the program is faster than the user then the code size will come into play. Quicker to load, uses less ram and so on. For other applications it is code speed.
That is one of the reasons I am not all that fond of 100% 64bit environments. ls does not need to use 64 bit pointers!

code size!=less resources

[lwatcdr]

code size!=less resources

Yes you can have a tiny code size and then allocate a dynamically allocate lots of memory. You can also use a terrible amount of CPU but you will often find smaller code does equal fewer resources.

[deanjo]

Yes you can have a tiny code size and then allocate a dynamically allocate lots of memory. You can also use a terrible amount of CPU but you will often find smaller code does equal fewer resources.

You would be surprised. There was a SLE bug filed a ways back by an ISV that reports VM intensive microbenchmark slows down by about 45%, and real world (for them) performance drop by 10-20% by using -Os rather than -O2 in SLES11 kernel.

This was verified by another ISV as well.

[wizard69]

Shouldn't make any difference, Clang is just the frontend, right?

This has likely been answered already but it has the potential to make a huge difference. I say potential because CLang is no where near what most people would call mature. In any event authoritative answers can be had on the LLVMM and CLang web sites.

Dave

[wizard69]

Not to rush the cook in the kitchen but how is the testing of CLang coming along. I fully realize it is a compiler fresh from the oven but the question is does it lead to tasty code or do the bits end up in the dogs dish?

As a new flavor for our bit baking endeavors I want to know if CLang is ready to spice up our code. Apple seems to think it is close to being ready for a topping of ice cream and to call it the best desert ever. I'm a bit reluctant and would like to see the test test results of others, before serving up CLang with my special sauces.


Dave

[ioannis]

Not to rush the cook in the kitchen but how is the testing of CLang coming along. I fully realize it is a compiler fresh from the oven but the question is does it lead to tasty code or do the bits end up in the dogs dish?

As a new flavor for our bit baking endeavors I want to know if CLang is ready to spice up our code. Apple seems to think it is close to being ready for a topping of ice cream and to call it the best desert ever. I'm a bit reluctant and would like to see the test test results of others, before serving up CLang with my special sauces.


Dave


damn it! now I'm hungry...