LLVM/Clang 3.2 Compiler Competing With GCC

[a user]

No even Open Source Software absolutely needs competition,

why "no"? did i claimed the opposite? no, i didn't! i just pointed out that in this case it worked well even without AND actually oos without competition has better chances to eveolve than closed source without competition.

this does not mean that competition in general is also for oos recommended.

particularly when it's a monolithic product as opposed to a more modular one*. The only reason that GCC is somewhat faster at this point in time is that it's been being developed since 1987, LLVM on the other hand was started in 2003, which means GCC has 13 years of development time in advance of LLVM, that's more than twice the lifespan of LLVM.

no, this is wrong! actually this has quite little to do with gcc's current state of performance. if your claim would be true than gcc wouldn't had performed so well in comparision with other compilers that existed over the years. just consider ms c++ compiler which actually performes quite bad, or compare it to the intel compiler, which used to perform better, though gcc still could hold up quite well.

the reason why llvm start to enhance so fast is that it could and did learned from the past of the others.

A perfect example of the competition bringing forth better products...

actually you don't need to post this here as i haven't seen somebody denieng that competition is good.

What we're seeing here though is the natural result in the difference in design between Modular and Monolithic architectures, a modular project make take a bit longer for initial design and setting up but once it's there it can accelerate far faster and far longer (almost if not indefinitely) whereas a monolithic application is much quicker to design and set up but without refactoring to a more modular design the development will slow eventually to a halt no matter how many people you throw at it, just because the codebase is so unwieldy, thus allowing a properly modularly designed application to accelerate past it.

you are simplifying things quite a lot. it is easy for interpret things like this while ignoring the fact that llvm can look back on the work of all the other compiler devs and learn from it.

* Now theoretically a Completely Modular codebase could exist in a vaccuum and do quite well, but it still benefits if nothing else from pulling ideas from competing products

i like how you are talking about a "completely modular codebase". there is no completely modular codebase, and even less a viable definition of such a thing. you can overmodularize things and make it worse. there is always the need of a design decision what and how much do you modularize. and afterwards you always see you could have done it better. LLVM has the advantage to look "afterwards".

this helps llvm to improve fast to the current level of gcc. but if it can top it significantly is yet to be seen.

[mayankleoboy1]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

[smitty3268]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

It most likely has to do with tunables - like at what point the compiler defaults to inlining methods versus using function calls, etc. Those are hard to get 100% right - there will always be some benchmarks that end up on the wrong side of the cutoff, but tweaking to speed them up may result in small losses across hundreds of other benchmarks and make it not worth it.

[Sidicas]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

I think you missed the part where in some charts, less is better.. While in other charts, more is better. LLVM is never far far ahead, it's far far behind in some tests. Supposedly due to lacking OpenMP support.

[XorEaxEax]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

You can dismiss any comparison where there isn't a -On flag set as we have no idea what optimization level is used. If you don't set a compiler optimization flag, GCC will deafult to -O0, which is no optimization.

Also, there's no way of ensuring that -O0,-O1,-O2 on GCC and Clang/LLVM are in any way equivalent optimization levels between compilers as there is no 'law' on what optimizations to enable at these levels. As such they are pretty worthless for compiler vs compiler benchmarks, atleast if you exclude -O3.

-O3 on the other hand is the highest optimization level which favors speed above everything else, so if you are only going to benchmark _one_ optimization level then this is it. There are also other special optimizations like link time optimizations (available in both GCC and Clang/LLVM), profile guided optimization (in GCC but not yet in Clang/LLVM) but these are beyond the scope of these tests in my opinion as Michael already have a hard time with the standard -On levels from the looks of it.

So looking at these tests, only the following declare setting the -O3 flag (best performant code, which is what we are benchmarking here):

Timed HMMER Search
Timed MAFFT Alignment
Blake2
Himeno Benchmark
C-Ray
Smallpt (OpenMP so rather pointless to compare)
GraphicsMagic (OpenMP so rather pointless to compare, how about configuring with --disable-openmp ?)
Tachyon (-m32 ? doesn't this work with 64-bit? )

Then we have Lame MP3 which defaults to -O3, so that should be ok unless Michael has changed the settings, however afaik this relies on assembly optimizations aswell.

JohnTheRipper defaults to -O2 (ignores CFLAGS environment variable so you need to manually edit makefile) and uses lots of assembly optimizations which makes it a poor candidate for benchmarking. However afaik it only uses OpenMP if you set OMPFLAGS to -fopenmp (not default), so why Michael is enabling it for this test is beyond me as it only makes sense to have if both compilers being benchmarked supports it.

VP8 libvpx and x264 would be great candidates for testing, both default to -O3 and offer a great challenge for optimizing compilers, except of course that Michael compiles them with assembly optimizations _enabled_ which makes them pointless as the compilers are left with pretty much no performance critical code to optimize, hence the pretty much identical results in these tests all these years.

The SciMark tests have no declared settings, and while Michael states that he sets CFLAGS/CXXFLAGS to -O3 -march=native, not all packages use/honor these environment variables, as seems to be the case here (as is the case with JohnTheRipper). Until we know the settings actually used in these tests they are worthless.

With Phoronix up for sale and with Michael indicating that he wants to continue focusing on the benchmarking I really hope takes the time to actually learn/take the time to properly configure the test software he benchmarks so that the results have better meaning.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

Yes there's obviously less low hanging fruit nowadays, obviously the lack of OpenMP is a big bottleneck and I don't understand why Michael persists in doing OpenMP based comparisons as of now at all as it's pointless until LLVM supports it. Just like it would be pointless to compare GCC + PGO against Clang/LLVM until it's supported in the latter.

[XorEaxEax]

GCC had major competition, what are you talking about?

Open source competition?

It's still the unpopular choice in a number of important fields where other compilers are dominant.

What are these in particular?

[XorEaxEax]

Does anyone know how the two compare on ARM processors? This seems to be an area where the two compete head-to-head.

Yes this is an area which I'm also interested in, in x86 64 which I am benchmarking GCC still holds supreme in terms of code performance, however I have no idea of how these compilers compare on ARM.

And given that this is an extremely strong architecture outside of the desktop it would be very interesting to see which compiler toolchain has the best performance here.

[roodemol]

A) Just supporting LLVM doesn't make someone an "apple fanboy" so go and take your apparent GCC-fanboyism somewhere else.

There's also the license issue, some OSes are stuck with a relatively outdated version of GCC. They are about to move to clang.

B) "Hands down" is a BIG stretch. LLVM was extremely competitive in just about every test that doesn't require OpenMP, and once OpenMP IS sorted out I bet those benchmarks will show LLVM is about the same as GCC in those tests as well.

Some time ago Michael Larabel published a small article about LLVM's Polly: http://www.phoronix.com/scan.php?pag...tem&px=MTIwMzg. How advanced is this project, is it possible to perform these benchmarks with this polyhedral optimizer?

C) The benefits in LLVM aren't in speed. 1) Modularity-- something GCC cant accomplish without scrapping a large chunk of the codebase SPECIFICALLY because GNU WANTED a monolithic codebase so that companies couldn't pull in select portions of gcc and just use those. 2) Cleaner code base. 3) Competition finally amongst compilers finally. GCC has gone uncontested in the open source world for a very long time, its good to see a project finally giving it a run for its money and ensuring that they don't start to rest on their laurels

There's another open source competitor, Open64. I hope this will bring some movement into the compiler scene and probably one of the contestant will outperform GCC.

Regards,
Gilles

[roodemol]

A) Just supporting LLVM doesn't make someone an "apple fanboy" so go and take your apparent GCC-fanboyism somewhere else.

There's also the license issue, some OSes are stuck with a relatively outdated version of GCC. They are about to move to clang.

B) "Hands down" is a BIG stretch. LLVM was extremely competitive in just about every test that doesn't require OpenMP, and once OpenMP IS sorted out I bet those benchmarks will show LLVM is about the same as GCC in those tests as well.

Some time ago Michael Larabel published a small article about LLVM's Polly: http://www.phoronix.com/scan.php?pag...tem&px=MTIwMzg. How advanced is this project, is it possible to perform these benchmarks with this polyhedral optimizer?

C) The benefits in LLVM aren't in speed. 1) Modularity-- something GCC cant accomplish without scrapping a large chunk of the codebase SPECIFICALLY because GNU WANTED a monolithic codebase so that companies couldn't pull in select portions of gcc and just use those. 2) Cleaner code base. 3) Competition finally amongst compilers finally. GCC has gone uncontested in the open source world for a very long time, its good to see a project finally giving it a run for its money and ensuring that they don't start to rest on their laurels

There's another open source competitor, Open64. I hope this will bring some movement into the compiler scene and probably one of the contestant will outperform GCC.

Regards,
Gilles

[vertexSymphony]

These benchmarks are as useful as looking at the sky to know if it's gonna rain during the day ... they lack so much information !!
for example, I couldn't manage to find which switches were used on each compiler and test, if inline assembly was disabled or not, blabla.

Well, I'm glad that Clang "does good" in this kind of gross measure; But this benchmark it's not my coup of tea: I would prefer something which I can draw more meaningful conclusions from