CompilerDeathMatch: surprising results

[staalmannen]

Hi all

I have played around a bit with comparing various compilers available to ARCH
First off, I have been running the Free compilers for 64-bit:

http://global.phoronix-test-suite.co...07-15267-29041

TODO:
- I will also add ICC and make an AUR for open64 and LCC (builds fail atm)
- 32bit tests, include SolarisStudio and openWatcom

The weird thing about these results is that it seems like TCC is beating all of the other compilers handsomely (I would have expected GCC to be the performance choice) and also PCC gives surprisingly good results.

I exported
CC=/path/to/compiler
CXX=/path/to/compiler
and did not add any extra flags (planning on trying -x c for PCC for GCC compatibility mode)

I had to manually edit the xml file to set the compiler info for system-info correctly for each set up in the comparison (I noticed that too late, hence the "spamming" of global.phoronix.com)

Is there something I have done wrong or are these results for real???

[crazycheese]

Awesome idea!

Indeed, from TCC homepage

FAST! tcc generates x86 code. No byte code overhead. Compile, assemble and link several times faster than GCC.

Compilation Speed:...
TinyCC is about 9 times faster than GCC.

I wonder if it is possible to compile the kernel with tcc and if it will bring any benefits.

[Mo6eB]

The really interesting part is how tcc-compiled executables seem to be faster than the gcc-compiled ones. The tcc documentation doesn't list any flags for optimisation, so my hunch is that tcc optimises more aggressively than gcc by default. Could you run the benchmark again, but set CFLAGS to -O2?

[staalmannen]

Awesome idea!

Indeed, from TCC homepage




I wonder if it is possible to compile the kernel with tcc and if it will bring any benefits.

Apparently one used to be able to compile linux with TCC, or even run it "just in time" directly from C-code.

No idea whether this still works.

It will be interesting to see if the trend stays the same for 32bit and how the compilers compare to Open64, LCC, openWatcom and non-free ones like ICC and SolarisStudio. If I am able to, I would also like to package and test KenCC. To really make a real-world comparison I guess different optimization flags, different hardware etc needs to be compared though...

[staalmannen]

The really interesting part is how tcc-compiled executables seem to be faster than the gcc-compiled ones. The tcc documentation doesn't list any flags for optimisation, so my hunch is that tcc optimises more aggressively than gcc by default. Could you run the benchmark again, but set CFLAGS to -O2?

Sure I can try that too, although that will have to wait a bit I am afraid (I run the benchmarks over night). People are ofcoure welcome to repeat the experiments with other settings With more independent analyses, the comparisons will be more reliable.

[sabriah]

I exported
CC=/path/to/compiler
CXX=/path/to/compiler
and did not add any extra flags (planning on trying -x c for PCC for GCC compatibility mode)

Excellent work!

If you find the time, play around with "Acovea - Using Natural Selection to Investigate Software Complexities" (http://www.coyotegulch.com/products/acovea/) to see which compiler flags may be the best!

[MùPùF]

TODO:
- I will also add ICC and make an AUR for open64 and LCC (builds fail atm)

Hey, good to see you want to test open64 but it's intended to be replaced by path64. I made an AUR package for path64 (https://aur.archlinux.org/packages.php?ID=41846). Hope you'll be able to test it If there is a problem with the package, please tell me

Is there something I have done wrong or are these results for real???

Check the level of optimization was the same across all the compilers, there is something really *wrong* here, but I'm not a compiler guy at all.

[staalmannen]

Hey, good to see you want to test open64 but it's intended to be replaced by path64. I made an AUR package for path64 (https://aur.archlinux.org/packages.php?ID=41846). Hope you'll be able to test it If there is a problem with the package, please tell me


Check the level of optimization was the same across all the compilers, there is something really *wrong* here, but I'm not a compiler guy at all.

Thanks for the tip about Path64 - I was interested in trying the AMD branch as well. I will try them both I made an AUR from the binary open64.net distribution (https://aur.archlinux.org/packages.php?ID=45558).

So for 64-bit the TODO now includes: ICC, Open64, Path64, and possibly LCC
and after that I can try some CFLAGS, and for those that have it, alternative LD and AR etc...

For 32-bit there are lots of things left to do, but also a number of extra compilers available that look interesting...

The annoying thing is that it basically takes a whole night to run one of these tests on my laptop so the results will not come directly. I hope you guys are patient
I will post updates here as they come.

[staalmannen]

update: Included ICC

http://global.phoronix-test-suite.co...3426-6741-3877

did even worse than the rest, which really highlights that it will be interesting to compare when various optimizations are set.

next up: open64 and then there will be aggressive optimizations set (-O3) and perhaps later more moderate ones.

[RealNC]

You cannot do "benchmarks" without optimization, because it tells us absolutely nothing. All the results are simply wrong.