Bringing Up Advancements In Compiler Testing
Phoronix: Bringing Up Advancements In Compiler Testing
For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community...
Thanks for the improvements, it looks great! Looks like 3.8 will be a great release
Yes! I do complain alot about the Phoronix tests because they often use totally pointless optimization options, like -O0, -O1, targeting other architectures than the benchmarks are being run on (-march) which actually ends up benefiting compilers doing a poorer job optimizing code for a particular architecture. Stating that the upstream source tarballs comes with these options still doesn't make them relevant since no packager will distribute these applications compiled with -O0 or -O1. Primarily the packages in use on these systems are compiled with -O2 or -O3 and thus these conservative 'upstream' options are irrelevant as they are not what is in use on people's machines.
Now I HOPE this means that we will see the official benchmarks compiled with these options (-O2, -O3) but if not then atleast it's important to have the compiler options displayed in each benchmark so that people will be able to take them with a grain of salt when they see the tests are being done on -O0, -O1 levels (in other words, worthless). The compiler options used should ALWAYS be displayed, else the results can't be accurately judged.
Here is an example of a benchmarking test where you have all the necessary data provided to you to not only accurately compare the results but also be able to validate them for yourself:
Hey, this looks very useful.
A suggestion about what else could be done:
It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.
I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.
Right, I'm not sure that there is any way with GCC or any other major compiler to dump what options are actually being utilized in generating a particular binary.
Originally Posted by smitty3268
Well this is the reason I find -O3 the most 'fair' of optimizations to compare with since at this level all compilers are favoring code performance, although -O2 is also targeting performance it's more conservative in terms of code size. Obviously the best is to show both -O2, -O3, but again if you only choose one then I say -O3.
Originally Posted by smitty3268
Also which gcc flags are you referring too? I know that clang doesn't currently support gcc's -Ofast flag which expands to -O3 -ffast-math (which are both supported on clang), also previously it didn't support -march=native but that works in the latest version. Other than that clang has -O4 which turns on link time optimization which would be the same as -O3 -flto -fwhole-program on gcc (clang supports -flto -fwhole-program afaik so that should cause no problem).
Still I think link-time optimization and profile-guided optimization lies outside the realm of these tests (profile-guided optimization isn't even available in clang/llvm) and I would be happy just to see the tests done with sane standard optimizations like -O2 and -O3, enough with the crazy -O0 -O1, -march=k8 settings which have plagued previous benchmark tests. -ffast-math could be interesting though in floating point intensive applications, here's hoping these benchmarks can finally live up to their potential.
Well the options used in a binary is governed by the -OX flags or by explicitly enabling them (-f<optimization>). You can look at http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html to see exactly which optimization options are enabled on each -OX level.
Originally Posted by Michael
Also looking at the screenshots of the tests I think they look good with one caveat in the last shot (stream copy) where the compiler options aren't shown, why not? Just showing them at the bottom would make it perfect.
I wasn't really thinking of any optimization in particular, and they would vary from compiler to compiler. One that Mesa uses, just for an example, is -fno-builtin-memcmp. I'm not sure that really makes a difference in the benchmarks, since it's mostly meant to work around a performance problem in gcc itself, but it's noteworthy that there is a difference there between compilers. Something like march=native would be more important - even if clang has added this option, other compilers may not have.
Originally Posted by XorEaxEax
Yes this is a problem when comparing compilers as there are no 'rules' as to which optimizations compilers will enable at for example -O2, so one compiler may use more optimizations on this level than another which would lead one to believe that compiler 'A' is better, and then when you compile using -O3 compiler 'B' wins instead.
Again this is why I think -O3 is the best option if you only test one optimization level since it's 'supposed' to generate the fastest code across the compilers, but even better still is comparing the compilers at both -O2 and -O3. In particular I recall this was a problem for the open64 compiler which unlike gcc, clang, and icc had a very big performance difference between -O2 and -O3 and when compared against these other compilers at -O2 it looked really bad while at -O3 it was really competitive.
yet again i ask for a full x264 supplied "checkasm --bench" , as that would be a very good option to add to the Phoronix standard tests too for both old and new platforms