Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Bringing Up Advancements In Compiler Testing

  1. #1
    Join Date
    Jan 2007
    Posts
    14,623

    Default Bringing Up Advancements In Compiler Testing

    Phoronix: Bringing Up Advancements In Compiler Testing

    For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community...

    http://www.phoronix.com/vr.php?view=MTA2NzU

  2. #2
    Join Date
    Jun 2008
    Posts
    136

    Default

    Thanks for the improvements, it looks great! Looks like 3.8 will be a great release

  3. #3
    Join Date
    Oct 2009
    Posts
    845

    Default

    Yes! I do complain alot about the Phoronix tests because they often use totally pointless optimization options, like -O0, -O1, targeting other architectures than the benchmarks are being run on (-march) which actually ends up benefiting compilers doing a poorer job optimizing code for a particular architecture. Stating that the upstream source tarballs comes with these options still doesn't make them relevant since no packager will distribute these applications compiled with -O0 or -O1. Primarily the packages in use on these systems are compiled with -O2 or -O3 and thus these conservative 'upstream' options are irrelevant as they are not what is in use on people's machines.

    Now I HOPE this means that we will see the official benchmarks compiled with these options (-O2, -O3) but if not then atleast it's important to have the compiler options displayed in each benchmark so that people will be able to take them with a grain of salt when they see the tests are being done on -O0, -O1 levels (in other words, worthless). The compiler options used should ALWAYS be displayed, else the results can't be accurately judged.

    Here is an example of a benchmarking test where you have all the necessary data provided to you to not only accurately compare the results but also be able to validate them for yourself:

    http://willus.com/ccomp_benchmark2.shtml

  4. #4
    Join Date
    Oct 2008
    Posts
    3,099

    Default Very nice

    Hey, this looks very useful.

    A suggestion about what else could be done:

    It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

    I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.

  5. #5

    Default

    Quote Originally Posted by smitty3268 View Post
    A suggestion about what else could be done:

    It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

    I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.
    Right, I'm not sure that there is any way with GCC or any other major compiler to dump what options are actually being utilized in generating a particular binary.

  6. #6
    Join Date
    Oct 2009
    Posts
    845

    Default

    Quote Originally Posted by smitty3268 View Post
    Hey, this looks very useful.
    It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

    I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.
    Well this is the reason I find -O3 the most 'fair' of optimizations to compare with since at this level all compilers are favoring code performance, although -O2 is also targeting performance it's more conservative in terms of code size. Obviously the best is to show both -O2, -O3, but again if you only choose one then I say -O3.

    Also which gcc flags are you referring too? I know that clang doesn't currently support gcc's -Ofast flag which expands to -O3 -ffast-math (which are both supported on clang), also previously it didn't support -march=native but that works in the latest version. Other than that clang has -O4 which turns on link time optimization which would be the same as -O3 -flto -fwhole-program on gcc (clang supports -flto -fwhole-program afaik so that should cause no problem).

    Still I think link-time optimization and profile-guided optimization lies outside the realm of these tests (profile-guided optimization isn't even available in clang/llvm) and I would be happy just to see the tests done with sane standard optimizations like -O2 and -O3, enough with the crazy -O0 -O1, -march=k8 settings which have plagued previous benchmark tests. -ffast-math could be interesting though in floating point intensive applications, here's hoping these benchmarks can finally live up to their potential.

  7. #7
    Join Date
    Oct 2009
    Posts
    845

    Default

    Quote Originally Posted by Michael View Post
    Right, I'm not sure that there is any way with GCC or any other major compiler to dump what options are actually being utilized in generating a particular binary.
    Well the options used in a binary is governed by the -OX flags or by explicitly enabling them (-f<optimization>). You can look at http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html to see exactly which optimization options are enabled on each -OX level.

    Also looking at the screenshots of the tests I think they look good with one caveat in the last shot (stream copy) where the compiler options aren't shown, why not? Just showing them at the bottom would make it perfect.

  8. #8
    Join Date
    Oct 2008
    Posts
    3,099

    Default

    Quote Originally Posted by XorEaxEax View Post
    Well this is the reason I find -O3 the most 'fair' of optimizations to compare with since at this level all compilers are favoring code performance, although -O2 is also targeting performance it's more conservative in terms of code size. Obviously the best is to show both -O2, -O3, but again if you only choose one then I say -O3.

    Also which gcc flags are you referring too? I know that clang doesn't currently support gcc's -Ofast flag which expands to -O3 -ffast-math (which are both supported on clang), also previously it didn't support -march=native but that works in the latest version. Other than that clang has -O4 which turns on link time optimization which would be the same as -O3 -flto -fwhole-program on gcc (clang supports -flto -fwhole-program afaik so that should cause no problem).

    Still I think link-time optimization and profile-guided optimization lies outside the realm of these tests (profile-guided optimization isn't even available in clang/llvm) and I would be happy just to see the tests done with sane standard optimizations like -O2 and -O3, enough with the crazy -O0 -O1, -march=k8 settings which have plagued previous benchmark tests. -ffast-math could be interesting though in floating point intensive applications, here's hoping these benchmarks can finally live up to their potential.
    I wasn't really thinking of any optimization in particular, and they would vary from compiler to compiler. One that Mesa uses, just for an example, is -fno-builtin-memcmp. I'm not sure that really makes a difference in the benchmarks, since it's mostly meant to work around a performance problem in gcc itself, but it's noteworthy that there is a difference there between compilers. Something like march=native would be more important - even if clang has added this option, other compilers may not have.

  9. #9
    Join Date
    Oct 2009
    Posts
    845

    Default

    Yes this is a problem when comparing compilers as there are no 'rules' as to which optimizations compilers will enable at for example -O2, so one compiler may use more optimizations on this level than another which would lead one to believe that compiler 'A' is better, and then when you compile using -O3 compiler 'B' wins instead.

    Again this is why I think -O3 is the best option if you only test one optimization level since it's 'supposed' to generate the fastest code across the compilers, but even better still is comparing the compilers at both -O2 and -O3. In particular I recall this was a problem for the open64 compiler which unlike gcc, clang, and icc had a very big performance difference between -O2 and -O3 and when compared against these other compilers at -O2 it looked really bad while at -O3 it was really competitive.

  10. #10
    Join Date
    Jan 2007
    Posts
    459

    Default

    yet again i ask for a full x264 supplied "checkasm --bench" , as that would be a very good option to add to the Phoronix standard tests too for both old and new platforms

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •