Quote Originally Posted by Brane215 View Post
These tests oftne land in "Meh, whatever" category, at least for me.
Sadly yes for me too.

Quote Originally Posted by Brane215 View Post
For example, whole point fo using gcc-4.7x for me is -flto optimisation.
Well, -flto was in earlier versions of GCC (introduced in gcc 4.5 iirc) so it's not something new to 4.7, also I personally haven't had huge impacts from -flto, however the binaries tend to be quite a bit smaller. Profile-guided-optimization on the other hand usually gives me a very noticeable speed increase however it does require more work than just adding a flag.

Looking at these tests, many of them still doesn't list any optimization flags, meaning they could very well be done with -O0 or -O1, thus making them totally pointless for a compiler optimization benchmark.

If we look at those few where we see the optimization flag listed (-O3), the results we often get a nice performance increase with the later GCC's, look at FFTE, C-Ray, POV-Ray, but the vast majority of the tests doesn't have any optimization flags listed and as such we don't know if those benchmarks are of any consequence whatsoever in terms of compiler optimization.

Then of course we have x264 and VP8, codebases which rely heavily on finely optimized assembly code for the performance critical parts, unless you explicitly disable the assembly code when configuring, these tests are totally worthless in terms of comparing compiler optimization. It's even more sad given that comparing how the compilers optimize this code would actually be very interesting results.

Quote Originally Posted by Brane215 View Post
Also, when finding regressions, it would be nice to go in-depth for their cause. Is error on the part of compiler, or simply program infrastructure misunderstood some compilers new feature, for example ?
Yes, but we don't even know if they are regressions if we don't know the actual optimization flag, GCC defaults to -O0 which is for debugging as it turns off all optimizations, so if -O0 generates slower code between versions it's hard to call it a regression as -O0 does nothing to improve code performance.

Now, given that -O0 is the default for GCC, unless we actually set a -On optimization level the tests will be done with no optimization, thus rendering them useless. So when Michael omits reporting any compiler optimization flags it's simply impossible to tell if the benchmarks have any value whatsoever.