Gentoo Linux vs. Linaro ARM Benchmarks

[RealNC]

I'm not drawing any conclusions about the governor. The conclusions are about the "Ubuntu-based Linaro 12.08" as a whole.


You have the results from some other linux distro (which happens to be not "Stable Gentoo", but who cares).

We just established that you can't look at those due to a different GCC. The benchmark was really totally useless. You can't do *anything* with it.

If you want to compare governors, you need to compare Linaro vs Linaro, with only the governor being different.

[ryao]

We just established that you can't look at those due to a different GCC. The benchmark was really totally useless. You can't do *anything* with it.

If you want to compare governors, you need to compare Linaro vs Linaro, with only the governor being different.

I would not consider it to be entirely useless. The benchmark demonstrates that a tuned Gentoo system can outperform an optimized binary distribution. Of course, that ignores the fact that we all should have already known this and the build failures that can occur with newer versions of GCC.

If this were repeated with GCC 4.6.3, I would expect Gentoo to still do well.

[RealNC]

I would not consider it to be entirely useless. The benchmark demonstrates that a tuned Gentoo system can outperform an optimized binary distribution

No, it does not.

I claim that if GCC 4.7 was used on Linaro to build and run the PTT, then Linaro would have won. There. This benchmark now shows that a binary distro outperforms Gentoo, because that's just the way I happen to feel today.

Seriously, the only thing one can draw from this, is that GCC 4.7 can outperform GCC 4.6.

[dagger]

I assumed that the phoronix test suite was a bunch of scripts that used the system libraries and binaries. Is that incorrect?

By the way, I really would like to know who this Robert Sanders is. Michael claims that he was given shell access to a "board to be used as the primary build host for Gentoo weekly ARM stages". This makes no sense to me because I was under the impression that the release engineering team cross compiled the stages using catalyst.

Hi,

I'm Rob Sanders (few months ago I've changed my surname from Piasek to Sanders. My Gentoo profile is still using my old surname - http://www.gentoo.org/proj/en/devrel...l/userinfo.xml). I'm responsible for building ARM stages for Gentoo. All stages available at http://distfiles.gentoo.org/releases/arm/autobuilds/ or mirrors are now built on odroid-x board using catalyst. Catalyst is the software we use to build stages - http://www.gentoo.org/proj/en/releng/catalyst

The article claims that Gentoo Stable is using GCC 4.7. This is not the case. Stable ("arch") is using GCC 4.5.4, while Testing ("~arch") is using 4.6.3. GCC 4.7 is actually hard-masked and marked as experimental. You cannot install it, unless you unmask it (in Gentoo, that means telling the system "I'm about to potentially shoot myself in the foot, and yes, that's what I want.")

So I'd say this benchmark is bogus. If you installed GCC 4.7 on Gentoo, you should have done the same for Linaro. Since you didn't do that, the benchmark is highly biased.

Edit:
Also, if GCC was upgraded to a non-stable (and even non-testing) package version, then who knows what else was. I suspect the person who made the images for Phoronix didn't even mention this.

The entire system was built using stable GCC 4.5.3. Before giving Michael access, I've installed GCC 4.6 and 4.7 to see how performance differs between different compiler versions. Michael test profiles were built using GCC 4.7.1. All compiler flags were left default. Sorry for all the confusion it caused.

The point of this benchmark wasn't to point out that one distro is better than the other. It's here to show how performance can change when you use different distribution and/or different compilers.
Once I've got some extra time, I'll rebuild my gentoo system with different GCC 4.6 and 4.7, optimize CFLAGS it for ODROID-X and provide new set of results.


Rob

[ssvb]

I claim that if GCC 4.7 was used on Linaro to build and run the PTT, then Linaro would have won.

Are you sure? Even with Linaro being hindered by thumb2? And a nice thing about Gentoo is that it provides a freedom of compiler selection, so it can also use linaro patched variant of gcc 4.7 if necessary.

[ssvb]

If this were repeated with GCC 4.6.3, I would expect Gentoo to still do well.

Yes, indeed. I made a preliminary run of this benchmark with GCC 4.6.3 (and with the whole system rebuilt using it) as the first task for my new ODROID-X: http://openbenchmarking.org/result/1...SU-1209028SU74
Some of the tests failed to run due to missing dependencies like blas/lapack, and maybe some of the results are not too great because this particular SD card was slow. I'll give it another try with USB HDD just for comparison.

But in any case, the most interesting results are "Monkey Audio Encoding" and "N-Queens". Does anyone want to bet about how their results are going to change if we switch gcc to use thumb2 by default? Switching to thumb2 in Gentoo is possible by doing:

mkdir /etc/portage/env
mkdir /etc/portage/env/sys-devel
echo 'EXTRA_ECONF="--with-mode=thumb"' > /etc/portage/env/sys-devel/gcc
emerge sys-devel/gcc
emerge -e --keep-going world

Maybe some of the packages will fail to rebuild due to unresolved thumb2 issues, but the most important stuff like glibc should be fine.

[RealNC]

Sure Gentoo will be faster (I'm a Gentoo user), but why do a benchmark that's bogus, and then say "OK, this benchmark failed, but we all know what the real results would have been anyway."

I fail to see the point :-/ It's like this:

Here we compare a Lotus vs a Ferrari:



OK, it wasn't very helpful, but we all know the Lotus is faster.

[Ansla]

Unless the benchmark is redone properly, we'll never know.

Note that all I'm saying here is that this particular benchmark is null and void. You can't draw any conclusions from it. It was pretty much a waste of time both for Michael, as well as for us.

Michael got his ad revenue money, so the waste of time was only for us.