Fedora 7 to 10 Benchmarks

[Michael]

With EIST disabled, did you check the frequency? Could it be that on the non-7.04 distro/kernels, it was operating at the lowest by default?

Usually without OS-controlled cpufreq, the bios handles the cpu policy and may run at the lowest speed until it determines enough load. The problem in such cases is that the load may not be high enough (meaning just enough idleness in cpu %) to increase the freq. This is usually when if you're not 100% spinning on the cpu i.e. doing lots memory access or even just a small but frequent amount of I/O access.

that's my 2cents worth of a guess anyways..

it was running at the highest frequency, otherwise PTS would have reported it differently, which it didn't.

[Rendus]

Be irritated - That's part of my goal.

Irritation leads to action. If I have to irritate to get these tests rerun, or to get someone to at least acknowledge that they're horribly flawed, then that's what I need to do.

Because you guys seem content with the alternative - Allowing people to think any number of the following:

1) The people behind these distributions are so incompetent they get half the performance possible, across the board, other than Ubuntu 7.04.
2) Linux performance is dictated by random chance or the will of the Gods or the position of the moon.
3) This site can't manage to benchmark a ham sandwich reliably.

Which is it? Are the results flawed, or are we really going to accept that Ubuntu 7.04 is twice as fast as any other version of Ubuntu of Fedora? Because this site seems to continue to perpetuate these benchmarks as accurate. It's clear they are not.

[npcomplete]

Yes, but that does not invalidate the fact that he got the results he did. Whether or not it's generally applicable is up for debate here (and I have my doubts too) but it personally irks me when others simply deny someone else's experience, as if they just imagined any issue they came across or worse yet are accused of lying. For example, I see this when dealing with Firefox 3.x memory issues -- in general, it's much improved but there are some pathological cases where it just skyrockets.

The point is, he got what he got, with multiple test runs on his very specific configuration.. now why? I would imagine that someone with exactly the same setup would get the same results too.

Again, I do agree that we cannot draw any conclusions from this without more testing but just as informative would be root-causing his results even if it turns out not to be the fault of the distros themselves.

[MaestroMaus]

@Rendus

I understand this is very strange, and I wouldn't have believed it myself if I hadn't noticed anything. However the first thing I noticed when I installed 8.04 where some programs taking double the time that was needed before. I thought it was some kind of regression that would be fixed soon, but that wasn't the case.

I don't know if it has something to do with it, but my laptop also runs on a Intel Core Duo. Maybe it's related to these processors. Michael will post the results from different hardware this week, so we'll know soon enough about that.

Also, I think you are to fast with your conclusions. Like a previous poster said, you haven't got any proof so you can't be sure if it's an sloppy error made by Michael. And no, your Wikipedialink is used wrong, sorry to inform you. You can't use that in this case.

[Cynebeald]

Is there an easy way to reproduce the tests? For example, a test suite that one could easily download? I've been spending quite some time paging through the original benchmark, manually downloading the needed tests (install-all hangs on doom3, which isn't used anyway), running each one of them manually on the commandline and THEN finding out that I'm probably using some different settings (the GtkDrawingArea - Pixbufs test takes only 8 seconds, while the game tests crawl with single-digit framerates and radio takes "normal" 22 seconds, so something is clearly wrong. Not the games, but the pixbuf test, mind you). I also couldn't really identify the results in the online result browser (I would've tried running a comparison).

[urfe]

Is there an easy way to reproduce the tests? For example, a test suite that one could easily download? I've been spending quite some time paging through the original benchmark, manually downloading the needed tests (install-all hangs on doom3, which isn't used anyway), running each one of them manually on the commandline and THEN finding out that I'm probably using some different settings (the GtkDrawingArea - Pixbufs test takes only 8 seconds, while the game tests crawl with single-digit framerates and radio takes "normal" 22 seconds, so something is clearly wrong. Not the games, but the pixbuf test, mind you). I also couldn't really identify the results in the online result browser (I would've tried running a comparison).

The phoronix test suite (http://www.phoronix-test-suite.com/) was used.

[Cynebeald]

The phoronix test suite (http://www.phoronix-test-suite.com/) was used.

Let me explain what I was asking about. The Phoronix test suite has "test profiles" (more or less individual tests), such as the "gtkperf" test, the "build-apache" test or the "encode-mp3" test. Additionally, test can be grouped into "suites", that group together a set of test profiles (or other suites). For example, there is the "compilation" suite (that has all the build-* profiles, such as build-apache), the "universe" suite (testing more or less everything), or the "encoding" suite that tests only audio and video encoding.

I was asking for a Phoronix Test Suite suite profile that has groups together the test profiles used in these latest benchmarks (ubuntu vs. fedora, or even the original ubuntu benchmark), so anyone could easily reproduce them, without having to go trough the webpage and running each profile manually. After having a closer look, the universe-cli suite looks somewhat promising, but it would be nice if the article mentioned this (or even better, provided a suite profile).

Additionally, I was wondering about why I got a radically different number on the DRAWING_PIXBUFS subtest of the "gtpkerf" profile, while the other subtest, RADIO_BUTTON, gives me normal results. I don't have a good graphics card, and there's no reason for the test to be 5x faster than then ones in the article, so I suspect a different testing profile.

[glasen]

Additionally, I was wondering about why I got a radically different number on the DRAWING_PIXBUFS subtest of the "gtpkerf" profile, while the other subtest, RADIO_BUTTON, gives me normal results. I don't have a good graphics card, and there's no reason for the test to be 5x faster than then ones in the article, so I suspect a different testing profile.

Don't even try to reproduce the numbers of GTKperf. This program is definitely not a good benchmark because it completely relies on the used GTK-theme-engine. And the fglrx-driver is known to have the slowest to 2D-part of all drivers. I get 2x better results with my six year old Thinkpad R31 and "intel"-driver.

The more pixmaps an engine uses the longer take the tests with GTKperf. For example the "Murrine"-engine is almost twice as fast the "Aurora"-Engine when painting normal widgets (radio-buttons, normal buttons, etc.). So if you want to compare several distributions, you must always use the same theme and theme-engine so the numbers can be compared. And Ubuntu 8.x are using Murrine-based themes, whereas Ubuntu 7.x are using Clearlooks-based Themes (Murrine is also faster then Clearlooks).

[conholster]

Any chance of seeing similar tests run on 64bit k/ubuntu?