Ubuntu 9.04 vs. Fedora 11 Performance

[phoronix]

Phoronix: Ubuntu 9.04 vs. Fedora 11 Performance

Fedora 11 was released earlier this week so we have set out to see how its desktop performance compares to that of Ubuntu 9.04, which was released back in April. Using the Phoronix Test Suite we compared these two leading Linux distributions in tasks like code compilation, Apache web server performance, audio/video encoding, multi-processing, ray-tracing, computational biology, various disk tasks, graphics manipulation, encryption, chess AI, image conversion, database, and other tests.

http://www.phoronix.com/vr.php?view=13950

[SyXbiT]

It's obvious why Fedora might be faster (newer kernel, and ext4)
But seeing as Fedora uses mostly newer versions of all the software (vs Ubuntu) does that mean that the places where Fedora loses are regressions ?

[kraftman]

It's obvious why Fedora might be faster (newer kernel, and ext4)
But seeing for Fedora uses mostly newer versions of all the software (vs Ubuntu) does that mean that the places where Fedora loses are regressions ?

Fedora has SELinux enabled for default and it has sometimes big impact on performance:

http://lkml.org/lkml/2009/6/3/350

Ubuntu however, uses AppArmour, but I don't know if this slow its down.

[RealNC]

Better slow than crashing every 2 minutes I says.

[nathanvaneps]

It is a bad idea to have compilation benchmarks with different versions of a compiler. Because the speed of different compilers is an added variable to determining the speed of the OS. In fact, the difference of speed between different compilers might be the only reason for a speed difference between the OSes.

Also, as a developer, the speed of compilation is one of the last factors for consideration when choosing a compiler. I'd be primarily concerned with the speed of generated executable and supported languages, standards, and libraries.

[Kano]

Yes, also compile benchmarks are really stupid to compare on other systems, because depending how many depends are installed on a system the compile time may be longer or slower. PTS sometimes shows the fastest results when it did not fully complete ;)

[nathanvaneps]

I should also say that I enjoy this site very much. And that is important to keep a tab on performance for the community. And one more thing, it isn't hard for me to skip looking at the compiler speed benchmarks when viewing the article.

It is a bad idea for me to offer constructive criticism, without saying I like everything else!

It is a bad idea to have compilation benchmarks with different versions of a compiler. Because the speed of different compilers is an added variable to determining the speed of the OS. In fact, the difference of speed between different compilers might be the only reason for a speed difference between the OSes.

Also, as a developer, the speed of compilation is one of the last factors for consideration when choosing a compiler. I'd be primarily concerned with the speed of generated executable and supported languages, standards, and libraries.

[mendieta]

I think "stupid" is a bit of harsh. I agree with the objections, but I think compilation of a rather large project has historically been used as an overall system benchmark because it exercises CPU, memory and disk read/writes. So, besides the caveats you all provided, it's still a nice benchmark if taken with a grain of salt

Michael: are you planning a 64 bit comparison? It would be nice to add it to the mix. Also, would it be possible to run Ubuntu on ext4 too see how much of the disk performance is coming from other factors? (different kernel, different compiler, libraries perhaps?)

[Kano]

Compile tests are not by definition bad benchmarks. As long as you use the SAME reference system and only change cpu/board it is valid to compare. But testing different distributions for compile speed leads to errors.

[mendieta]

(sorry, I resubmitted the same post by mistake, deleting it)

[l8gravely]

My big beef here is that you use a *fast* system for your tests. Now how about running each OS on a netbook, where the slower CPU, more limited memory, slower disk would all hightlight the differences better.

I've also wondered about how many times each test is run, and whether you collect meaningful statistics. I'd love to see the mean, median and std deviation for each test. I think in alot of cases, we'd see that the systems really are just tied, or have just a small improvement.

It would also be nice to see how reproduceable each sub-test really is, which would tell us alot about how useful each test really is.

Another tweak to show would be to run each test multiple times, but to drop and not drop the vm caches between tests, to see how well the VM and it's caching helps.

I do like these benchmarks, they're certainly improving over time, but they could be better. More data please!

[Michael]

I've also wondered about how many times each test is run, and whether you collect meaningful statistics...It would also be nice to see how reproduceable each sub-test really is, which would tell us alot about how useful each test really is.

All of that data is easily available and clear through the Phoronix Test Suite.

[diegocg]

I wonder why phoronix benchmarks always focus on the tiobench latency, and never shows the tiobench troughtput...tiobench outputs many data, and many of them are more interesting than latency.

[Apopas]

Michael: are you planning a 64 bit comparison? It would be nice to add it to the mix. Also, would it be possible to run Ubuntu on ext4 too see how much of the disk performance is coming from other factors? (different kernel, different compiler, libraries perhaps?)

As the article says in the ifrst page: "The x86_64 builds of both Fedora 11 and Ubuntu 9.04 were used."

[Apopas]

It's obvious why Fedora might be faster (newer kernel, and ext4)
Both use 2.6.29 kernel.

[Kano]

U uses a patched 2.6.28 kernel.

[mtippett]

It is a bad idea to have compilation benchmarks with different versions of a compiler. Because the speed of different compilers is an added variable to determining the speed of the OS. In fact, the difference of speed between different compilers might be the only reason for a speed difference between the OSes.

Also, as a developer, the speed of compilation is one of the last factors for consideration when choosing a compiler. I'd be primarily concerned with the speed of generated executable and supported languages, standards, and libraries.

Most (casual) developers don't look to closely at the compiler that ships with the system, if it has gcc they start using. It is only in more formal environments where there is tool selection (or an individual developer is really focused on some metric like above.

I have built *way* too many compilers myself (crosstool really rocks), but for most purposes, I don't bother looking to closely at the compiler for general ad-hoc development tasks.

Finally, remember that a lot of people _like_ to stay bleeding edge with end user functionality, like kernels, gnome, firefox, etc. They aren't focused on speed or size of the result, but they know they will be building on a regular basis. They too don't focus on selecting the right compiler, but instead grab what is easily available.

Matt

[cruiseoveride]

Other than the kernel related performance differences, Ubuntu kicks Fedora anal. Time to switch.

[glyj]

I regret there is nothing about Mandriva this time.
I'd love to see "Ubuntu vs Fedora vs Mandriva performance" ;-)

regards,
glyj

[Kano]

Maybe better give a gui rating not a performance rating ;) I prefer solutions executed by scripts - run it, then something works. Of course lots of distros provide extra guis for this and that. I did not try Mandriva nor Fedora lately, but from history usually Mandriva is maybe just behind SuSE's yast from gui tools. When you like the tools a distro provides additionally to its preconfig/stability that's usally the logical reason to choose it. Nobody would use Mac OS X because it is faster in a few benchmarks - same applies for any distro. I don't know of anybody who selects a distro because some apps run slightly faster.

As PTS compares mainly selfcompiled binaries it would not be that hard to bootstrap a newer compiler if needed. I never did that because the default compiler was slower or generated slower binaries. The only time i definitely had to compile even an older gcc (2.95) instead of using the default (some 2.96 prerelease) of old Red Hat 7.x systems was because the compiler was so broken that it was not able to generate binaries from standard source code, i still have got no idea how many changes red hat did to compile everything they shipped precompiled ;) Well i did not prefer to fix the code, i just added another compiler - all in my home only, so nothing could hurt the rest of the system.

[mendieta]

As the article says in the ifrst page: "The x86_64 builds of both Fedora 11 and Ubuntu 9.04 were used."

Arrg, how lame of me, thanks for pointing it out. Then it would be nice to add the 32 bit ;-) Seriously, this is the most FAQ I see in forums from people who just bough a 64 bit system (namely: "Does 64 bit perform better?").

[kxmas]

I agree with the objections, but I think compilation of a rather large project has historically been used as an overall system benchmark because it exercises CPU, memory and disk read/writes. So, besides the caveats you all provided, it's still a nice benchmark if taken with a grain of salt

In summary, it's the best benchmark ever, except that it doesn't mean anything.

[Azmo]

a typo in the article:

Ubuntu 9.04 ships with the Linux 2.6.29 kernel,

https://wiki.ubuntu.com/JauntyJackalope/TechnicalOverview#Linux%20kernel%202.6.28

[kraftman]

Other than the kernel related performance differences, Ubuntu kicks Fedora anal. Time to switch.

Just disable SELinux and performance should be better.

[chronomatic]

You guys are overlooking the biggest reason Fedora performs worse in some of these tests (and frankly I am disappointed no one has mentioned it).

Fedora's performance is impacted by the fact that many of its binaries are compiled with options like FORTIFY_SOURCE and -fstack-protector as well as ASLR hardening. Further, exec-shield and Selinux are on by default. All of these protections will put a small hit on performance (SELinux alone can sometimes result in as much as 7% overhead on some benchmarks) Bottome line: when all of these protections are combined, the results in this benchmark seem about right. You can read about Fedora's security features here. (http://fedoraproject.org/wiki/Security/Features)

No other desktop Linux distro I am aware of offers the security that Fedora does out of the box. So, really, we're comparing apples to oranges here. Essentially, it's a trade off (just like many things are when choosing a distro), and in this case it happens to be performance vs. security. Personally, I am willing to give up some performance in some areas for the increased security.

[chronomatic]

(I already posted my reply once and it didn't show up, so if it does show up later, I apologize for the double post).

I had a long reply as to why this benchmark showed the results it did, but I will summarize:

Fedora has increased security that can only be achieved through some compiler options like FORTIFY_SOURCE, -fstack-protector, PIE, and exec-shield. Most of these options will result in a performance hit. And that doesn't even count SELinux, which in itself has shown overhead as high as 5-7% in some tests. It's obvious to me that (especially in the Apache tests) this is why Ubuntu performed so much better. Ubuntu is not protected like Fedora is! (AppArmor on Ubuntu only protects CUPS by default and has no impact on overall system performance).

Read about Fedora's security features here. (http://fedoraproject.org/wiki/Security/Features)

[l8gravely]

All of that data is easily available and clear through the Phoronix Test Suite.

Ok, but it would be good to see the error bars on the graphs in the report as well then, to give people a better idea visually of the nuances.

John

[Kano]

@chronomatic

Usally all apps are compiled from source if they are not binary only. That usally does not affect it, but you might have installed more build-deps on one system that would enable additionally features and compile longer for example.

[bjrosen]

Just reporting benchmark results isn't all that useful, please try to figure out the reasons for the discrepancies in performance. For CPU bound programs the differences in performance between Fedora and Ubuntu should be in the noise, when you find large differences it's worth digging deeper to find out the reason. Are the CPU Speed Governors set the same? Is it SeLinux? are there background processes that are running on one and not the other. Is it compiler switches? Is it the kernel rev? It's easy to enable/disable/change SeLinux, daemons, speed governor modes, firewall and even file systems. It's also easy to compile a standard kernel using identical switches for both distros, that would allow you to isolate the effects of the kernels.

[highlandsun]

Or just run the benches with the CPU governor set to powersave, and lock it at the lowest speed. This will reduce the influence of peripheral device speed, and focus on CPU/software efficiency.

I sometimes have to do that when testing/debugging multithreaded code, because problems that don't occur on a fast multicore test can sometimes be revealed on a slower system...

[feistybird]

Hi,

Can you guys consider to test ArchLinux vs. Fedora, Ubuntu, Suse, or other Distro. next time?

Many Arch Linux user seems to be very proud of their OS performance, and claims that the Arch is the fastest Distro. on the i686 system.

I wonder if that's true or not. I couldn't find any benchmark tests of Arch Linux on the interent...

[Apopas]

Hi,

Can you guys consider to test ArchLinux vs. Fedora, Ubuntu, Suse, or other Distro. next time?

Many Arch Linux user seems to be very proud of their OS performance, and claims that the Arch is the fastest Distro. on the i686 system.

I wonder if that's true or not. I couldn't find any benchmark tests of Arch Linux on the interent...

Arch is fast yes, but the fastest are the source based distros, like gentoo simply because you can tweak your system to maximum during compilation.

[kraftman]

Arch is fast yes, but the fastest are the source based distros, like gentoo simply because you can tweak your system to maximum during compilation.

Exactly. When comes to Arch vs. Ubuntu (if the same kernel, gcc and apps versions are used) performance shouldn't differ much in Phoronix benchmarks. You may feel Arch is faster, because there aren't so many scripts like those used in Ubuntu, but they shouldn't affect mentioned tests.