Intel Atom On Ubuntu, Fedora, OpenSuSE, Mandriva

[phoronix]

Phoronix: Intel Atom On Ubuntu, Fedora, OpenSuSE, Mandriva

Back in September we looked at the Intel Atom performance on a few Linux distributions using the ASUS Eee PC 901, but now with new stable releases of some of the most popular distributions out in the wild, we've decided to re-conduct these tests. We are using a slightly different Atom-based system this time and we are comparing the performance on Ubuntu 8.10, Fedora 10, Mandriva 2008, and OpenSuSE 11.1.

http://www.phoronix.com/vr.php?view=13259

[luca]

hi,

I am very interested in your benchmarks, but I wanted to know if there is any reason not to display a standard deviation in those graphics. (Maybe to know also how many times you have performed the measures would be interesting too).

I mean, to say:
ubuntu: 40.5
fedora: 35.5

is quite different than:
ubuntu: 40.5 +/- 20
fedora: 35.5 +/- 20

Thanks a lot for your work.
Bye
Luca

[Sacha]

phoronix test suite results shows the deviation

[luca]

Ehm... where? In the graphics in the article?
thanks

[Luis]

As far as I know, only OpenSUSE turns barriers on by default (using ext3). That would explain why it's so much slower in some tests (like the SQLite test), but strangely it makes no difference in IOzone. I guess this could be more a proof of IOzone being a bad benchmark than anything else, since in real world tests barriers do make a big difference when it comes to IO intensive tasks.

[deanjo]

As far as I know, only OpenSUSE turns barriers on by default (using ext3). That would explain why it's so much slower in some tests (like the SQLite test), but strangely it makes no difference in IOzone. I guess this could be more a proof of IOzone being a bad benchmark than anything else, since in real world tests barriers do make a big difference when it comes to IO intensive tasks.

More then likely beagle (the indexing service) which is enabled by default is the culprit.

[yoshi314]

i have a mainboard with atom 330 (same as atom 230, only dual-core).

i found fedora to be a bit sluggish (didn't test much). gentoo seems unfeasible at that hardware ;-). archlinux works like a charm.

aside from that the desktop experience is quite acceptable. but don't hope for playing any recent 3d-intensive games at good framerate.

[Extreme Coder]

You give the impression that you're a tad biased against Mandriva, by using the 2008 version while 2009 is out :/ And I didn't see Mandriva in the GTKPerf tests, and there wasn't even a mention for it in the final page :/

[Smarter]

I don't know if other distributions do that too, but Ubuntu has a special architecture named "lpia" with all the packages rebuilt to take advantages of Atom features. So, did you use Ubuntu lpia or Ubuntu x86 for your tests?

[R3MF]

I have a Lenovo S10e netbook with a non-standard resolution of 1024x576, and i would be very interested to see how completely opensuse 11.1 will install on a netbook.

1. does the wireless get configured properly
2. does the webcam get configured properly
3. is the native resolution easily selectable
4. how well does the power management work

regards

[jeffro-tull]

Not that I think it would make a world of difference, but why couldn't you guys wait three days until openSUSE 11.1 goes final?

Also, even though I get the impression you guys are all about Gnome, how about some benchmarks on KDE? I ran normal ubuntu on my Aspire, and it ran fairly well. Mandriva 2k9 ran quite well. Kubuntu is very, VERY sluggish. Of course, it's not just KDE - it starts slower than normal ubuntu did (don't know why).

Actually, how about that, too - start-up time benchmarks.

[Thetargos]

Not only from being a Fedora user, but I have indeed noticed that for some reason (maybe downstream compatibility?) Fedora kernels tend to do fairly bad on HDD IO tests (be it IOZone or good-ol' hdparm) with some ICH controllers. From the benchmarks, though, it would seem other distros either do not apply some patches that hinder these chipset's performance, or Fedora does not apply those other distros seem to apply. Wonder what would the perf be with a vanilla kernel?...


I say this simply because on my hardware I do get pretty decent HDD throughput with ext3 on Fedora (then again, I'm using x86_64, and an nForce chipset), however, it would seem as if Fedora's packages are quite well optimized for the Atom (I wonder how would it perform on a similar CPU [Pentium M, maybe?] with the i686 version)

[Mithsir]

I would also like to see some more info in ALL THE phoronix benchmark articles:

- Test Methodology, e.g.: Are the tests performed multiple times? If not, the results are nearly worthless.
- Standard deviation of multiple runs of the same benchmark.
- System Load: Every result should be accompanied by the (average/max) system load experienced during the benchmark. This would allow a clue on whether there are background tasks (e.g. beagle) running that affect the result.
- ...

So maybe the phoronix team can include these metrics in future runs.

[jeffro-tull]

Yeah, system load! Good call!

I don't know if they managed to get it sorted out with 11.1, but with 11.0, openSUSE could have a bit of an issue with background services. Let's say you started with KDE3.5.9 on your system, and then you decide you want some fancy new 4.1 from the Build Service. Well, you can do that just fine, except then you'll have beagle AND nepomuk in the background - two services doing essentially the same thing. Things like that can definitely make a difference if left unchecked.

[sreyan]

I don't know if other distributions do that too, but Ubuntu has a special architecture named "lpia" with all the packages rebuilt to take advantages of Atom features. So, did you use Ubuntu lpia or Ubuntu x86 for your tests?

From my understanding lpia not only uses different compiler optimizaitons but also some of the software itself is compiled with different options/dependencies.

It would be interesting to see an i386 ubuntu vs lpia ubuntu comparison -- I might just do this on my aspire one if I get a chance this week.

[drhsqlite]

I am the primary developer for SQLite. The SuSE people have contacted me asking why I think SQLite runs so much slower on their system than on Ubuntu. I don't know, but I can posit a guess or two....

The culprit is likely the fsync() system call. SQLite makes use of fsync() in order to insure that transactions survive unexpected power loss. As long as you never loss power, whether or not fsync() works is probably unimportant to you. But if, someday, you do lose power unexpectedly, and you are in the middle of an SQLite transaction, a working fsync() might become very important to you.

It could be that SuSE has a kernel bug that somehow makes fsync() really slow. Or (more likely) it could be that the other distributions are skimping on fsync() and not taking the time to do it right.

I am writing because I am concerned that benchmarks such as those you publish are encouraging vendors to skimp on fsync() and other services, which make average performance better, but which result in catastrophe when an anomaly such as a power loss occurs. We've already seen that most disk drive and memory stick makers take shortcuts on their flush operations. We at the SQLite development team have heard many woeful tales from people who thought their data was safe, because the disk drive said it was on oxide, but who lost data during a power loss because the disk drive was really lying.

Benchmarks are important. But it is also important to understand that some vendors might increase benchmarks performance at the expense of reliability.

I do not know what is going on with these tests. There really could be something wrong with SuSE for all I know. (Nobody on the SQLite development team has any connection with SuSE and SQLite does not receive any funding from SuSE or Novell.) But my suspicions are that the other distros are in fact skimping on fsync somehow, which is boosting performance and the risk of corrupting data during a power loss.

Can anyone suggest ways of testing the rebustness of a distro against power loss? Such a test would likely settle this question.

[Luis]

I suggested before that barriers could be the reason of OpenSUSE's slower performance in that test. It's the only distro that turns them on by default. Maybe just testing with and without barriers (with mount option barrier=1 and barrier=0)could give an answer to the question.

Someone else also pointed at the Beagle indexer as the culprit. So turning that off could be another way of finding out if it makes a difference.

I don't think it's a different implementation of fsync. No distro would disable fsync to improve performance.

[deanjo]

Andreas Jaeger commented on the performance issues with openSUSE 11.1.

http://lizards.opensuse.org/2008/12/16/comments-on-phoronix-benchmarking-opensuse-111/

Phoronix has run some tests (http://www.phoronix.com/forums/../scan.php?page=article&item=intel_atom_four&num=1) comparing the openSUSE 11.1 release candidate (RC1), Ubuntu 8.10, Fedora 10 and Mandriva 2009.0 on Intel Atom.
We have looked at the results and they are not good for openSUSE 11.1. I’ve talked with a few engineers and want to present below our first analysis.
While the benchmarks were done on a specific hardware, they might be relevant for other hardware as well.
Note that the numbers I cite below are not benchmark numbers comparable to the one Phoronix measured, they are measured on totally different machines by different engineers and not all are done as real benchmarks. But they show some of the problems.

Disk I/O - Safe or Fast?

We have barriers enabled for the ext3 filesystem. This is needed for filesystem integrity but forces at certain points a flush of disk writeback caches to prevent data corruption.
As Wikipedia (http://en.wikipedia.org/wiki/Ext3#No_checksumming_in_journal) states:
“Ext3 does not do checksumming when writing to the journal. If barrier=1 is not enabled as a mount option (in /etc/fstab), and if the hardware is doing out-of-order write caching, one runs the risk of severe filesystem corruption during a crash.”
With openSUSE barrier=1 is the default and even AFAIK openSUSE and SUSE Linux Enterprise are the only distributions enabling barriers by default.
If you want to disable barriers, use “mount -o barrier=0″ on ext3 (or change /etc/fstab).
The gzip test for example gives on one of our machines the following results:
kernel-default-2.6.27.8-1.1 / ext3: 1348s
kernel-default-2.6.27.8-1.1 / ext3 barrier=0: 437s
X11 and Graphics - Performance of the Intel Driver

Looking at the graphics results, I see that OpenGL has the same performance but XRender is horribly slow, but Ubuntu sometimes(!) hits the same issue.
We have an upstream bug open about X11 speed (see here (http://bugs.freedesktop.org/show_bug.cgi?id=18389)), and it’s considered the highest priority bug, still nobody has a clue where it comes from. This needs to be rechecked with the final version of openSUSE 11.1, though, because there are some indications that it got improved. Intel has apparently fixed some of that in a newer driver that is not available yet.
It would also be interesting to know whether Mandriva uses XAA or EXA, we do not use XAA for Intel driver any longer, since Intel as driver author does not want to support it, and it has issues with suspend and resume. The old XAA is currently better optimized than the new EXA.
Performance is Important

Performance is important for us and our engineers are working on performance improvements and try to help with known regressions.
For example, if you look at discussions about the tbench regression (LWN reports about these at “Tracking tbench troubles (http://lwn.net/Articles/304845/)” and “Tbench troubles II (http://lwn.net/Articles/307746/)“, you see a couple of Novell engineers involved including Rafael Wysocki and Jiri Kosina - and we have run quite some benchmarks to help track and fix these problems.
Call for Action

I’m interested to see what’s we can do to improve the performance and invite you to discuss on the openSUSE-factory mailing list what to do. Contribution in benchmarking and figuring out bottlenecks is important as well.
Conclusion

So, whom does the problems hit - and how hard? So far, I see two culprits: The ext3 filesystem with barriers enabled and the Intel graphics driver. If you’re not using either, you’re on par with other recent distributions. http://lizards.opensuse.org/wp-includes/images/smilies/icon_smile.gif
The ext3 filesystem regression can be worked around at the risk of filesystem integrity with disabling barriers. I doubt that I will hit this during normal desktop usage on my system.
The Intel issue has been discussed already at the openSUSE-factory mailing list and it was even suggested to not ship openSUSE 11.1 because of this. It hits some Intel graphics driver users quite card
and I wonder why I don’t see it on my machine. It might be better/worse on specific graphics chip.
If this issue is fixed with a new driver that works in our 11.1 setup, we will consider doing an update via our update repositories.

[spkert]

How about benchmarks on lighter distros ?
Zenwalk, Vector, Puppy and PC-BSD ?

[fxfuji]

I'd love to see in the test suite a benchmark to measure Flash video playback performance and resource consumption (CPU and RAM) using the Adobe Flash plugin (for Linux, Windows, etc.). I doubt it would happen since the plugin isn't available on all platforms, and besides it's not open-source.

However, such a test would resolve (in mind at least) whether a Atom-based Linux netbook could reasonably be used for web-surfing sites featuring Flash video (YouTube, Hulu, etc.)

[deanjo]

I'd love to see in the test suite a benchmark to measure Flash video playback performance and resource consumption (CPU and RAM) using the Adobe Flash plugin (for Linux, Windows, etc.). I doubt it would happen since the plugin isn't available on all platforms, and besides it's not open-source.

However, such a test would resolve (in mind at least) whether a Atom-based Linux netbook could reasonably be used for web-surfing sites featuring Flash video (YouTube, Hulu, etc.)

Well this wouldn't be a problem for the ION systems in windows as Flash now supports nvidia's Purevideo. As far as vdpau support for Flash in linux I'm still waiting for a reply from Adobe on that if it is ever going to be incorporated in their linux versions.