Linux 2.6.24 Through Linux 2.6.33 Benchmarks

[phoronix]

Phoronix: Linux 2.6.24 Through Linux 2.6.33 Benchmarks

At Phoronix we have been benchmarking the Linux kernel on a daily basis using Phoromatic Tracker, a sub-component of Phoromatic and the Phoronix Test Suite. We launched our first system in the Linux kernel testing farm just prior to the Linux 2.6.33 kernel development cycle and found a number of notable regressions during the past three months. Now with the Linux 2.6.34 kernel development cycle getting into swing, we have added an additional two systems to our daily kernel benchmarking farm. One of the systems is an Atom Z520 system but what makes it more interesting is that the system is using a Btrfs file-system and then the second new system added to the kernel tracker is a 64-bit setup. However, to provide a historical look at the Linux kernel performance, we have ran some fresh benchmarks going back to the Linux 2.6.24 kernel and ending with the recently released Linux 2.6.33 kernel.

http://www.phoronix.com/vr.php?view=14633

[Xheyther]

I can't believe such big regressions like apache where left untouched and unresolved during so much release. I don't understand either why the PostgreSQL performance regression passed through the rc phase... Either you just prove to the worl the kernel development model is flawed, either their is a flaw in the tests you ran. In both case, some analysis and explanation would have been nice along the results.

[kraftman]

I can't believe such big regressions like apache where left untouched and unresolved during so much release. I don't understand either why the PostgreSQL performance regression passed through the rc phase... Either you just prove to the worl the kernel development model is flawed, either their is a flaw in the tests you ran. In both case, some analysis and explanation would have been nice along the results.

Usually those tests are meaningless. If there's intentional change in the kernel it's described here like regression. In example some file system has different mode set as default in newer kernel and this can make big impact on some benchmarks - PostgreSQL etc. Apache benchmark is meaningless, because it's not done properly.

[kraftman]

Apache benchmark is meaningless, because it's not done properly.

http://lkml.indiana.edu/hypermail/li...9.1/02631.html

Be careful not to run ab on the same machine as you run apache, otherwise
the numerous apache processes can limit ab's throughput. This is the same
reason as why I educate people so that they don't run a single-process
proxy in front of a multi-process/multi-thread web server. Apparently
it's not obvious to everyone.

[Jimbo]

I like this article a lot!! testing from 24-33, When I first read the Phoromatic Tracker i was wondering if that could be done. Phoromatic Tracker is a superb tool! well done.

I always thought that there will be regression on performance from 24 to 33, because kernel is implementing more APIs, graphics code, graphics focus... but this article shows me that i was worng.

[n0nsense]

I was very excited about PTS at the beginning.
Now it produces meaningless numbers. your conclusions based on the PTS results could be wrong.
Like with HD test. It's results leading to very wrong assumption. 20$ CPU can not play HD movie with high bit rate.
I can't find testing file systems with different settings producing something useful.
You can use some tests to evaluate some components , but i can't see the point to post this 24-33 tests. they are at least misleading.

[Xheyther]

I always thought that there will be regression on performance from 24 to 33, because kernel is implementing more APIs, graphics code, graphics focus... but this article shows me that i was worng.

This article mainly shows that tests are often meaningless.

[korpenkraxar]

It would be nice to hear heavy users/developers of PostgreSQL to chime in on these numbers. There must be businesses around to which performance of an order of magnitude less would mean disaster.

OR is this just another case of Phoronix relying on the "default" Ext3 settings for each kernel and not bothering to compare the same settings across kernels to generate FUD and page hits?

[bugmenot]

I'm interested in the special commit that made PostSQL run so much faster and slower again. And I also miss the feature you described a few weeks ago, that you can see the deviation of every single run if you move the mouse over the graph.

All in all interesting I think, thank you

[mtippett]

http://lkml.indiana.edu/hypermail/li...9.1/02631.html

Skip to the next response to that thread..

I turned on apache, and played with ab a bit, and yup, ab is a hog, so
any fairness hurts it a badly. Ergo, running ab on the same box as
apache suffers with CFS when NEW_FAIR_SLEEPERS are turned on. Issuing
ab bandwidth to match it's 1:N pig nature brings throughput right back.


http://lkml.indiana.edu/hypermail/li...9.1/02861.html

Remember that you can't test anything, and testing in the obvious path will usually result in flat lines - since they represent the 95% path.

As indicated above, what has been identified is that in some scenarios CFS completely tanks. The ab is just a tool to make this visible.

As per usual, if there is any benchmark which you believes provides a suitable equivalent scenario but is more "correct", please tell us.