The Linux 2.6.23 kernel has been released today and we have some preliminary benchmarks of the 2.6.23 kernel as we compare it to the past Linux 2.6.22 kernel. We will have more on the Linux 2.6.23 kernel once we have tested it more extensively, but the benchmarks we have ran so far include Quake 4, LAME encoding, Gzip compression, and RAMspeed. If you missed it, among the features for the Linux 2.6.23 kernel include the CFS process scheduler (Completely Fair Scheduler), a variety of virtualization improvements, on-demand read-ahead, and XFS and EXT4 file-system improvements are among the interesting changes.
First off, lemme say I love the fact that Phoronix takes on hardware as it relates to Linux. That kicks ass, 2 topics I'm quite fond of. However, I think your approach is a tad off. In the article you highlight the significant changes between the .22 & .23 releases (good) then demonstrate a series of graphs which are all within a per % of each other (boring). I the reader then observe that performance hasn't changed much overall and go on with my life feeling completely indifferent about the new kernel (bad). I'm not trying to troll here or anything but you seem to be taking on benchmarking like every other hardware site out there when they get a windows release. Unfortunately that doesn't seem to work well with the subject matter you are tackling.
My recommendation for benchmarking a new kernel would be to take a look at the new features users would care about (ext4 tweaks, CFS, etc) and benchmark these features specifically. Ext4 would be easy, something like bonnie, tar, compiles, etc. CFS would be a little tougher because it would involve loading the system with multiple tasks but a quick review of the (rip) ck mailing list should be able to provide some utilities to test this. Virtualization, kinda tough, but doable.
Hope this doesn't sound too negative. I am one of those few people who reads about new hardware and wonders how my Linux distribution will run on it. Keep up the hard work.
Good to see some benchmarks. However as was stated, CFS was designed for multitasking, while favoring more interactive tasks.
From kernelnewbies.org (http://kernelnewbies.org/Linux_2_6_23)
"NOTE!: Applications that depend heavily on sched_yield()'s behaviour (like, f.e., many benchmarks) can suffer from huge performance gains/losses due to the very very subtle semantics of what sched_yield() should do and how CFS changes them. There's a sysctl at /proc/sys/kernel/sched_compat_yield that you can set to "1" to change the sched_yield() behaviour that you should try in those cases. It must be also noticed that CFS is also available as a backport for 2.6.22, 2.6.21 and 2.6.20."
It would be interesting to see some benchmarks with sched_compat_yield enabled.
one good test would be to measure system behaviour when there are sudden cpu usage spikes.
i encountered one when switching to console from X while mpd was playing - sound would stop for a second. in general sound playback would get interrupted pretty often when cpu usage would get heavy.
good thing is - it doesn't happen anymore on 2.6.23. it could be cfs, it could be switch to uvesafb from vesafb-tng in my case. it could be something else. who knows. i can see the difference. and i'm happy about it.
at least , the important part of information we can extract by looking at those graphs indicates that the 'overall' performance of CFS scheduler is, if not better, at least side by side with the regular scheduler methods, so for me that's an actual advance
I'd be more interested, in my case, to watch actual performance numbers at tasks like gaming, but not games like doom3 or UT but more classic applications like xmame/sdlmame, zsnes, gens, pSX, pcsx2 or any other kind of emulator prone to be 'tampered' by a poorly-grained implementation of process scheduling (not saying it in a concrete way, just trying to imagine a worst-case scenario)