Five Years Of Linux Kernel Benchmarks: 2.6.12 Through 2.6.37

We used the Threaded I/O Tester to carry out eight threads of 64MB random writes. While the numbers were volatile along the way, with the Linux 2.6.37 kernel the threaded random write performance in this configuration was at 8.15MB/s where as in the Linux 2.6.12 kernel it was 5.83MB/s.

Looking at the time to compress a 256MB file using Parallel BZIP2 compression, the performance fluctuated by less than one second the entire time as PBZIP2 performance is largely CPU-bound and the kernel had little influence on this program's performance.

When this testing began last week, it really was unknown what to expect since we have never carried out a Linux kernel performance comparison to this scale before nor have we ever seen such a report. Some users and developers that we talked to beforehand bet the performance of Linux kernel would drop over the past five years as the Linux kernel gets increasingly large and more cruft is added, but in most areas, this is actually not the case. However, at the same time, in few areas did the Linux kernel performance actually increase over the 26 versions of the Linux 2.6 kernel that were benchmarked.

As the Linux kernel aged, the performance improved in some areas like with John The Ripper, Himeno, code compilation performance, PostMark, FS-Mark, and the Threaded I/O Tester. At least this was the case with the x86_64 Linux kernel when a KVM-virtualized copy of Fedora. Where the Linux kernel is left being slower at this point is with GnuPG, Loopback TCP Network Performance, IOzone, and in other areas on a more miniscule scale. For many of the application benchmarks the Linux kernel advancements caused little performance change, at least for a multi-core x86_64 system running in a virtual machine.

Fortunately, in the future such large comparisons may be obsolete since the introduction of Phoromatic and for nearly a year have been monitoring the Linux kernel performance on a daily basis, so organically we will have these historical measurements in real-time going forward across multiple systems. Going forward, this kernel test farm and its capabilities will only continue to grow. We also do benchmarks on a daily basis for the latest Ubuntu packages as do other independent organizations leveraging our software for their specific needs.

For some of these regressions, we will be reporting on their precise causes in a future article; seeing as, after all, we are funded by advertising with page views plus premium subscriptions, PayPal tips (also in the form of Augustiner beer at Oktoberfest), and affiliate shopping links. One of the many Phoronix Test Suite capabilities is the ability to automatically bisect a Git tree (and other revision control systems) looking for performance and functional regressions, which we have already used to find performance regressions within the Linux kernel down to the precise commit in an automated manner. Combine these modularized capabilities plus those from other Phoronix Test Suite and Phoromatic features we will make public in the near future and to professional customers, it becomes a powerful utility to trivially locate such issues with little manual intervention on the user's part.

Again, special thanks go out to Intel for their support and supplying the Intel Core i7 970 "Gulftown" that made benchmarking the past 26 Linux kernels possible in a timely fashion, especially when it comes to bisecting these regressions with building even more kernels. With the Core i7 970 and a solid-state drive, it is possible to build the Linux kernel in about four minutes or less. We are also in the process of doing a similar type of performance comparison, but on the compiler side with GCC, DragonEgg, and LLVM/Clang, to see how the performance has evolved as they play a much greater role in the performance of user-space programs. These compiler results will be out next week for the Core i7 and other systems.