What Linux Distribution Should Be Benchmarked The Most?
As Ubuntu deviates more and more from the "conventional desktop Linux" stack with the continued evolution of Unity, the adoption of Mir over X.Org or Wayland, and other changes to distinguish Ubuntu from the hundreds of other Linux distributions, more readers are calling for Ubuntu not to be our default testing platform.
The reason for using Ubuntu, of course, has been that by Canonical's distribution is easily the most popular desktop Linux distribution. With it being the most widely used desktop Linux distribution, the results obviously apply to the greatest number of readers. This has been the primary reason for benchmarking with Ubuntu is that it gives the hardware reviews and benchmarking articles maximum relevance -- and thus page views.
As I have repeatedly explained in the forums and elsewhere, I have nothing against benchmarking other Linux distributions. It's commonly done at Phoronix in the context of server Linux distribution comparisons, our common multi-way Linux/BSD distribution comparisons, major releases of tier-one Linux distributions, etc. These multi-way Linux distributions shown at Phoronix have shown that most Linux distributions perform close to the same -- the common changes between distributions tend to be facial and not too performance sensitive, though with some exceptions. Additionally, with the latest Ubuntu packaging, the Unity+Compiz desktop configuration no longer has a big performance penalty like it once did compared to other desktop environments. Of course, XMir will change that (at least temporarily), but we're not quite there yet.
Prior to Ubuntu's rise, Fedora (Core) was my personal favorite Linux distribution and prior to that and the original Red Hat days, and prior to that I started off with Mandrake. The reason Fedora (and some other distributions) are not benchmarked more in the ever-changing Linux landscape is their use of debug builds by default for development packages, which can negatively affect performance. Around the time of a major stable release of Fedora, there's plenty of benchmarks on Phoronix with its bleeding edge but stable and non-debug state.
Besides the other mentioned distributions I am fond of, I also really like Xubuntu a lot. On my few personal systems not devoted to production purposes and/or benchmarking, Xfce is the most common desktop environment after the GNOME3 fallout. I've also enjoyed stints with Manjaro Linux, the defunct KateOS, Knoppix, openSUSE, CentOS, etc. Contrary to the belief of some Phoronix readers, I am not some "diehard Ubuntu fan boy" and I am also not some "Canonical hater." This is just like how Phoronix readers commonly go from calling me an AMD or NVIDIA "fan boy."
Aside from the debug build situation during the development cycle, another benefit of Ubuntu is their "daily mainline kernel PPA" where Debian packages are available daily (and tagged) releases of the mainline Linux kernel. When I maintain dozens of systems in my office/labs and am routinely reformatting them, it's very convenient to simply download the very latest Linux kernel code (important for Phoronix tests) without having to build the kernel from source. Beyond saving a few minutes each time, the mainline kernel PPA is very easy to cite Phoronix readers to when asking about how to install the very latest Linux kernel, what the kernel configuration was like as found in a Phoronix article, etc (enhancing the reproducibility of Phoronix tests). I'm not aware of any other Linux distribution offering easy, daily-updated kernel binary packages. On a similar note, LLVM/Clang has recently begun offering their own daily Ubuntu packages as well, which add convenience there.
Of course, once Mir/XMir is mainline in Ubuntu, many of our graphics driver articles and GPU reviews will likely be on other Linux distributions -- in either replacement of Ubuntu testing or as a complement depending upon the situation. Phoronix testing has already shown using XMir leads to performance penalties at present, plus the Mir EGL platform support isn't even mainline yet in Mesa. With being a very frequent Mesa Git user, I don't want to deal with having to patch the Mir support on Mesa each and every time plus having to deal with any breakage. But as long as Ubuntu remains very relevant on the Linux desktop, it will likely still be used a lot for other areas like Linux kernel and compiler testing where it's more of a virgin environment.
As expressed many times already in the forums and Twitter, why Arch Linux, Gentoo, and other popular rolling-release source distributions aren't used more is that it's hard to define a good reference point. With all of the test results shared on Phoronix I try to strive on reproducibility and accuracy, but with a rolling-release distribution and/or everything built from source, it's harder for others to exactly mirror my configurations than if I am using a binary distribution and only a few defined tweaks (if any). Plus with my many different systems and constantly switching out hardware and reformatting, I want something that's quick, easy, and reproducible during setup. With being responsible for nearly all of the content on Phoronix and seven days a week working on anywhere from 1~3 featured articles per day with benchmarks plus up to 12 or more news articles per day, on top of Phoronix Test Suite / OpenBenchmarking.org / Phoromatic programming, and other tasks every day, properly utilizing time is very important...unless you want Phoronix to be another Linux site to shutdown.
Anyhow, with those initial thoughts shared, I'm interested in hearing what other Phoronix readers would be interested in seeing used as the Linux platform for software testing, so please voice your opinion in the forums (use the comments link below) or let me know on Twitter via @MichaelLarabel. Thanks!
224 Comments
