Why Software Defaults Are Important & Benchmarked

Almost every time benchmarks are published on Phoronix, there's always at least a handful of people - or more - that will right away say the benchmarks are flawed, meaningless, or just plain wrong. Why? Because the software configuration is tested with its default (stock) settings. These users then go on to say that the defaults are not optimized for performance and that "everyone else knows better" to use a particular set of options, etc. But it's my firm belief that it's up to the upstream maintainer -- whether it be the project itself developing the software in question or the distribution vendor that's packaging and maintaining the given component -- to choose the most sane and reliable settings, and that's what most people use. In addition, with open-source software, there's endless possibilities for how a given piece of software can be tuned and tweaked. Here's some numbers confirming these beliefs of testing software at its defaults.

One vector of OpenBenchmarking.org that's not been talked about much up to this point and that's the system logs that can be optionally attached when uploading test results. While the option is given to the user whether to upload their system logs and select configuration files with their benchmark results, hundreds of users have opted to do so in the past week and a half that OpenBenchmarking.org has been active. This provides a fairly nice set of data to look at for starters.

The average user of OpenBenchmarking.org / Phoronix Test Suite is also likely not the absolute Linux newbie but rather experienced and would know about various tweaks and options to optimize one's performance, seeing as this is testing software targeted at enthusiasts, gamers, and enterprise users -- not a brand new Linux user looking for Compiz/Kwin plug-ins and the like.

Take for example this morning's article looking at open-source ATI graphics in Ubuntu 11.04. Within hours of that article being published, there were already responses like: "The benchmark is flawed. He's running the the opensource drivers with SwapBuffersWait on, for maximum fps it must be turned off. It's like having a street car race where one car (radeon) stops for every red light while the other one (fglrx) just ignores them. Of course the car which ignores the red lights is going to win (or crash..)."

It was back in January when I looked at the performance of the latest ATI Gallium3D code once the KMS page-flipping support was merged into the Radeon DRM for Linux 2.6.38. In that article, in looking at that feature, it was tested in a stock configuration and when SwapBuffersWait was disabled (the feature is designed to reduce screen tearing at the cost of performance). But in today's article, the subject in question is the stock Ubuntu 11.04 performance as what someone sees "out of the box" when upgrading or performing a new install.

There's some users in support of my default methodology, e.g. "The benchmark is good. Those are the defaults. This benchmark is only about the default drivers on Ubuntu. Users don't know stuff about swapbuffers and whatnot. They simply use the stuff at the recommended defaults supplied by Canonical." Exactly.

There's also some users in the middle, "That is a decent argument. It's the default behaviour that people will get without tweaking, but one has to admit that the default Ubuntu behaviour is really stupid (Compiz always on). Still, it would be nice to benchmark what the drivers can actually do at this moment, especially if we're talking about trivial changes. The way it is right now, you have to add 60% to every number you publish for open source drivers, and this can't be the point of a benchmark. What about a "default" benchmark (no tweaks) and a "best" benchmark (minor one-liner tweaks that everybody uses anyway)?"

But just how much is "everybody" in terms of "minor one-liner tweaks that everybody uses anyway"? Well, based upon the OpenBenchmarking.org system logs and configurations collected, it's not everyone.

Of the people that opted to submit their system logs/configuration files when uploading test results to OpenBenchmarking.org, only 60.3% of them even had a xorg.conf file where the X.Org driver options are most often stored.

When looking at these xorg.conf files, what was the most common option line? Some performance hack? Nope, it was just the option to force enabling of DPMS (Display Power Management Signalling) for the monitor(s). Of those with xorg.conf files, the DPMS option line was there 42% of the time. The other most common xorg.conf option lines for setting up NVIDIA's TwinView, disabling Emulate3Buttons, and various other common input-related options.

On a performance note, only 5.8% of those with xorg.conf files had the CoolBits option for enabling NVIDIA overclocking of their GPUs with the proprietary driver. For those with xorg.conf files, only 3% of them had the option to enable color-tiling for their graphics driver (granted, the early ATI GPUs with R300g have color tiling enabled by default). 2.3% of users had the AccelDFS option, and finally, only 2.25% of users had the SwapbuffersWait line in their xorg.conf. SwapbuffersWait is just used by xf86-video-ati 6.14+ and with the recent Radeon kernel DRM, but the percentage of those on the latest ATI stack that are using OpenBenchmarking.org is much higher than that. This percentage is only counting the percent of people who even had a xorg.conf file, in reality, the overall figure is less than one percent of people having this option. And this is with people caring to run Linux benchmarks, not the very common Ubuntu user-base.

With Phoronix file-system benchmarks it's also often criticized for using default mount options for the given file-system. Fortunately, OpenBenchmarking.org has the /proc/mounts too for those that submit their system logs/configuration files.

For those that use the EXT4 file-system, 82% of the EXT4 file-systems were mounted with their distribution's defaults. The rest of the time the mount options were scattered with no clear "third-party winner" in terms of popular non-default mount options each being 1~2% or less. Had these numbers been for Linux users as a whole, the defaults would have likely carried even a greater percent. (For those wishing to tap this data themselves, an external OpenBenchmarking.org API will come.)

Hopefully this makes more sense and confirms the beliefs on why the defaults are most important and represent the largest group of users. It's also not that I am opposed to using anything but the defaults, as in articles looking at a particular feature other options are tested, e.g. the swap buffers wait, color tiling, and page-flipping back in January. On the file-system side there's also been tests of Btrfs space cache, file-system disk compression, etc.

Matthew Tippett and I have also made it clear that I am more than happy to run more tests, if the software projects under tests themselves would like to publicly document what they recommend as the settings for best performance. It's been proposed frequently. Yet not a single project has taken us up on that offer yet. How do the developers, who should have the most knowledge on the safe options and those for yielding the best performance, expect their normal users and even distributors / package vendors to magically pick the best values if they don't even communicate it themselves via even a simple page on the Wiki. As these numbers show in this article, clearly most people are content with the defaults.

It's also worth pointing out the five stages of benchmark loss by Matthew Tippett.