Making Compiler, Disk Testing More Reproducible

There's some enhancements to share that make compiler and disk testing more reproducible under Linux, plus many other features.

Available today is Phoronix Test Suite 3.8-Bygland Milestone 3, which should be the last development release before this major update goes gold before month's end. (Depending upon commit activity this week, it may end up being released one week from today.) Several of the enhancements to this latest Phoronix Test Suite 3.8 version are not only relevant if you are a user of this open-source benchmarking software yourself, but even if you just look for Linux / BSD / Solaris / Windows benchmark results on Phoronix and the other publications that use the Phoronix Test Suite.

As I shared last week, there's been many compiler testing enhancements made within the Phoronix Test Suite framework during the 3.8-Bygland cycle for one of the commercial customers. Fortunately, this company is letting the work be committed to the open-source upstream version of the Phoronix Test Suite for all users to benefit. Since last week there's been more compiler testing enhancements, plus some disk testing enhancements, and there will be much more work committed in the coming weeks as work on Phoronix Test Suite 4.0 heats up, which they are continuing to support.

If you didn't read last week's article, read it, but the work then was for ensuring the C/Fortran/C++ compiler flags when any test is installed is easily available and rendered on the graphs showing the test results. Plus if any of the compiler flags changed between test results, there's special enhancements for highlighting the differences, etc. Here's a very example simply showing the important compiler flags being used by a given test:
As the latest work since then, the compiler's build configuration is also being automatically and seamlessly tracked by the Phoronix Test Suite. If there's any changes in the compiler's build configuration between test, it will be shown. Here's an example of that latest development:
That's the short, parsed form of the compiler's build configuration. Clicking upon that will then yield the full CC output, for example, when available (The system logs have long been captured by the Phoronix Test Suite, this contracted work is just about exposing more of it in an easy and attractive manner). [This is all for the SVG version of the tables and graphs. If you're browser doesn't yet fully support SVG, it will fall-back to the less-rich PNG graphs.] There's also some other ongoing compiler reporting work for highlighting the differences, etc. Feedback is also always welcome.
Committed yesterday is similar support to the compiler reporting improvements, but on the disk side. If a disk test is run, the kernel's scheduler is automatically reported plus the file-system mount options for wherever the test(s) were installed, as shown on the example above. As with most Phoronix Test Suite features, it's all handled automatically.

There's more system hardware / software reporting advancements being worked on, this is just the start of the work for this PTS Commercial client that's being made available to the public. A few more items may land prior to the Phoronix Test Suite 3.8-Bygland while the rest should be ready for Phoronix Test Suite 4.0. Again, the Phoronix Test Suite has long captured these system logs and other outputs, but most of this work will be centered around making it more presentable and easier to analyze when reproducing tests.

With the Phoronix Test Suite being focused upon fully automating tests, trying out these latest features on 3.8 Milestone 3 is as easy as running a command like phoronix-test-suite benchmark c-ray postmark. That command will download and install the C-Ray and PostMark tests followed by executing them in a fully automated manner under supported operating systems.

These reporting improvements aren't all that there is though to Phoronix Test Suite 3.8 Milestone 3, but there is much more. Some of the other new features include:

- The introduction of a look-aside cache and last-minute look-aside cache. Even prior to Phoronix Test Suite 1.0, there's been support for handling a download cache within the system, on a removable storage device, or via a networked interface. This allows those with many systems or routinely installing tests to house all of their needed test files locally, in order to eliminate needing to download files remotely and thus speeds up the install/setup process. (The HTML/PDF documentation covers how to setup a Phoronix Test Suite download cache.) The new look-aside cache support now looks at other installed test profiles to compare hashes of files in their download lists in an attempt to mirror matching files to eliminate downloading multiple copies or having to re-download files when installing a newer version of a test profile in a separate location (such as when testing multiple disk drives).

- Better detection of potential premature test failure and invalid results.

- Various bug-fixes.

- Refinements to the graphing improvements and the new SVG DOM IR implementation.

- Initial work towards supporting Minix 3.2.0. I've casually been working on seeing that the Phoronix Test Suite supports Minix along side its existing Linux, BSD, Hurd, Solaris, Windows, and Mac OS X operating system support. Within this third milestone release there's some basic hardware detection support, but more work is still left to be done before Minix would be fully supported by this GPLv3-licensed software.

- The graphics event checker module has been largely re-written. This module is loaded by the Phoronix Test Suite when a graphics test is being run and ensures that afterwards the system is still left in a clean state. In particular, it checks various kernel/driver interfaces for GPU errors and also ensures the system's resolution (if changed) was restored to its proper mode.

Plus there's various other changes and this is all on top of the Phoronix Test Suite 3.8 improvements made during milestone 1 and milestone 2 with items like improved graph generation and Mac OS X 10.8 support.

Feel free to give Phoronix Test Suite 3.8-Bygland a try at Phoronix-Test-Suite.com. Also be sure to share your results, tests, and suites via OpenBenchmarking.org.

For anyone wishing to learn more about the Phoronix Test Suite for personal or commercial purposes (or are just an active Phoronix readers / community members), this weekend through the middle of next week I'll be in San Diego and La Jolla for meet-ups. The Linux Foundation Collaboration Summit is also coming up again in early April. Contact me for that or PTS Commercial for other matters.

Look for more announcements soon with the official Phoronix Test Suite 3.8 release happening in a matter of days.