Test Automation With PTS, OpenBenchmarking.org

Tonight at the Ubuntu Developer Summit I'm talking about the Phoronix Test Suite and OpenBenchmarking.org. Here's some of my notes for those unable to make it or were able to attend but would like additional information.

What Is The Phoronix Test Suite? An open-source test automation framework designed for benchmarking. The Phoronix Test Suite is designed to be extensible, multi-platform (any Linux distribution, Mac OS X, Solaris, BSDs, and even limited ports to Windows and Hurd and MINIX3), and easy-to-use while producing easily reproducible results. Everything from test download and installation to installed software/hardware reporting, test execution, and result reporting is fully-automated. The Phoronix Test Suite has been publicly available for four years now and builds upon the eight years of Linux benchmarking experience at Phoronix.com along with input from various tier-one vendors. There's over 200 test profiles available and more than 60 test suites while new, custom tests can be very easily introduced for public or private use. The Phoronix Test Suite isn't limited to solely quantitative results but can also handle image quality comparisons, pass/fail scenarios, and other abstract result types.

The Phoronix Test Suite can be easily used for automated testing in regression testing on a per-commit (or timed) basis, comparing hardware performance, comparing software performance, testing tuned vs. untuned systems, etc. The Phoronix Test Suite client is freely and openly available. There's commercial services offered for custom test development, software customizations, behind-the-firewall results storage, behind-the-firewall OpenBenchmarking.org, and other custom engineering work. The Phoronix Test Suite is already widely deployed amongst tier-one hardware vendors, universities, energy companies, financial institutions / proprietary trading firms, data centers, and much more.

What Is OpenBenchmarking.org? A complement to the Phoronix Test Suite that provides a centralized location for result storage, test profiles, and test suites. It's collaborative and allows new results to be easily shared, compared, and analyzed in different ways. OpenBenchmarking.org offers various features for result analytics, shopping assistance features, hardware indexes, etc. OpenBenchmarking.org provides as well a cloud-based package-management-like system for managing test profile versioning to ensure comparisons are properly binded to correct test versions, etc. OpenBenchmarking.org is basically a collaborative, crowd-sourced benchmarking platform that has been available since late 2010 and succeeds the former "PTS Global" implementation. There's also Phoromatic that will be offered as part of this for

Some Of The Recent and Upcoming Improvements:

- Compiler reporting improvements to show any switches/flags passed by any test profile (it's all fully automated, of course) to the C/C++/Fortran compilers when the program under test is being built (optimization flags, linked libraries, etc). There's also support for showing how the compiler being used in such comparisons is being built.

- When detecting disk comparisons, it now also highlights the various file-system mount options used for the disk partition under test. This is on top of all the reporting, system log capturing, and other safeguards the Phoronix Test Suite has long provided for ensuring sane, accurate, and reproducible testing. These reporting improvements basically just make the most relevant bits for any given result file become more easily to spot and compare rather than digging through the archived system logs.

- Likewise, now when detecting a CPU comparison is being conducted, a table showing the various CPU flags, cache sizes, and other CPU attributes is now prominently displayed beyond the other details and logs already captured.

- Lastly, when a GPU/graphics-driver comparison is detected, a complementary table appears that shows the differences in supported OpenGL extensions, GPGPU cores, and other GPU attributes to easily compare the technical features of each hardware product. More details on the CPU and GPU reporting enhancements here.

Some Of What's Coming Very Soon To OpenBenchmarking.org:

- Shopping-cart-like comparisons.

- Major search enhancements.

- Greater visibility into test profile operations.

- Phoromatic-over-OpenBenchmarking.org.

- Graphing and other interaction improvements.

Other Random Notes:

- There's more than 1.5 million computer hardware component configurations/metrics stored on OpenBenchmarking.org, more than 464,000 benchmarks were recorded on OpenBenchmarking.org, detailed statistics/compatibility information on more than 150,000 systems, and much more.

- The Phoronix Test Suite is extremely extensible and nearly any test can be added.

- The Phoronix Test Suite also supports modules, such as for system monitoring to automatically record the CPU temperature, system power consumption, memory usage, CPU usage, and other system vitals/sensors while testing is being done. From there the Phoronix Test Suite can automatically provide performance-per-Watt data and more.

- A long overdue and major update to PTS Desktop Live is going to be released around the same time as Phoronix Test Suite 4.0 with many exciting features. Details to be announced soon.

- Unlike the Canonical test products, the Phoronix Test Suite will work well on any Linux distribution or nearly any non-Linux operating system for that matter. Red Hat Enterprise Linux, Fedora, FreeBSD, and others all just work "out of the box" and will function and act the same way just as it does on Ubuntu Linux. This is very important for vendors concerned more than just being Ubuntu-specific.

- Due to the visibility of test results on Phoronix.com, there's frequent feedback from upstream software developers, hardware vendors, and other organizations to better evolve testing methods, etc. The Phoronix Test Suite is truly independent from any single hardware/software vendor.

- A test profile basically just consists of a few XML files and then a simple bash script for managing the test installation and execution. (The documentation right now is a bit limited in this area, but if anyone has any questions please contact us and happy to help out. Also happy to do custom engineering work or custom test profile development via the PTS commercial entity.)

- The Phoronix Test Suite software itself has minimal dependencies and will then dynamically install "external dependencies" based upon what's needed for a given test. The Phoronix Test Suite supports using the operating system's native package manager in these cases on nearly all Linux distributions.

- Test profiles can be easily modified in any manner to suit your needs. Ubuntu guys (as they have asked), you can easily modify test profiles to use the distribution packages where available rather than building the test itself from source.

- If any questions, feel free to ask via public or private channels!