Phoronix Test Suite 4.4: Blasting New Features With Forsand

The first development release of Phoronix Test Suite 4.4-Forsand is now available for those interested in open-source benchmarking and automated testing. This major quarterly update is poised to introduce new features that will benefit all users from hardware enthusiasts to enterprise customers.

It's been just over one month since the release of Phoronix Test Suite 4.2-Randaberg, so the first development milestone of 4.4-Forsand has been released as the Q1'2013 update is prepared. This release doesn't exclusively focus upon improvements in one particular area but there's work going on across most areas of our open-source testing platform that powers more than one million benchmarks annually. The Git version of Phoronix Test Suite 4.4 is taking great shape and is already used in the production benchmarking at Phoronix.com.

The Phoronix Test Suite 4.4 codename is "Forsand", a city within Rogaland county, Norway. Forsand follows the codename theme of Phoronix Test Suite 4.0-Suldal and 4.2-Randaberg with being named after regions within Rogaland -- a theme that will continue with future Phoronix Test Suite 4.x releases. This builds upon the long-standing rhythm of Phoronix Test Suite 1.x codenames originating from Sør-Trøndelag, 2.x codenames from Troms, and 3.x codenames from Aust-Agder. The current plan is to release Phoronix Test Suite 4.4 officially in late February or early March while several more development "milestone" releases will be made available in the next few weeks. Among the highlights so far include:

New & Updated Tests

With OpenBenchmarking.org integration since Phoronix Test Suite 2.0-Sandtorg, test profiles and suites are no longer tied to specific versions of the Phoronix Test Suite client but remotely managed in a package-management-like solution. There's been some new test profiles released recently plus updates to other test profiles, so while they don't require you to be on Phoronix Test Suite 4.4, they are deserving of being noted.

13 New Intel Benchmarks - As written about last week, coming out of Intel's Open-Source Technology Center are 13 new test profiles covering systemd boot performance monitoring, performance testing of system libraries, and other workloads that Intel has internally used the tests for in comparing different Linux distributions.

Parboil - The Parboil software package from the IMPACT Research Group at University of Illinois has been incorporated as the newest test profile. Parboil provides OpenMP, OpenCL, and CUDA scientific tests. With the initial test profile release, selected test options include Cutoff Pair Potential, Matrix Q, MRI Gridding, Seven-Point Stencil, Lid-Driven Cavity Fluid Dynamics, and Two-Point Angular Correlation computational tests.

N-Queens - The N-Queens test profile has been updated to ensure that the compiler honors the OpenMP support.

Enhanced Error Reporting

Up to now when a test/benchmark has failed to properly install, the user would be notified and the log-file of the failed installation would be preserved. The user could then choose to dig through the log-file in hopes of seeing what's wrong -- if the upstream benchmark fails to build on a given compiler, if there happens to be some build dependency missing from the target system, or some other problem. With Phoronix Test Suite 4.4, the user will still be notified of failed installations and the log file(s) preserved, but the Phoronix Test Suite client will try to be a bit more helpful. The pts-core engine now has support for automatically and intelligently analyzing failed log files in an attempt to try to quickly determine the problem. The log files of tests that fail to properly install will be immediately scanned to look for common problems like a configre/autogen.sh script generating an error, commands not being found on the system, missing header files for tests to be compiled from source, and other common issues. A concise error string is then returned to the user at install-time about the likely problem. Enhanced error reporting will continue to be worked on through the Phoronix Test Suite 4.4 cycle and there have already been a few more improvements that landed in Git this morning for 4.4 Milestone 2.

As with all Phoronix Test Suite work, this enhanced error reporting will seamlessly work for any test written against our extensible testing architecture. Included here are screenshots showing off a few of the conveniently reported errors.

Support Checks

Phoronix Test Suite 4.4 brings a new optional feature to the test profile definitions of a "support-check" hook. Test profiles can now choose to ship a "support-check" script to analyze the target system's state prior to test installation. Up to now the support checks built into the Phoronix Test Suite have been for checking against the system's operating-system/distribution, architecture, necessary package dependencies, and other common and defined attributes. With the new "support-check.sh" handling, if a test profile ships the file, it will be called upon by pts-core to evaluate the system state. If the test profile determines the system is incompatible with the test profile, it can write its own custom error string to the file specified by the TEST_CUSTOM_ERROR environment variable that the Phoronix Test Suite sets. The Phoronix Test Suite will then deem the test not supported on the system and pass the custom error message to the user.

This extra support check has been intended to be added for a while, but was necessitated with the publicizing of the new Intel test profiles that are designed to make use of the system-provided library/state. For example, the systemd boot performance tests have support-check hooks to verify the system being tested is in fact using systemd and not another init daemon.

Phodevi

The Phoronix Device Interface (Phodevi), the multi-platform hardware/software abstraction layer used by the Phoronix Test Suite, continues to receive various fixes and other improvements. With Phodevi in Phoronix Test Suite 4.4 Milestone 1, there's improved detection of the video RAM capacity when using the Nouveau DRM driver on recent Linux kernel releases (they changed the vRAM reporting string recently), a Radeon GPU frequency detection fix for systems with integrated AMD graphics and for older graphics cards, and support for checking the GALLIUM_MSAA environment variable to see if a multi-sample anti-aliasing level was forced by user-space.

Forsand

Phoronix Test Suite 4.4 Milestone 1 also features a variety of other improvements. The enhancements include but not limited to improved external dependency handling, fallback detection for operating system lineage based upon the installed package manager, a concise public TODO list for the Phoronix Test Suite client has been made available, and the list-available-tests / list-available-suites sub-commands are now faster to display the results.

Expect Phoronix Test Suite 4.4 Milestone 2 to be released within the next two weeks. The latest release of the Phoronix Test Suite can be downloaded from Phoronix-Test-Suite.com. Commercial support, behind-the-firewall installations of OpenBenchmarking.org, and other custom enterprise services continue to be offered to organizations by contacting us, or for companies simply wishing to sponsor/support this open-source multi-platform benchmarking framework. Feature requests and other suggestions and comments can be directed to the forums.

Below is the short change-log for Forsand Milestone 1.

Phoronix Test Suite 4.4 Milestone 1
23 January 2013

- pts-core: Introduce 'common-dependencies' concept to PTS External Dependencies
- pts-core: Fix reporting of some already installed dependencies
- pts-core: Support for listing 'local/' tests from the list-supported-tests command
- pts-core: Support for test profiles providing a 'support-check.sh' file
- pts-core: Support for reporting the Graphics AA/AF values to the system notes table
- pts-core: Add a public version of the TODO list
- pts-core: Don't attempt copying install.log to install-failed.log on empty files
- pts-core: Add fallback to allow matching an operating system to an external dependencies package list based upon the detected package manager
- pts-core: When anonymous usage reporting is enabled, report test installation failures to OpenBenchmarking.org for analyzing problem
- pts-core: Support for detecting common errors within test profile install failures and reporting the relevant information to the user
- pts-core: When installing many tests and there are install failures, conveniently show the errors at the end
- pts-core: When installing multiple tests, report any unknown test profiles/suites
- pts-core: Faster loading of list-available-tests command by not forcing all test profile data be downloaded
- pts-core: Progressively download OpenBenchmarking.org in list-available-suites command to display information faster
- phodevi: Improve Nouveau video RAM memory capacity detection on newer Linux kernels
- phodevi: Warning fix for Radeon GPU frequency detection
- phodevi: Drop "special_settings_string" support
- phodevi: Support for checking if the Gallium3D anti-aliasing level was overrode using GALLIUM_MSAA environment variable