Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org

For those not tracking the Phorogit code of the Phoronix Test Suite, all of the test profiles and test suites living within our open-source benchmarking software have been removed. All 130+ test profiles and ~50 test suites have been dropped, as they're no longer living within the benchmarking package.

Why? They're moving to OpenBenchmarking.org. As talked about in late October about how Linux benchmarking will change, with Phoronix Test Suite 3.0 "Iveland" we are are introducing OpenBenchmarking.org as the next-generation Phoronix Global implementation that far trumps the current capabilities. The OpenBenchmarking.org infrastructure provides public and private storage of test result data (including system logs and benchmark outputs) and effective collaboration tools for sharing results and efficiently comparing multiple test result sets. OpenBenchmarking.org also provides a package management system for making accessible new, updated, and third-party test profiles / suites to the users of the Phoronix Test Suite. Plus a lot more.

With OpenBenchmarking.org and Phoronix Test Suite 3.0+, the test profiles and suites live out in the "cloud" and are pulled down automatically in a manner like a package management system. This provides a number of benefits, including:

- You no longer need to wait for a new version of the Phoronix Test Suite to be released to gain access to new or updated test profiles in an easy-to-use manner. Now you can instantly tap into new/updated test profiles in a seamless manner upon their upload to OpenBenchmarking.org. As long as the test profiles (or test suites) don't depend upon capabilities that require an updated version of the Phoronix Test Suite (pts-core will automatically resolve incompatible tests/suites), you can opt to install/upgrade.

- Upgrading your Phoronix Test Suite client no longer requires you to use the new/updated test profiles and suites. This is great especially in the enterprise environment where you may be tracking the performance of hardware across product generations or benchmarking the latest software stack on a daily basis over a long term (i.e. with Phoromatic and using Phoromatic Tracker to benchmark the Linux kernel daily) and would prefer to stick with the same exact test version for a number of months or even years. Now you can still upgrade the Phoronix Test Suite to gain access to any new pts-core features or other improvements while still binding your testing to a particular test profile or suite version.

- Multiple versions of a test/suite can be installed simultaneously.

- If running a performance comparison of your system against someone else's system, you can ensure now that you have the same exact version of the test(s) installed that are used in the comparison, regardless of the Phoronix Test Suite release you are using.

- Moving to OpenBenchmarking.org ensures that third-parties can easily publish their own test profiles while gaining visibility. Users will be allowed to upload their own test profiles and suites (you can also build your own test suites easily through the web-interface!) to OpenBenchmarking.org and they will be treated in the same way as the "official" test profiles and suites. Users will not have to jump through any hurdles or manually setup any files no longer to utilize third-party resources. When you upload results to share on OpenBenchmarking.org using one of these custom/third-party tests, the Phoronix Test Suite can automatically fetch the needed profiles/suites automatically too!

- Plus some other features being explored that aren't yet ready to be announced ;)

Right now in the latest Git code there are still a few bugs and issues being worked out, but there will be a bigger call for testing beginning with the Phoronix Test Suite 3.0 Alpha 4 release later in the month.

Below are some notes on the technical implementation of the OpenBenchmarking.org package management system implementation in the Phoronix Test Suite.

- With Phoronix Test Suite 3.0, you still can run tests just like you have done since Phoronix Test Suite 0.1. Running a command like phoronix-test-suite benchmark apache will work just fine. The official Phoronix Test Suite test/suite repository (simply named "pts") is enabled by default in Iveland as the primary package source. Internally, this evaluates to the equivalent of running phoronix-test-suite benchmark pts/apache-1.1.0. The Phoronix Test Suite scans the available repositories for the matching name and if no version for the test profile is supplied, it will default to the latest. Though if you want to run an older version of a test, you can simply supply it in the phoronix-test-suite benchmark repository/test-version format.

- For third-party repositories you can add the repository to be part of your default index (thereby no longer needing to supply the repository/ portion, unless the same test name happens to exist in pts/ or a higher-level repository). Otherwise you can always install/upgrade a test or suite by simply append the repository prefix (i.e. phoronix-test-suite benchmark some-hardware-vendor/new-opencl-test).

- This same string evaluation also occurs within result files and test suites. This allows you to either define a particular test profile version to use when running a test suite or to always just use the most recent test profile version as its default. This also means all test profiles and suites are treated equally and you can even mix and match tests/suites from different OpenBenchmarking.org repositories.

- For those enterprise users running the Phoronix Test Suite on an intranet where the test nodes do not have access to the Internet, it will be possible to copy over the OpenBenchmarking.org files and import them locally. There is also HTTPS communication support.

Plus there's many more features on the horizon for the package system along with OpenBenchmarking.org (result collaboration, interactive and embed-able graphs, etc) and PTS Iveland in general. Stay tuned for more information and we invite your feedback. As said earlier, "OpenBenchmarking.org is a new online component for benchmarking that will launch in conjunction with Phoronix Test Suite 3.0 "Iveland" early next year. While it's not even out publicly yet, I expect OpenBenchmarking.org to eventually become bigger -- both in terms of traffic and even greater industry importance -- than Phoronix.com itself." The official release is coming in Q1'2011.