1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems

Facebook RSS Twitter Twitter Google Plus

Phoronix Test Suite

OpenBenchmarking Benchmarking Platform
Phoromatic Test Orchestration

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org


Published on 07 December 2010 12:23 PM EST
Written by Michael Larabel in Phoronix
Add A Comment

For those not tracking the Phorogit code of the Phoronix Test Suite, all of the test profiles and test suites living within our open-source benchmarking software have been removed. All 130+ test profiles and ~50 test suites have been dropped, as they're no longer living within the benchmarking package.

Why? They're moving to OpenBenchmarking.org. As talked about in late October about how Linux benchmarking will change, with Phoronix Test Suite 3.0 "Iveland" we are are introducing OpenBenchmarking.org as the next-generation Phoronix Global implementation that far trumps the current capabilities. The OpenBenchmarking.org infrastructure provides public and private storage of test result data (including system logs and benchmark outputs) and effective collaboration tools for sharing results and efficiently comparing multiple test result sets. OpenBenchmarking.org also provides a package management system for making accessible new, updated, and third-party test profiles / suites to the users of the Phoronix Test Suite. Plus a lot more.

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org

With OpenBenchmarking.org and Phoronix Test Suite 3.0+, the test profiles and suites live out in the "cloud" and are pulled down automatically in a manner like a package management system. This provides a number of benefits, including:

- You no longer need to wait for a new version of the Phoronix Test Suite to be released to gain access to new or updated test profiles in an easy-to-use manner. Now you can instantly tap into new/updated test profiles in a seamless manner upon their upload to OpenBenchmarking.org. As long as the test profiles (or test suites) don't depend upon capabilities that require an updated version of the Phoronix Test Suite (pts-core will automatically resolve incompatible tests/suites), you can opt to install/upgrade.

- Upgrading your Phoronix Test Suite client no longer requires you to use the new/updated test profiles and suites. This is great especially in the enterprise environment where you may be tracking the performance of hardware across product generations or benchmarking the latest software stack on a daily basis over a long term (i.e. with Phoromatic and using Phoromatic Tracker to benchmark the Linux kernel daily) and would prefer to stick with the same exact test version for a number of months or even years. Now you can still upgrade the Phoronix Test Suite to gain access to any new pts-core features or other improvements while still binding your testing to a particular test profile or suite version.

- Multiple versions of a test/suite can be installed simultaneously.

- If running a performance comparison of your system against someone else's system, you can ensure now that you have the same exact version of the test(s) installed that are used in the comparison, regardless of the Phoronix Test Suite release you are using.

- Moving to OpenBenchmarking.org ensures that third-parties can easily publish their own test profiles while gaining visibility. Users will be allowed to upload their own test profiles and suites (you can also build your own test suites easily through the web-interface!) to OpenBenchmarking.org and they will be treated in the same way as the "official" test profiles and suites. Users will not have to jump through any hurdles or manually setup any files no longer to utilize third-party resources. When you upload results to share on OpenBenchmarking.org using one of these custom/third-party tests, the Phoronix Test Suite can automatically fetch the needed profiles/suites automatically too!

- Plus some other features being explored that aren't yet ready to be announced ;)

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org

Right now in the latest Git code there are still a few bugs and issues being worked out, but there will be a bigger call for testing beginning with the Phoronix Test Suite 3.0 Alpha 4 release later in the month.

Below are some notes on the technical implementation of the OpenBenchmarking.org package management system implementation in the Phoronix Test Suite.

- With Phoronix Test Suite 3.0, you still can run tests just like you have done since Phoronix Test Suite 0.1. Running a command like phoronix-test-suite benchmark apache will work just fine. The official Phoronix Test Suite test/suite repository (simply named "pts") is enabled by default in Iveland as the primary package source. Internally, this evaluates to the equivalent of running phoronix-test-suite benchmark pts/apache-1.1.0. The Phoronix Test Suite scans the available repositories for the matching name and if no version for the test profile is supplied, it will default to the latest. Though if you want to run an older version of a test, you can simply supply it in the phoronix-test-suite benchmark repository/test-version format.

- For third-party repositories you can add the repository to be part of your default index (thereby no longer needing to supply the repository/ portion, unless the same test name happens to exist in pts/ or a higher-level repository). Otherwise you can always install/upgrade a test or suite by simply append the repository prefix (i.e. phoronix-test-suite benchmark some-hardware-vendor/new-opencl-test).

- This same string evaluation also occurs within result files and test suites. This allows you to either define a particular test profile version to use when running a test suite or to always just use the most recent test profile version as its default. This also means all test profiles and suites are treated equally and you can even mix and match tests/suites from different OpenBenchmarking.org repositories.

- For those enterprise users running the Phoronix Test Suite on an intranet where the test nodes do not have access to the Internet, it will be possible to copy over the OpenBenchmarking.org files and import them locally. There is also HTTPS communication support.

Plus there's many more features on the horizon for the package system along with OpenBenchmarking.org (result collaboration, interactive and embed-able graphs, etc) and PTS Iveland in general. Stay tuned for more information and we invite your feedback. As said earlier, "OpenBenchmarking.org is a new online component for benchmarking that will launch in conjunction with Phoronix Test Suite 3.0 "Iveland" early next year. While it's not even out publicly yet, I expect OpenBenchmarking.org to eventually become bigger -- both in terms of traffic and even greater industry importance -- than Phoronix.com itself." The official release is coming in Q1'2011.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux News
  1. Linux 4.2 Bringing Support For ARCv2, HS38 CPU Cores
  2. Libdrm 2.4.62 Is An Important Update For Open-Source GPU Drivers
  3. The State of Unity 3D Game Engine, Editor On Linux
  4. ZFS On Linux Brings Linux 4.1 Support, Fixes
  5. Old Net Burst Tests, Ubuntu Phone & Assembly x86 Were Popular Topics Last Month
  6. Qt 5.5 Officially Released
  7. Global Shortcuts In KDE Plasma Under Wayland
  8. LLVMpipe FP64 Support Knocks Off Some GL4 Extensions
  9. Dell Gets An Airplane Mode Switch Driver In Linux 4.2
  10. I Gave Up Waiting On The Water-Cooled Radeon R9 Fury X
Latest Articles & Reviews
  1. How KDE VDG Is Trying To Make Open-Source Software Beautiful
  2. Attempting To Try Out BCache On The Linux 4.1 Kernel
  3. CompuLab's Fitlet Is A Very Tiny, Fanless, Linux PC With AMD A10 Micro
  4. AMD A10-7870K Godavari: RadeonSI Gallium3D vs. Catalyst Linux Drivers
Most Viewed News This Week
  1. Kubuntu 15.10 Could Be The End Of The Road
  2. NVIDIA Starts Supplying Open-Source Hardware Reference Headers
  3. KDBUS Won't Be Pushed Until The Linux 4.3 Kernel
  4. The Staging Pull For Linux 4.2: "Big, Really Big"
  5. The State & Complications Of Porting The Unity Editor To Linux
  6. SteamOS "Brewmaster" Is Valve's New Debian 8.1 Based Version
  7. Jonathan Riddell Steps Down From The Kubuntu Council
  8. ARM Posts Pictures Of AMD's New Development Board