Reasons To Make A PTS/OB Test Profile For Your Software

While there's hundreds of test profiles and test suites (a collection of pre-configured test profiles) available to the Phoronix Test Suite, here's some reasons why you should consider making/submitting a test profile for benchmarking of your own software or software that is of interest to you. The same reasons also apply mostly for why you should be benchmarking with the Phoronix Test Suite, OpenBenchmarking.org, and Phoromatic.

Creating a test profile for any arbitrary software is designed to be easy -- a test profile is effectively just a collection of bash scripts and XML files. If the software you wish to test can be easily automated from a command-line and produces some meaningful result, it can be easily adapted to the Phoronix Test Suite. If uploading that test profile to OpenBenchmarking.org, it then becomes available to all Phoronix Test Suite users out there.

If you want a look at what a test profile effectively looks like, there's some straightforward examples via some "easy" tests like C-Ray or Xonotic while there's also many other advance tests out there. One shortcoming is that I don't have exhaustive documentation to share for those wishing to know all the intricate details and possibilities, but I'm happy to explain various test profile options via our forums or by contacting us. The lack of documentation though hasn't stopped companies like Intel figuring things out and creating lots of tests.

Below is a look at various facts and reasons why you should consider coming up with test profiles for your own software -- or software in general that interests you. These test profiles can be made public (via OpenBenchmarking.org) or kept locally on systems behind-the-firewall if coming up with any proprietary/company-specific tests.

- The test profiles/suites are OS/distribution agnostic. The few other public test harnesses that sort of aim for automated benchmarking tend to be focused to a particular distribution (e.g. Ubuntu), tied to a specific package management system, or other specific hardware/software interests. The Phoronix Test Suite and the test profiles are intended to offer universal compatibility, though there's nothing stopping you from say modifying the upstream tests to make them use distribution-supplied RPM/Debian packages for benchmarks. Besides Linux, the Phoronix Test Suite also runs on BSD, Solaris, and OS X operating systems along with limited support for Windows and GNU Hurd -- it's up to the test profiles to express via the XML definitions whether the software under test supports a given platform.

- Aside from supporting all major operating systems, the Phoronix Test Suite runs and fits on most architectures from embedded ARM/MIPS devices to large super-computers. That's one of the reasons why the Phoronix Test Suite core is in PHP since the "LAMP" stack tends to be quickly ported to most new (and even obscure) architectures so in most instances it allows seamlessly running the Phoronix Test Suite on new hardware platforms without needing changes to the test client. In addition to bare metal, the Phoronix Test Suite also runs (and is very popular) in cloud-based deployments too.

- It's easy! If the software already is scripted to run a benchmark, it can often take just a matter of minutes to wrap it into a OpenBenchmarking/PTS-supported package. For companies that have their own aging, internal, crusty test systems have found it very easy to migrate to the Phoronix Test Suite with excellent results.

- Having a test profile of the software you made or are interested in makes these tests immediately available to many IHVs and ISVs who use as many tests as they can get for testing their next-generation hardware, drivers, etc. Intel, AMD, NVIDIA, and many others have publicly mentioned Phoronix Test Suite usage to varying extents. There's also countless other notable companies out there using the Phoronix Test Suite (and in cases even making additions) like AT&T, Los Alamos National Lab, National Taiwan University, Duke, General Electric, Cloud Harmony, and countless others -- including many that can't be mentioned publicly. The Phoronix Test Suite is popular for carrying out routine benchmarking, comparing hardware/software when making purchasing decisions, per-commit/daily testing of software in-development, and being adapted to a variety of other tasks which is among the reasons why it's very popular with automated testing in research papers. By writing a test profile, there's thousands of organizations that will be able to immediately use your test for burning in their systems, adding extra workloads for monitoring performance of their drivers/software, and use it for other testing purposes.

- The Phoronix Test Suite is open-source under the GPL and available via GitHub.

- Once a test profile is created, it exposes the test to all of the features of the Phoronix Test Suite, OpenBenchmarking.org, and Phoromatic components. The features range from graph rendering, easily and dynamic facilitation of side-by-side result merging, automated system log collection, compiler flag tracing, detailed system hardware and software recognition, streamlined result sharing, etc. Nothing in the Phoronix Test Suite code is tied to any specific test profile / software.

- The test profiles are then exposed to the full potential of the Phoronix Test Suite modules and its modular architecture that allows for near limitless possibilities. For example, there's the system monitoring module that makes it easy to do power efficiency testing and recording of various software/hardware vitals and sensors while running any test profile. With your few lines of bash scripts and XML, the Phoronix Test Suite can be simultaneously be logging the CPU temperature(s), memory usage, power consumption, and other sensors -- with the sensor detection itself also being easily extensible. It's all test profile agnostic thanks to the standard script format and XML test profiles. There's also other modules for matters like automated bisecting of performance regressions, pushing real-time iOS/Android notifications to your devices of the test process/results, and much more. You can also make your own modules to tie the Phoronix Test Suite in to whatever you wish; here's a simple module that by setting a simple environment variable (TIMED_SCREENSHOT) prior to running the Phoronix Test Suite will generate a screenshot every defined period of seconds while running benchmarks.

- Via Phoromatic you can trivially scale out the testing from just a single system to dozens, hundreds, or even thousands of systems. The Phoronix big basement server room farm is completely controlled via our open-source benchmarking software.

- Result sharing is made simple and standardized via OpenBenchmarking.org. You can upload your test result data (say this result file) and then anyone else running the Phoronix Test Suite can simply run one command (e.g. phoronix-test-suite benchmark 1504270-DE-REDHATHVM83) to fetch all of the relevant test data, download the same versions of the test profile(s) / benchmark(s), and run the benchmarks in the same exact manner as the original tests.

Well, that's the short list of reasons why you should consider making a test profile for your software. Numerous organizations have already replaced their test systems (often internal and hacked-about-over-the-years software scripts) with Phoronix Test Suite driven solutions. If you're interested in learning more or have questions on any topic you can contact us or post in the forums. To organizations, commercial support and custom engineering services are available.

You can try it out at Phoronix-Test-Suite.com or getting the latest code over on GitHub.