How-To: Find The Mesa Performance Regressions
As shown last week with The Grinch That Stole The Radeon Gallium3D Performance, it's not too difficult to find regressions within the open-source Mesa / Gallium3D drivers. Well, it's not hard to find regressions in many areas of the Linux stack, but in particular for Mesa/Gallium3D as even though this is a critical piece to the Linux / free desktop desktop, they don't currently rely upon any continuous integration infrastructure or any continued performance monitoring aside from the developers mostly pushing Piglit for looking towards functional problems. Of course, I've been working on my own means to address some of these problems, the Phoronix Test Suite stack can already:
- Leverage Phoromatic (soon a new version integrated atop OpenBenchmarking.org) to continually test Mesa drivers for OpenGL performance regressions, image quality comparisons, and other functional tests. This has been demonstrated with monitoring the daily performance of the Linux kernel and various other open-source projects for several years already. The Phoronix Test Suite can also log the GPU fence counts, GPU temperatures, GPU fan speeds, CPU usage, and various other system vitals all in a fully automated manner while tests are being executed. Phoromatic could issue Mesa tests on a per-commit basis to the Git repository or on a timed (daily) basis.
- Utilize the Phoronix Test Suite's bisecting capabilities that hook into git-bisect for automatically finding regressions by looking for changes in performance from a test profile and triggering git bisect automatically. This is how last week's Radeon performance regressions were tracked, finding Linux kernel bugs, and other Linux regressions.
- A combination of solutions encompassing the Phoronix Test Suite, as is done by Wine / CodeWeavers.
- Using MATISK. (What this article is talking about.)
MATISK is a Phoronix Test Suite module I demonstrated earlier this year for benchmarks thousands of revisions of a software project (it happened to be Mesa, again, in that article). Right now I'm in the process of benchmarking (well, temporarily delayed while in Vienna this week and waiting for the January advertising campaigns to kick in before publishing data) all the revisions of Mesa since 7.11 to see what other regressions and issues are outstanding for the upcoming Mesa 7.12/8.0 release. Phoronix Test Suite with the MATISK module can do this extremely easy as it's just fed a simple INI-based file with defining a custom context, what to do with each context, etc.
I've recently made some improvements to the MATISK module too that are now in the public Git repository in preparation for Phoronix Test Suite 3.8-Bygland in Q1'2012 (so if you want to try this out, use the Phorogit code). Below is a basic MATISK example (you can also auto-generate a template using phoronix-test-suite matisk.template) that's similar to what I'm using in testing the R600 Gallium3D driver.
; Sample INI Configuration Template For Phoronix Test Suite MATISK
; //www.phoronix-test-suite.com/
[workload]
; A boolean value of whether to save the test results. The default value is TRUE.
save_results = TRUE
; A string that is an XML test suite for the Phoronix Test Suite. If running a
; custom collection of tests/suites, first run phoronix-test-suite build-suite.
suite = local/quick-regression-check
; The string to save the test results as.
save_name = mesa-regression-testing
; The test description string.
description = "Checking for Mesa regressions for a few different drivers/hardware... Future article on Phoronix.com by Michael Larabel."
; The test result identifier string, unless using contexts.
result_identifier = "$VIDEO_CARD: $MATISK_CONTEXT"
[installation]
; Check to see that all tests/suites are installed prior to execution. The default value is TRUE.
install_check = TRUE
; Force all tests/suites to be re-installed each time prior to execution.
force_install = FALSE
; The option to specify a non-standard PTS External Dependencies download cache
; directory.
external_download_cache =
; If Phodevi should not be caching any hardware/software information.
block_phodevi_caching = TRUE
[general]
; A boolean value whether to automatically upload the test result to
; OpenBenchmarking.org
upload_to_openbenchmarking = FALSE
[environmental_variables]
; The environmental_variables section allows key = value pairs of environmental
; variables to be set by default. The default value is EXAMPLE.
LD_LIBRARY_PATH = /home/phoronix/mesa-install/lib/
LIBGL_DRIVERS_PATH = /home/phoronix/mesa-install/lib/dri/
[set_context]
; The pre_install or pre_run fields must be used when using the MATISK context
; testing functionality. The set_context fields must specify an executable file
; for setting the context of the system. Passed as the first argument to the
; respective file is the context string defined by the contexts section of this
; file. If any of the set_context scripts emit an exit code of 8, the testing
; process will abort immediately. If any of the set_context scripts emit an exit
; code of 9, the testing process will skip executing the suite on the current
; context.
; An external file to be used for setting the system context prior to test
; installation.
pre_install =
; An external file to be used for setting the system context prior to test
; execution.
pre_run = "cd /home/phoronix/mesa && make realclean && git checkout $MATISK_CONTEXT && ./autogen.sh --with-dri-drivers= --with-gallium-drivers=r600 --enable-gallium-r600 --disable-egl --prefix=/home/phoronix/mesa-install/ --libdir=/home/phoronix/mesa-install/lib/ --with-state-trackers=dri && make -j12 && make install"
; An external file to be used for setting the system context after the test
; installation.
post_install =
; An external file to be used for setting the system context after all tests have
; been executed.
post_run =
; If any of the context scripts cause the system to reboot, set this value to true
; and the Phoronix Test Suite will attempt to automatically recover itself upon
; reboot.
reboot_support = FALSE
; An array of context values.
context[] =
context[] =
; An external file for loading a list of contexts, if not loading the list of
; contexts via the context array in this file. If the external file is a script it
; will be executed and the standard output will be used for parsing the contexts.
external_contexts = "cd /home/phoronix/mesa && git log mesa-7.11..master --pretty=format:%h --topo-order --grep=r600"
; The delimiter for the context_file contexts list. Special keyword: EOL will use
; a line break as the delimiter and TAB will use a tab as a delimiter. The default value is EOL.
context_file_delimiter = EOL
; A boolean value of whether to reverse the order (from bottom to top) for the
; execution of a list of contexts.
reverse_context_order = TRUE
; A boolean value of whether to log the output of the set-context scripts to
; ~/.phoronix-test-suite/modules-data/matisk/
log_context_outputs = FALSE
So what it's doing is defining various parameters for the Phoronix Test Suite (result identifier, file name, test installation settings, etc) that is similar to what's set by Phoromatic or when running the Phoronix Test Suite in its batch mode. And then also in this INI file are various parameters that are used by the MATISK module for gathering contexts. A context could be virtually anything from a Git hash, a ISO date, a software (RPM/DEB) package name, a build identifier / UUID, or any other unique identifier. It could also be an array of various kernel module parameters, file-system mount options, or really any other string. The MATISK module then calls upon the Phoronix Test Suite to execute each of these after running the phoronix-test-suite matisk.run [the-INI-file] command. At one of the hook stages, you can then call upon an external executable file or a series of commands to then do something while receiving one of the unique contexts. Right now MATISK isn't focused upon having the MATISK files be widely distributed across platforms or for setting up initial environments from scratch, but based upon the direction of OpenBenchmarking.org and other Phoronix Test Suite, you can likely see the future direction and my grand plans.
In the case of this example, the contexts is a list of Git hashes between the 7.11 tag and master for the mainline Mesa git tree that's automatically being parsed from git log. Prior to running the test queue, one of the contexts (a Git hash) is being passed to a series of commands that pulls that specific Git revision, runs the Mesa build script, and installs the new drivers locally (FYI, this INI file example is intentionally quite basic and not fully complete). Then the tests are executed and the process repeated -- all in a fully automated manner. MATISK also attempts to automatically recover itself and the Phoronix Test Suite if one of the commands or scripts would like to reboot the system, such as for trying a different kernel or other low-level configuration change.
As with all of the Phoronix Test Suite related products, it's meant to be turnkey and fully automated once configured for any process. MATISK also has support for skipping a particular context based upon a script's exit code and other various configurable parameters. Feedback, as always, is welcome.
The current MATISK module in the Phoronix Test Suite is compatible with Linux, BSD, Mac OS X, and Solaris operating systems.
Any questions can be directed to the forums. Phoronix Media is also happen to engage with leading open-source projects gratis and does offer a variety of enterprise products and services for commercial organizations interested in continuous integration, performance monitoring, system analytics, and more.
More exciting Phoronix Test Suite / OpenBenchmarking.org features are also coming forward in 2012!
[ P.S. speaking of open-source graphics driver performance: it looks like there may be a Nouveau DRM performance change concerning ctxprogs. See this revised mailing list thread for more details. ]
4 Comments
