Running Benchmarks On Newer, Popular AAA Linux Games***
For anyone that's read Phoronix for any length of time or the forums know that the selection of OpenGL games/tests used for benchmarking come down to those that meet the strict criteria for being capable of being fully-automated and reproducible. The automation and reproducibility of the test -- for any subsystem, just not graphics -- is needed so that it can be controlled via the Phoronix Test Suite, so that it can be easier if then bisecting any performance regressions or problems, gives the necessary steps for those wishing to reproduce a bug or investigate a performance claim further, etc. By being Phoronix Test Suite supported, anyone in the community can independently validate the results found in Phoronix articles, see how their own system compares to the Phoronix results and compare to results found on OpenBenchmarking.org, and for numerous other reasons -- including so that it can be controlled via Phoromatic for cases like the LinuxBenchmarking.com test farm with daily real-world tests of the Linux kernel, Mesa, etc.
Metro Redux Linux GPU/Driver Tests Are Coming
The issue with newer Linux games is that many of the developers don't implement the necessary infrastructure for being able to conduct automated tests -- either a lack of a time demo / replay ability, support for auto-launching such an option, and the ability to then quit the game while dumping any related performance metrics. Such features sadly aren't too common with recent Linux games. Even for some games that claim to have such support, it ships broken. With most Linux games now being distributed through Steam is also a problem when there's no control over fetching a specific version of the game / pinning or locking the revision of the game to use for testing (in order to ensure everyone is using the same revision of the game for producing comparable numbers), having to use Steam is a no-go within some PTS customer environments due to corporate restrictions / firewalls / proxies where as HTTP/FTP file caching is much easier, etc. For driver developers there's also some value in the tests being open-source as they can dig much deeper into what's happening in the case of poor performance or a regression.
Counter-Strike: Global Offensive NVIDIA/AMD Benchmarks On Linux
Of course, I realize the need to include newer Linux games too. One of the other alternatives for handling Linux game benchmarking of titles that can't be fully automated would be to benchmark OpenGL traces -- using VOGL or APITrace to produce a dump of the OpenGL command stream while playing a game for a few seconds/minute, automating the playback process of that, and then running that on the systems under test. It can be easily automated and would work fine for stressing the graphics drivers / hardware in a relative manner.
Running Benchmarks On Other New Linux Games
Benchmarking OpenGL traces though wouldn't be representative of the real Linux gaming experience though since it's just replaying the OpenGL trace and not reproducing any of the game logic, etc. So it would only be useful for Linux gamers if wanting to know relatively if a particular driver or GPU increases/decreases by a certain percent compared to the previous state. Unless doing multiple traces of the same exact steps, these tests would lose out on the resolution and in-game quality setting options for comparison too.
In the past I've made and played around with an APITrace test profile, but it didn't go too far for the aforementioned reasons. Other issues with the OpenGL trace benchmarking is that the file sizes can become very huge quite quickly and there's a legal issue whether the traces are redistributable due to being the copyrighted game's assets.
In taking a new stab at benchmarking OpenGL traces with Valve's encouragement, traces from any Valve games are fine to redistribute through the Phoronix Test Suite. Valve would also be willing to work on getting agreements in place from other game publishers to ensure that traces of their games are fine for public redistribution.
So with all of that said, do OpenGL game trace benchmarks interest you? It would be newer graphical tests in cases where the games cannot be properly automated, but they wouldn't be representative of the real-world game performance, etc. If it interests you, what Linux games would you be most interested in seeing included for the trace benchmarks?
31 Comments
