As a developer of 0 A.D., I'll have to say I quite like 0 A.D. It's far from the most technically impressive engine, but it's getting better (I've been adding some more usable shader support recently), and the artists are doing a great job
I'm interested in adding a usable graphics benchmarking mode into it some time, to compare across different hardware and OSes and drivers (mainly to identify performance-related bugs in our code or in drivers, and to choose sensible default graphics settings), so it could be good to get input from people with experience developing or running benchmarks. I think the biggest issue is that performance depends heavily on implementation detail - e.g. we can render with GLSL shaders or with equivalent ARB shaders (GL_ARB_fragment_program etc), with similar performance in most environments, but the GLSL is ~30% faster than ARB on proprietary NVIDIA drivers (for unknown reasons). Or e.g. changing the implementation of alpha testing can make rendering significantly faster on some Intel GPUs, and slightly slower on other Intel ones, and no different on NVIDIA ones. How can we do a fair comparison when there are so many unpredictable variables?
Maybe the best approach is for the benchmark to try every combination and report the best? but what if drivers are buggy and some combinations don't even give correct rendering, so fastest isn't best? We already collect a load of data from the game (
OpenGL capabilities,
in-game FPS, etc), so it's easy to collect and analyse whatever data the benchmark generates, but I'm not really sure what the best approach would be.