cyclictest, interbench, and framerate jitter measurements. cyclictest measures scheduler latency, interbench is supposed to measure interactive latency, and better framerate stats will show actual game playability/video watchability a lot better than just the average framerate.
Specifically, for meaningful latency results, cyclictest reports should show min, avg, max, and stddev. Games should have their framerate checked multiple times per second (or ideally just measure the interval between every call to glSwap()), with min, avg, max, and stddev frame times (single frame time is 1/fps, so instantaneous fps is 1/frame time).
Also, a histogram of the game frame times and cyclictest/interbench latency measurements will make it really easy to see what the typical latency will be like. A 3D plot where histograms from various kernels are stacked next to each other will allow comparing kernel versions as part of the ongoing tracking.
I also think tracking wine's performance would be really interesting and could potentially benefit the project in the long run.
However, at least for me, using wine together with the xf86-video-ati driver (r300) is basically near impossible, as far as running games(3d) is concerned, especially since switching over to the gallium/dri2/kms stack.
I'd assume that any performance improvements or regressions that could be measured when rendering direct3d/opengl would be more down to the driver side of things than to wine's code.
Then again, a comparison to using fglrx under the hood would levitate this discrepancy I guess...
Probably, other, non-render related tests would be more showing.
We are seeking requests for trackers to continuously monitor performance, not individual performance article benchmarks. Post those requests in another thread as I will just be ignoring them in this thread.
I think asking people to stick to tracker suggestions is an exercise in futility considering the title of the news article.
Many people, myself included, upon reading the title "Default What Shall We Benchmark Next? Let Us Know!" think you were asking for suggestions on what you will benchmark (as a "one time" thing) and immediately rushed to post with an opinion, skipping the actual article contents in the process.
I also like the idea of monitoring the performance of some non-Ubuntu distribution, especially one which is specifically designed to be a rolling release (Arch or Gentoo or even Debian Sid), with latest STABLE software, not the way Ubuntu fluctuates between buggy new untested software and old and therefore slow software.
Arch would be awesome Except there is the slight problem that sometimes it needs user input and configuration to upgrade or continue running after an upgrade. I don't know anything about Gentoo, but I think this could be a good option. Sid would be too much like Ubuntu.
My top pick would be a tracker for the open video drivers (specifcally intel, radeon and nouveau) but I understand the difficulty considering libdrm and kernel compatibility issues to which you alluded.