Again, the point of the article is to compare performance of different driver versions in the different kernels. To do that Michael has chosen a setup that makes it impossible to do that in a reliable way.
This makes the article pointless. Of course, PeterKraus, if you feel that making people aware of the faults in a measurement method is pointless then just don't read it and believe the unreliable data presented by Phoronix, instead of actually trying to make future benchmarks results more reliable.
It's not a fault in "a measurement method", as the measurement of time (or frames per second) is reasonably precise and repeatable. There is no random error, as the significant errors of most of the measurements (apart from 1) are below 0.5 FPS. If anything, there might be a systematic error, but you are testing the whole system. Of course the kernel might affect the system performance, but THATS EXACTLY WHAT YOU ARE MEASURING.
Now shut up.
I understand your point. But strict technically: newer features also compose the performance of the kernel versions v3.4 through v3.8.
Furthermore, I would like to give you an example of where your reasoning goes haywire: Zcomp. It's a new feature that introduces a performance increase if exposed. Now, what would it be:
- disable feature: someone could say you are deliberately inhibiting the driver's performance
- enable feature: someone else could say you are enabling new features which makes the test invalid since you changed more than the kernel version
One way or another, both are arguments that relate to the validity of the test. HOWEVER, strict technically: only changing the kernel is testing with only changing the kernels. So this *includes* features being enabled / disabled and software behaving differently. Since these exact changes are part of these kernel versions. The fact that userspace acts differently does not matter, becuase userspace is the same piece of software for each testrun.
This is where you and I are disagreeing: newer kernel contain newer features that should be included in the test. yes or no? I say yes, because newer kernel imply newer features.
I quit now...
I do not say that newer features should not be enabled if available, but since this is a test about changes in graphic driver performance only changes in the graphic driver should be considered.
What I say is that the testbed should be chosen in a way that all that changes is the part that will be benchmarked. Anything else should be as equal as even possible. These benchmarks are specifically aimed at video driver performance, not overall system performance.
But we do not know if the video driver is all that changed, since different available functions or a change in the implementation of already existing functions may have a positive or negative impact on Unity's performance regression, which is not recognizable with this specific test setup. Therefore a testbed should be chosen that does not contain sources of possible ambiguity.
That is what I am saying, nothing more, nothing less, it just comes down to choose a WM without those performance problems and this benchmark is as good as it can be.