One Month Of Monitoring The Linux Kernel Performance
Phoronix: One Month Of Monitoring The Linux Kernel Performance
For those that may have forgot, at the start of December we launched the Phoronix Kernel Test Farm to begin benchmarking the Linux kernel on a daily basis using the automated tools that we provide via the Phoronix Test Suite and Phoromatic. Towards the middle of December we then unveiled the Phoromatic Tracker, which exposes these test results in real-time to the public. Well, it's now been a month of monitoring the kernel's performance and the entire Linux 2.6.33 kernel development cycle thus far, with many interesting findings.
I always miss some more statistical information about these benchmarks. Whats the RMDS (root mean square deviation) like with 10 runs per day for example?
crispy, I miss some more statistical information about the RMDS. What is the power of the RMDS (root mean square deviation) with 10 runs per day?
Is the RMDS the correct tool for this? Is the data normally distributed? Do you want to find outliers? What do you want to do? Maybe some more robust statistics are the way to go here - http://en.wikipedia.org/wiki/Robust_statistics.
Sorry, some time since ive done these error calculations, what I was thinking of was standard deviation.
Don't worry, you are not alone.
Most people unfortunately do use standard deviation, ordinary least regression, etc.
However, these methods are very very sensitive to outliers. Their high power are only applicable under very restricted circumstances, that rarely are valid. They can be good to detect outliers, but, many wish to determine a trend or a reliable central value for few observations, and then they are really bad.
Then it will give us an idea if the changes in performance for each kernel version is a result of the code or just standard variations of the dataset...
I mean, in any type of science you do this simple data analysis...
The Phoronix Test Suite already does this if you run a result file with "phoronix-test-suite analyze-all-runs", it just is not implemented in the web interface on Phoromatic at this time. Though you can assume that it is always less than 3.5% deviation otherwise the run-count dynamically increases (see an earlier Phoronix posting about statistical significance and Phoronix Test Suite). Other stats can be added in if you 1. provide patches or 2. explain it all quite well what you would like and how.