Besides, what if there was a updated version of the compiler available that improves the run time performance? What would you conclude then about the performance comparison between OSes? In other words, it's still exactly the same OS -- you touched nothing of the kernel, system commands or libraries -- and yet, after updating the compiler the benchmarks "magically" got better? How can such benchmarks tell us anything at all about the OS if the OS did not change?
Yes PTS pulls from the source for it's tests. It does not however compile base system components that were used to build the base system. Many (if not most) of the tests do not come with most distributions (mostly because of BS licensing and potential legal issues).