Observations On Long-Term Performance/Regression Testing

[phoronix]

Phoronix: Observations On Long-Term Performance/Regression Testing

At the Ubuntu Developer Summit later this month in Orlando for the Ubuntu 11.04 "Natty Narwhal" release, it looks like performance testing may finally be discussed at length by Canonical and the Ubuntu developers...

http://www.phoronix.com/vr.php?view=ODY0Nw

[grigi]

Michael, I don't think the issue is the ability to track performance, but more the decision on what minimal set of metrics to track.

Even if you can provide a graph, someone needs to analyze the results, and report to it on human terms.

I'm thinking for this to be at all effective, it needs to be managed on a case-by-case basis, like how Ubuntu papercuts is managed.
That needs buy-in from the people upstream.

I suspect the issue is more of a political one, than a technical one. Policy & process need to be defined, else what happens is that the people in-fight as to whom is at blame, or whom is right, whom's metric is more important (especially if a change affects one metric positively and another negatively)

Trust me, political a nightmare, people are.
Argue they do, on the same side of the fence, and none dares admit it.

[mtippett]

Since I threw together the extremely rudimentary observations, I can make comments .

Michael, I don't think the issue is the ability to track performance, but more the decision on what minimal set of metrics to track.

The discipline of long term testing is something that is missing in most organizations - professional or open source. A lot of the times, it is ensuring there is systemized and ingrained testing is the biggest hurdle. Once you have the testing part of the way that the business is done, you suddenly optimize in the tests, and not bicker about testing.

As the systems evolve, the minimal set of metrics evolve as well. It's better to have the system running with tests that add some value rather than not testing because you can't find tests with extreme value. Further, the more extreme value that you extract from a measure, the more likelihood that it becomes singular in purpose.

It is extremely hard to maintain a system that maintains a set of metrics across a broad set of subsystems. Even harder to tune.

Even if you can provide a graph, someone needs to analyze the results, and report to it on human terms.

I'm thinking for this to be at all effective, it needs to be managed on a case-by-case basis, like how Ubuntu papercuts is managed.
That needs buy-in from the people upstream.

I agree fully with that. However, when you go from simple identification of a regression to identifying which component and which team is responsible, you start needing to mesh with the development process for the team. Right now, we can detect the change, and it is up to the people driving the systems (be it kernel or ubuntu) to go the next level.

For the most we can automate down to the package level for the Ubuntu distribution testing, and the daily down to the kernel level. We are planning extensions to the PTS and Phoromatic that will allow for easy integration.

Note that as an external effort, we are just using Ubuntu as a convenient system to test theories and concepts on. We'd love to go deeper into working the Canonical and the community, but time will tell on that.

I suspect the issue is more of a political one, than a technical one. Policy & process need to be defined, else what happens is that the people in-fight as to whom is at blame, or whom is right, whom's metric is more important (especially if a change affects one metric positively and another negatively)

Trust me, political a nightmare, people are.
Argue they do, on the same side of the fence, and none dares admit it.

Exactly. We are trying to lower the effort of integration and automation. This of course beyond a purely technical issue and can't be solved mechanically.

Thanks though for your considered response.

[grigi]

Hi Matthew

You raise very interesting points. I only know on testing to make sure my stuff works well, a way to efficiently test some things completely elude me :P

Hmmm, maybe you could write an article regarding a culture of disciplined testing? I think a lot of Open-Source projects need better testing processes. Maybe expand on some processes that work reasonably well, and when automated testing is not possible (or not viable). A lot of developers are against "testing" because they hear "unit testing", which works only in a very narrow spectrum, and some manager want you to have close to 100% test coverage (which means nothing, how do you test that a data model is flexible for your incredibly creative users?)

Being a prior manager at ATI, I suspect you might know more about that than almost anybody working on Open-Source?

Regards

[mtippett]

Michael and I are working through ways that we can improve the testing within the community.

One thing to consider here, the issue that comes up way too often is people obsess about the value of what is being tested, rather than starting testing and optimizing.

This is analogous to people not wanting to start building a bridge until they had the perfect materials to build from. Usually it's better to start build a bridge with whatever is usable and safe, and then look to optimize and improve once you have can make your way across.

Matthew+

[grigi]

Is that akin to the the old argument of "waterfall" projects, versus rinse-and-repeat "agile" projects?

Best of luck! I'm definitely interested in what you come up with.