Autonomously Finding Performance Regressions In The Linux Kernel

[bugmenot]

Wow, this is amazing work. I wasn't paying much attention to PTS thinking big deal, so it is cool, but is it really important? But now, HELL YEAH!

Just the other day I was remembering something I read and thinking, while Windows and Mac has proper regression tests, who does proper regression tests in Linux? Well, step up to the plate PTS, at least for performance regressions. Yaaay!! PTS FTW.

[Veerappan]

Very cool. I'm not working on any projects that would benefit from this directly, but I can see using this to help when I want to report bugs regarding performance regressions in mesa (or other projects).

[kraftman]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure mount switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

What are you talking about? Afaik slower mode - ordered is used science 2.6.31. In 2.6.32 they're replacing some I/O related parts and maybe that's why there's such slowdown. Just wait for stable release. You've got to be kidding if you consider Ext3 is faster - afaik in some previous benchmarks Ext3 was using writeback mode (with possibility of occasional file corruption) while Ext4 was using ordered mode (really_slow_PostgreSQL, but I wonder if this is a meaningful test).

[Apopas]

XFS rocks!

[mtippett]

Just the other day I was remembering something I read and thinking, while Windows and Mac has proper regression tests, who does proper regression tests in Linux? Well, step up to the plate PTS, at least for performance regressions. Yaaay!! PTS FTW.

Could you expand on your "proper regression tests" for Win and Mac?

Bisection is extremely powerful, but unfortunately, most people don't really know or understand it.

Now of course to blow you mind, IMHO, *any* unexpected change (beneficial or detremintal) are regressions. If you get an extra 20% in performance when you weren't expecting it, there is either something bad happening elsewhere - or - you hit some good Juju accidently, and you should really understand why and get some more.

Regards,

Matthew

[Ant P.]

Can't speak for OS X, but Windows' idea of a regression test is RTM.

[frische]

mtippet: they have a few halls of machines running unit tests every (week|day|?). i guess you could call that regression testing.


on the topic; please read this and the three weblinks as well:
http://en.wikipedia.org/wiki/Delta_Debugging

[kernelOfTruth]

XFS rocks!

yeah but only if you're not running anything exotic underneath (e.g. encrypted home with LUKS, ...)

that leads with 100% certainty to system corruption / no return -> new installation

and I can't afford to install my system from scratch only because of an occasional hardlock (fglrx ) and the filesystem which means that it has to zero all files or much files out because of security concerns


@mtippett:

I just saw that catalyst 9.10 will get out during the next few hours (if it hasn't already): my box always hardlocks if I do the following

sudo su (

or su)

then

time sync && reboot

or

time sync && shutdown

if I forgot to enter a command before wanting to reboot I e.g. would enter

df -h

,

genlop -t

... (just before it switches to terminal)

for omitting having to rebooting into my system again and it hardlocks

-> no magic sysrq key, etc. ...

this didn't happen with nvidia's or the opensource radeon driver (afaik) so I don't know why fglrx is insisting on complete "silence" while shutting down

hope you understand what I mean

thanks


@on topic:

nice ! so in the future the kernel devs can set up some test boxen to continuously check kernels for regressions and subsequently bisecting it automagically

[mutlu_inek]

Awesome feature! Congratulations, Michael!

[mtippett]

mtippet: they have a few halls of machines running unit tests every (week|day|?). i guess you could call that regression testing.

Okay. I thought you meant something public that end-developers could use. Regression farms, yeah, most good companies will have that, independent of OS.

on the topic; please read this and the three weblinks as well:
http://en.wikipedia.org/wiki/Delta_Debugging

With continuous integration or at least the ability to simulate it after the fact, this appears as though it is a good unclean regression mechanism. Have you experience in it?

Regards,

Matthew