Autonomously Finding Performance Regressions In The Linux Kernel

[phoronix]

Phoronix: Autonomously Finding Performance Regressions In The Linux Kernel

Last weekend a few Phoronix benchmarks were underway of the Linux 2.6.32-rc5 kernel when a very significant performance regression was spotted. This regression caused the PostgreSQL server to run at about 18% of the performance found in earlier kernel releases. Long story short, in tracking down this performance regression we have finally devised a way to autonomously locate performance regressions within the Linux kernel and potentially any Git-based project for that matter. Here are a few details.

http://www.phoronix.com/vr.php?view=14285

[BlackStar]

Very nice, PTS is getting better and better!

[droidhacker]

Amazing.
That this thing could actually be used to do something *useful* :eek:

[Apopas]

Terrific idea!

[rehabdoll]

rc-5 was a dog. caused some nice fs-corruption on my /

[pvtcupcakes]

So this would be completely automatic? Meaning you could go out to lunch and not be anywhere near the machine while PTS tracks down the regression?

rc-5 was a dog. caused some nice fs-corruption on my /

Seemed to happen to me in rc4 as well. I use Gentoo, and I booted up one day and my /etc/profile.env got all messed up somehow. After fixing that from a livecd, I booted up again and ended up with a hole bunch of corruption, and lost a bunch of files. Pidgin preferences, /etc/portage/package.use, some KDE configuration stuff, etc. Oh and I lost my world file, so Portage forgot all the packages I had installed.

[Michael]

So this would be completely automatic? Meaning you could go out to lunch and not be anywhere near the machine while PTS tracks down the regression?

Yes, that is how it works in PTS Bardu 2.2.

[mirza]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure mount switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

[myxiplx]

Wow, just wow.

I'm no programmer, but this is an amazing piece of work. I can see this being an invaluable tool.

[JeanPaul145]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure kernel switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

I was thinking the same thing. I think I'll stick with Ext3 for now :D
(I have had Ext4 corruption before, so I simply don't trust the fs).

[bugmenot]

Wow, this is amazing work. I wasn't paying much attention to PTS thinking big deal, so it is cool, but is it really important? But now, HELL YEAH!

Just the other day I was remembering something I read and thinking, while Windows and Mac has proper regression tests, who does proper regression tests in Linux? Well, step up to the plate PTS, at least for performance regressions. Yaaay!! PTS FTW.

[Veerappan]

Very cool. I'm not working on any projects that would benefit from this directly, but I can see using this to help when I want to report bugs regarding performance regressions in mesa (or other projects).

[kraftman]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure mount switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

What are you talking about? Afaik slower mode - ordered is used science 2.6.31. In 2.6.32 they're replacing some I/O related parts and maybe that's why there's such slowdown. Just wait for stable release. You've got to be kidding if you consider Ext3 is faster - afaik in some previous benchmarks Ext3 was using writeback mode (with possibility of occasional file corruption) while Ext4 was using ordered mode (really_slow_PostgreSQL, but I wonder if this is a meaningful test).

[Apopas]

XFS rocks! :D

[mtippett]

Just the other day I was remembering something I read and thinking, while Windows and Mac has proper regression tests, who does proper regression tests in Linux? Well, step up to the plate PTS, at least for performance regressions. Yaaay!! PTS FTW.


Could you expand on your "proper regression tests" for Win and Mac?

Bisection is extremely powerful, but unfortunately, most people don't really know or understand it.

Now of course to blow you mind, IMHO, *any* unexpected change (beneficial or detremintal) are regressions. If you get an extra 20% in performance when you weren't expecting it, there is either something bad happening elsewhere - or - you hit some good Juju accidently, and you should really understand why and get some more.

Regards,

Matthew

[Ant P.]

Can't speak for OS X, but Windows' idea of a regression test is RTM.

[frische]

mtippet: they have a few halls of machines running unit tests every (week|day|?). i guess you could call that regression testing.


on the topic; please read this and the three weblinks as well:
http://en.wikipedia.org/wiki/Delta_Debugging

[kernelOfTruth]

XFS rocks! :D

yeah but only if you're not running anything exotic underneath (e.g. encrypted home with LUKS, ...)

that leads with 100% certainty to system corruption / no return -> new installation

and I can't afford to install my system from scratch only because of an occasional hardlock (fglrx :rolleyes: ) and the filesystem which means that it has to zero all files or much files out because of security concerns :D


@mtippett:

I just saw that catalyst 9.10 will get out during the next few hours (if it hasn't already): my box always hardlocks if I do the following

sudo su (or su)

then time sync && reboot

or

time sync && shutdown

if I forgot to enter a command before wanting to reboot I e.g. would enter df -h, genlop -t ... (just before it switches to terminal)

for omitting having to rebooting into my system again and it hardlocks

-> no magic sysrq key, etc. ...

this didn't happen with nvidia's or the opensource radeon driver (afaik) so I don't know why fglrx is insisting on complete "silence" while shutting down

hope you understand what I mean

thanks


@on topic:

nice ! so in the future the kernel devs can set up some test boxen to continuously check kernels for regressions and subsequently bisecting it automagically :)

[mutlu_inek]

Awesome feature! Congratulations, Michael!

[mtippett]

mtippet: they have a few halls of machines running unit tests every (week|day|?). i guess you could call that regression testing.


Okay. I thought you meant something public that end-developers could use. Regression farms, yeah, most good companies will have that, independent of OS.


on the topic; please read this and the three weblinks as well:
http://en.wikipedia.org/wiki/Delta_Debugging

With continuous integration or at least the ability to simulate it after the fact, this appears as though it is a good unclean regression mechanism. Have you experience in it?

Regards,

Matthew

[sabriah]

Amazing.
That this thing could actually be used to do something *useful* :eek:

Raining hard is it? ;)

[xianthax]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure mount switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

possibility of data corruption on power fail is there any time write caching is enabled on the disk, regardless of file system.

if you have a system you care that much about performance or data integrity on you certainly have a battery backup and power line conditioning UPS hooked to it right? in which case you can safely disable the extra integrity protection.

[mtippett]

possibility of data corruption on power fail is there any time write caching is enabled on the disk, regardless of file system.

if you have a system you care that much about performance or data integrity on you certainly have a battery backup and power line conditioning UPS hooked to it right? in which case you can safely disable the extra integrity protection.


Data Integrity and performance are at opposite sides of a scale. The correct "default" slides around from somewhat secure to somewhat fast. The main sort of focus seems to be moving towards you might lose data, but at least make sure that you lose data in a consistent manner.

Barriers and so on go a long way to "chunk" the written data, balancing between caching and synchronous writing (a barrier guarantees that the data before the barrier is written before the data afterwards, rather than each forcing in-order data to be written.

I don't think we'll ever get to a "right" balance point, it will always be wrong in some way.

As a user, you have two choices, either accept the default (and hence the "best judgement/best awareness" of the maintainers), or tune to your requirement (either performance at the risk of data, or integrity at the costof speed).

Regards,

Matthew

[dlang]

git has the command 'git bisect run <script>' that will automate the bisect and testing process (the script returns one code if the test passes, a different on if it fails, and a third one if the test was unable to be run for some unrelated reason)

I'm curious if you took advantage of this function, or if you ended up recreating it (or most of it) in your code?

[Michael]

git has the command 'git bisect run <script>' that will automate the bisect and testing process (the script returns one code if the test passes, a different on if it fails, and a third one if the test was unable to be run for some unrelated reason)

I'm curious if you took advantage of this function, or if you ended up recreating it (or most of it) in your code?

No, the git bisect run portion is not being used within this module, since it doesn't suit the requirements for PTS.

[dlang]

No, the git bisect run portion is not being used within this module, since it doesn't suit the requirements for PTS.

interesting.

what was it lacking? (so that I can pass it on the the git developers as a possible enhancement)

how do you handle the case where a kernel picked by the bisect can't compile, crashes on boot, etc? (this is one thing that git bisect run does have a mechanism to handle)

what do you do if new compile options appear as you bisect?

one extreme case of this a year or so ago was that a bunch of compile options were moved into a submenu, with the menu needing to be selected before the other options would work. (this broke a lot of people's processes, and there never was a good solution for it found that I know of)

please do not take this the wrong way, I am not trying to attack you for building this feature. I am just trying to point out land mines that other people discovered doing this so that you can work on fixing them before they blow up on you.

[mtippett]

Bisection is CM agnostic, hell, you don't even need CM. You just need the following


An ordered list with identifiers
A way to set up a system based on the identifier
Something to run that generates a quantative result
A fulcrum (my term) that you want to detect


Assuming that the ordered list is has a single transition you can do all sorts of funky things.

Determine optimum cluster size for a filesystem


Cluster size (512,1k,2k,4k,8k)
mke2fs
PTS doing a benchmark of some sort
A perfomance value you can't go outside


Somewhat contrived, but if you have hard performance criteria, and want to balance that against size. The above would work, just set it up, and a bisection would be able to tell you what cluster size meets your requirement.

Work out when a driver slowed down a 2D operation


Driver releases (CAT 9.1, 9.2, 9.3)
curl to download, script to install, reboot
PTS doing a benchmark of some sort
A known before and after value for a benchmark



Of course defect tracking through bisection is easiest to grok, and is usually the best value for a developers time :).

[mtippett]

interesting.

what was it lacking? (so that I can pass it on the the git developers as a possible enhancement)

how do you handle the case where a kernel picked by the bisect can't compile, crashes on boot, etc? (this is one thing that git bisect run does have a mechanism to handle)

what do you do if new compile options appear as you bisect?

one extreme case of this a year or so ago was that a bunch of compile options were moved into a submenu, with the menu needing to be selected before the other options would work. (this broke a lot of people's processes, and there never was a good solution for it found that I know of)

please do not take this the wrong way, I am not trying to attack you for building this feature. I am just trying to point out land mines that other people discovered doing this so that you can work on fixing them before they blow up on you.

It shouldn't be git-centric. Perforce, svn, cvs, driver releases, etc can all get some love.

The "setup" ie: download, build, install stage can be rigged to do different things based on different builds. ie: if after this commit, setup this way, otherwise set up this way.

It does bring in a danger that you end up commiting a cardinal sin modifying two variables, (the setup and the commit point), but in some cases you can't avoid it.

[znmeb]

Phoronix: Autonomously Finding Performance Regressions In The Linux Kernel

Last weekend a few Phoronix benchmarks were underway of the Linux 2.6.32-rc5 kernel when a very significant performance regression was spotted. This regression caused the PostgreSQL server to run at about 18% of the performance found in earlier kernel releases. Long story short, in tracking down this performance regression we have finally devised a way to autonomously locate performance regressions within the Linux kernel and potentially any Git-based project for that matter. Here are a few details.

http://www.phoronix.com/vr.php?view=14285

This is awesome! Can anyone just run this, or do I need to buy something?

[misiu_mp]

Autonomous (btw, why not automatic) regression testing is super useful.
How about extending it for regression testing wine with a set of windows programs? This could at least test for crashes. You could add automatic screen shots to get some of the functionality testing done too.

[sreyan]

Autonomous (btw, why not automatic) regression testing is super useful.
How about extending it for regression testing wine with a set of windows programs? This could at least test for crashes. You could add automatic screen shots to get some of the functionality testing done too.

http://test.winehq.org/data/

[Michael]

This is awesome! Can anyone just run this, or do I need to buy something?

It's all open-source, but this module is just not something you can "just run". First you need to create a build script for whatever you wish to test, then module-setup bisect option to set all of the options, etc. Documentation for it will come once everything has been stabilized and new features added in. Of course, normal phoronix-test-suite is extremely easy to use and free.

[mtippett]

Autonomous (btw, why not automatic) regression testing is super useful.
How about extending it for regression testing wine with a set of windows programs? This could at least test for crashes. You could add automatic screen shots to get some of the functionality testing done too.

Ordered list, setup, test, fulcrum.

The screenshot testing is a considerably more difficult, and really needs application support to do it nicely.

[znmeb]

It's all open-source, but this module is just not something you can "just run". First you need to create a build script for whatever you wish to test, then module-setup bisect option to set all of the options, etc. Documentation for it will come once everything has been stabilized and new features added in. Of course, normal phoronix-test-suite is extremely easy to use and free.

Thanks! I've spent the last week or so doing Iozone testing on openSUSE 11.2 RC1 and I've automated as much of it as I can. But your suite is better than Iozone. ;-)

[Michael]

Thanks! I've spent the last week or so doing Iozone testing on openSUSE 11.2 RC1 and I've automated as much of it as I can. But your suite is better than Iozone. ;-)

Too bad you didn't discover:

phoronix-test-suite benchmark iozone

A week ago :)

[znmeb]

Too bad you didn't discover:

phoronix-test-suite benchmark iozone

A week ago :)

I actually did discover Phoronix a few months ago, tweeted and blogged about how wonderful it was, then promptly forgot about it and moved on to something completely different - social media analytics research. ;-) It's only been since openSUSE 11.2 Milestone 8 that I've started to care about Linux I/O performance again. ;-)

[mtippett]

I actually did discover Phoronix a few months ago, tweeted and blogged about how wonderful it was, then promptly forgot about it and moved on to something completely different - social media analytics research. ;-) It's only been since openSUSE 11.2 Milestone 8 that I've started to care about Linux I/O performance again. ;-)

As an OT diversion, do you have any references to your social media analytics research?

[znmeb]

As an OT diversion, do you have any references to your social media analytics research?

http://borasky-research.net/smart-at-znmeb

[kraftman]

@Michael

Why there are ia32-libs "needed" in PTS while I'm using 64bit OS (Kubuntu 9.10)?

When I wanted to install pgbench test, PTS wanted my password to install ia32-libs. I typed it and then, there was such error:

sudo: apt-get -y --ignore-missing install ia32-libs: command not found

however test is installed and working, but every time I want to run it, it displays this:

The following dependencies will be installed:
- ia32-libs

This process may take several minutes.
sudo: apt-get -y --ignore-missing install ia32-libs: command not found

[Michael]

@kraftman:

Some of the tests (particularly some of the games) require some of the ia32-libs, so I just added that check on there if running 64-bit to always fetch those 32-bit libs. However, for Ubuntu 9.10 and others where it is no longer present, I will make a workaround.

[kraftman]

@kraftman:

Some of the tests (particularly some of the games) require some of the ia32-libs, so I just added that check on there if running 64-bit to always fetch those 32-bit libs. However, for Ubuntu 9.10 and others where it is no longer present, I will make a workaround.

Thank you :)

[dlang]

@kraftman:

Some of the tests (particularly some of the games) require some of the ia32-libs, so I just added that check on there if running 64-bit to always fetch those 32-bit libs. However, for Ubuntu 9.10 and others where it is no longer present, I will make a workaround.

command not found sounds more like it's not finding apt-get, not that ia32-libs is not there.

[kraftman]

There's a small mistake in the article:

All that this PTS module needs to know is the starting and end points that return a different state, an accessible repository (Git, currently) to be tested to be tested,

[llama]

Wait, this means that if I want to have consistent data on ext4 I must look at 5x slower PostgreSQL and other write-intensive applications? And proposed solution is either obscure mount switch with possibility of occasional file corruption or really _slow_ PostgreSQL. WTF?

... Or get a battery-backed SATA controller, like 3Ware's RAID cards.

Although as others have said, it's the write cache in the disk that's the problem so only hard power-loss can cause the corruption that this change prevents. If you have a UPS, you can run with nobarrier. Linux crashing won't stop the disk from performing the buffered writes. (at least AFAIK. Is that true even if the crash causes a machine reset? i.e. on typical x86 hardware, does the system reset signal do anything that could flush the write buffer in a SATA hard drive?)

If Linux just locks up and stops sending SATA commands, you're fine, because the disk will finish processing its queue.

[dlang]

... Or get a battery-backed SATA controller, like 3Ware's RAID cards.

Although as others have said, it's the write cache in the disk that's the problem so only hard power-loss can cause the corruption that this change prevents. If you have a UPS, you can run with nobarrier. Linux crashing won't stop the disk from performing the buffered writes. (at least AFAIK. Is that true even if the crash causes a machine reset? i.e. on typical x86 hardware, does the system reset signal do anything that could flush the write buffer in a SATA hard drive?)

If Linux just locks up and stops sending SATA commands, you're fine, because the disk will finish processing its queue.

unless you mount the partition with the sync option it is not safe to run without barriers even with a UPS.

if te kernel goes haywire it may just stop sending commands, or it may lock up the SATA bus (or send garbage to the drive)

but you are absolutly right that if you want good database performance you need battery backed cache, a 7200 rpm drive can only do ~160 seperate writes/sec, anytime you get numbers much higher than that you are seeing buffering take place, and if you don't have a battery on that buffer it means that a power failure could end up with things written in the wrong order, which may not be recoverable.