KVM Virtualization Performance With Linux 2.6.31

[phoronix]

Phoronix: KVM Virtualization Performance With Linux 2.6.31

Earlier this month at the Red Hat Summit where Red Hat Enterprise Linux 5.4 was released with support for the Kernel-based Virtual Machine. At this Red Hat event, virtualization -- particularly KVM -- and cloud computing were the most talked about topics. But how is KVM performing these days? With new virtualization refinements going into almost every new Linux kernel release, we have published a new set of KVM benchmarks using the Linux 2.6.31 kernel, to provide updated numbers against our KVM benchmarks from last year and our Core i7 virtualization numbers. This time around, we are also using a Phenom II processor for testing out the AMD-V technology.

http://www.phoronix.com/vr.php?view=14201

[leidola]

What about a comparison of different virtual machines, let's say:

qemu-kvm vs. VirtualBox (with guest additions) vs. VMWare client (with guest additions).

By the way. Was the virtual harddisk a file or a block device?

[Blinkiz]

I reinstalled Ubuntu with the latest Ubuntu 9.10 yesterday. My virtual machines was much slower than 9.04 when it comes to i/o performance. Something is wrong with the virtualization stack in 9.10. I have now switched back to 9.04.

Someone should do the test for 9.04 kvm vs 9.10 kvm.

[Raine]

SQLite though did extremely well in a virtualized setting due to a bug within EXT4 / Linux 2.6.31

What bug are you talking about? Are you meaning that is a possible bug?

Raine

[hubick]

What bug are you talking about? Are you meaning that is a possible bug?

I think the only way the performance could go *UP* by that much, is that if the virtualization layer isn't properly synchronizing writes or locking or something like that from guest to host, which is almost certainly a serious bug.

[DDevine]

I think there is an issue with VirtIO in that version of Ubuntu...

The I/O benchmarks are really not representative of a proper KVM setup. I don't know why they published these benchmarks.
Without VirtIO all the network and disk stuff is horrendous, a Google search will tell you that. They really should have used Fedora for this benchmark as well. Ubuntu has always been dodgy in terms of virtualization setups (something is always broken in my experience).
Check out the KVM enhancements in Fedora 12! Wow! (https://fedoraproject.org/wiki/Releases/12/FeatureList)

[ian.woodstock]

I think the only way the performance could go *UP* by that much, is that if the virtualization layer isn't properly synchronizing writes or locking or something like that from guest to host, which is almost certainly a serious bug.

It's not a virtualization bug it's a flaw in the testing.
What's happening is that the host is caching the writes so the data is not being written right away so it would appear to be faster.

[ian.woodstock]

Phoronix: KVM Virtualization Performance With Linux 2.6.31

Earlier this month at the Red Hat Summit where Red Hat Enterprise Linux 5.4 was released with support for the Kernel-based Virtual Machine. At this Red Hat event, virtualization -- particularly KVM -- and cloud computing were the most talked about topics. But how is KVM performing these days? With new virtualization refinements going into almost every new Linux kernel release, we have published a new set of KVM benchmarks using the Linux 2.6.31 kernel, to provide updated numbers against our KVM benchmarks from last year and our Core i7 virtualization numbers. This time around, we are also using a Phenom II processor for testing out the AMD-V technology.

http://www.phoronix.com/vr.php?view=14201

Testing it on Ubuntu probably won't yield the best results.
You might want to try on RHEL 5.4 where it's been optimized or even on Fedora 11/12

There's lots of optimizations to look at, from setting the IO scheduler to be deadline in the host, to using huge pages, cache settings etc.
Have a look at the video here
http://www.redhat.com/promo/summit/2009/highlights/ then click on the "Summit sessions" tab and then virtualization video

[mibo]

What is the outcome of this test?
Software in a VM runs slower - what a suprise ;-)

Is there a benefit of the AMD-V cpu features? How does it compare to Intel? So many interesting questions but the article has not a single answer - just meaningless numbers :-(

(Please correct me if I did not see the conclusion of the article)

[nanonyme]

What is the outcome of this test?
Software in a VM runs slower - what a suprise ;-)Wrong, according to the test SQLite ran magnitudes better under VM that native. Most CPU-intensive tasks were just as fast under VM, most hard-disk-intensive tasks were slower under VM. This would imo hint towards a need to develop better virtualization technology for hard disks.

[screemo]

Phoronix: KVM Virtualization Performance With Linux 2.6.31

Earlier this month at the Red Hat Summit where Red Hat Enterprise Linux 5.4 was released with support for the Kernel-based Virtual Machine. At this Red Hat event, virtualization -- particularly KVM -- and cloud computing were the most talked about topics. But how is KVM performing these days? With new virtualization refinements going into almost every new Linux kernel release, we have published a new set of KVM benchmarks using the Linux 2.6.31 kernel, to provide updated numbers against our KVM benchmarks from last year and our Core i7 virtualization numbers. This time around, we are also using a Phenom II processor for testing out the AMD-V technology.

http://www.phoronix.com/vr.php?view=14201

Would it be possible for you to test RHEL 5.4, so its possible to compare the two ? (the talk from redhat mentions the disk i/o as one of the key aspects they're trying to improve - and some changes have already made it into 5.4.)

[VinzC]

Hi.

I would have been much more interested in how a virtual machine performs with paravirtualized guest drivers -- unless this is what was actually done as there is no mention of using paravirtualized or virtio drivers at all.

[ian.woodstock]

Wrong, according to the test SQLite ran magnitudes better under VM that native. Most CPU-intensive tasks were just as fast under VM, most hard-disk-intensive tasks were slower under VM. This would imo hint towards a need to develop better virtualization technology for hard disks.

Wrong.
SQLite didn't run faster under VM than native.
You're just seeing misleading results because the host was caching.

[mibo]

Wrong, according to the test SQLite ran magnitudes better under VM that native. Most CPU-intensive tasks were just as fast under VM, most hard-disk-intensive tasks were slower under VM. This would imo hint towards a need to develop better virtualization technology for hard disks.

If you were right, everybody would run his/her database in a VM in a VM in a VM...

[SEJeff]

How can you test the IO performance of a virtualized Linux host and not use virtio? It is akin to testing the top speed of a porsche on the spare doughnut... It makes 0 sense.

Can you do some honest benchmarking and use virtio? These numbers are completely ignorant at best. If you want good IO on kvm, you use the paravirt drivers.

[mtippett]

How can you test the IO performance of a virtualized Linux host and not use virtio? It is akin to testing the top speed of a porsche on the spare doughnut... It makes 0 sense.

Can you do some honest benchmarking and use virtio? These numbers are completely ignorant at best. If you want good IO on kvm, you use the paravirt drivers.

I don't agree with you.

Someone lay person who is examining comparitive results are the target for most of Phoronix articles. The key is "available at arms length" benchmarking, ie: easily available - out of the box.

If you want to create a recipe that shows KVM to the the best capability, can I suggest the following

1) Identify all the steps needed for the tuned setup
2) Create a fully described list of directions for building the tuned setup
3) Execute the list, keeping track of the length of time to do it.
4) Repeat the benchmark (Michael can probably supply a PTS test suite to do so)
5) Post the collection here.

I'd be interested to see the effort and results.

[mtippett]

Wrong.
SQLite didn't run faster under VM than native.
You're just seeing misleading results because the host was caching.

The SQLite test was faster under the VM.

Although I don't have explicit information on the benchmark, it would appear that the VM is doing the operations synchronously (similar to the native execution), but the VM's disk driver is caching and batching the writes.

This would result in a faster experience under a VM, however you risk data integrity that you don't risk in the native position.

I would expect that it is a possible bug in the KVM driver stack, however the any SQLite based application would experience the performance boost.

So this sort of benchmarking does show extreme value and doesn't mislead. It highlights a questionable delta between Host and Guest, and consequently requires deeper examination.

The net result is that currently, out of the box KVM support is slow for most operations except some CPU bound operations, but even more concerning it potentially risks data by not making synchronous IO operations asynchronous.

Regards,

Matthew

[nanonyme]

The net result is that currently, out of the box KVM support is slow for most operations except some CPU bound operations, but even more concerning it potentially risks data by not making synchronous IO operations synchronous.Seems I interpreted IO wrong then. But yeah, I do agree that KMV is likely slower for most operations. (covering real-life usage that this test was related to only slightly) The virtualization extensions should only really be helpful for the CPU bound operations as far as I've understood.
I'd have assumed native should cache and batch too though unless you explicitly flush.

[wolf550e]

Wrong, according to the test SQLite ran magnitudes better under VM that native. Most CPU-intensive tasks were just as fast under VM, most hard-disk-intensive tasks were slower under VM. This would imo hint towards a need to develop better virtualization technology for hard disks.

If you LD_PRELOAD a library that implements fsync() as a no-op, you can get the same speedup on the host, without virtualization! I suggest you use this (http://www.flamingspork.com/projects/libeatmydata/) handy library someone provided. Heck, it might be better to just patch your glibc and not mess with pesky preloading. Instant speedup!

You obviously don't know what you're talking about. Read a little. Actual technical stuff, not "benchmarks" done by a monkey trained to run the single-click "suite" without knowing what is measured or what it means.

For the record, that result just means all their numbers are invalid and completely useless.

[nanonyme]

You obviously don't know what you're talking about. Read a little. Actual technical stuff, not "benchmarks" done by a monkey trained to run the single-click "suite" without knowing what is measured or what it means.That single case was brought up as a peculiarity, not as a recommendation to use SQLite under VM. It kinda jumps up when you've immensively better results in VM than native, that's what I meant with that first bit. It was a single case where VM seemed faster so I picked it up. Naturally native being tons faster than VM is uninteresting (except for CPU-bound tasks) since it's expected. What I continued with was that I intuitively assumed that reading an SQLite database file inside an image file on the hard disk would be slower than just reading a file on the hard disk. (on latter thoughts I'm not even sure if this matters, might not even significantly increase fragmentation)
Also read my message again: My conclusion was that virtual I/O needs to develop, not that real I/O needs to take example of virtual I/O. Even though the basis for my conclusion was wrong, the conclusion was apparently right.

[mtippett]

What about a comparison of different virtual machines, let's say:

qemu-kvm vs. VirtualBox (with guest additions) vs. VMWare client (with guest additions).

By the way. Was the virtual harddisk a file or a block device?

That would be lots of fun with each camp saying you need to tune each host and each guest further to make it a fair comparison :). Hell, throw in Xen just for good measure :).

Matt

[mtippett]

That single case was brought up as a peculiarity, not as a recommendation to use SQLite under VM. It kinda jumps up when you've immensively better results in VM than native, that's what I meant with that first bit. It was a single case where VM seemed faster so I picked it up. Naturally native being tons faster than VM is uninteresting (except for CPU-bound tasks) since it's expected. What I continued with was that I intuitively assumed that reading an SQLite database file inside an image file on the hard disk would be slower than just reading a file on the hard disk. (on latter thoughts I'm not even sure if this matters, might not even significantly increase fragmentation)
Also read my message again: My conclusion was that virtual I/O needs to develop, not that real I/O needs to take example of virtual I/O. Even though the basis for my conclusion was wrong, the conclusion was apparently right.

You've captured my personal interest in benchmarking with this post.

1) Benchmarking for numerical comparisons is somewhat borderline and interesting
2) Outliers bear particular interest and investigation
3) Investigation identifies areas for improvement/correction/isolation

PTS has a growing list of lots of these cases, where there is a "disbelief -> anger -> investigation -> acceptance -> change" cycle.

Matt

[mtippett]

For those people who have been on this thread, I dug around with the KVM and Ubuntu qemu-kvm maintainers.

It looks like "write-back" caching is turned on by default vs the recommended "write-through".

This increases performance and usablity, but it appears that it ultimately ignores requests for synchronous fileIO. Although this is the default configuration of Ubuntu currently, it effectively renders Ubuntu as not suitable for high-reliability workloads.

I have raised a defect in launchpad

https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/437473

So the test was useful, and since it was clearly different than the other results, it has borne value by investigating.

Michael's assertions within the article are still correct. If you want a default SQLite install to absolutely fly, run it under KVM under Ubuntu. But be aware of the risk to your data - I personally don't expect Michael to invest in the way that I did to understand and present data for each and every unusual result.

Regards,

Matthew

[VinzC]

Apart from raising a defect in Ubuntu, I see no interest in this benchmark. Who might see some interest in virtualization but companies? These companies would certainly not waste their forces in a system that is a) not fine tuned for virtualization at its best and b) based on a defect making the system unreliable. Maybe I'm too negative but that's how I see it.

[mtippett]

Apart from raising a defect in Ubuntu, I see no interest in this benchmark. Who might see some interest in virtualization but companies? These companies would certainly not waste their forces in a system that is a) not fine tuned for virtualization at its best and b) based on a defect making the system unreliable. Maybe I'm too negative but that's how I see it.

I personally use virtualization to run Windows within a window under Linux. I switch between Vista and XP by just starting an image. With virtualbox, you can also access USB which gives you driver access to USB devices.

There are other reasons such as "near-disposable images" that you can copy rather than re-install, there is also the experimentation options by installing into a VM too.

These is definitely a user-oriented thing rather than corporate.

The trigger for the caching policy change was users complaining about performance. So I can't speak for Ubuntu, but there is a reasonably strong pull for them to invest.

Regards,

Matthew

[cowmix]

For those people who have been on this thread, I dug around with the KVM and Ubuntu qemu-kvm maintainers.

It looks like "write-back" caching is turned on by default vs the recommended "write-through".

This increases performance and usablity, but it appears that it ultimately ignores requests for synchronous fileIO. Although this is the default configuration of Ubuntu currently, it effectively renders Ubuntu as not suitable for high-reliability workloads.

I have raised a defect in launchpad

https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/437473


The bug now says the problem should be fixed for the release.


So the test was useful, and since it was clearly different than the other results, it has borne value by investigating.

Michael's assertions within the article are still correct. If you want a default SQLite install to absolutely fly, run it under KVM under Ubuntu. But be aware of the risk to your data - I personally don't expect Michael to invest in the way that I did to understand and present data for each and every unusual result.

Regards,

Matthew

[mtippett]

The bug now says the problem should be fixed for the release.

Well, it is a lot more complex than that.

After initial disbelief, the KVM team eventually accepted that it was a current version, reproduced it themselves.

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/41353


The "closure" of the bug was really just a knee jerk reaction by Anthony because he didn't want to look deeper into the issue because he did not believe it was valid.

In the end, cooler heads prevailed, they accepted there was an issue, created and applied the patch

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/41592

I didn't bother going back to re-open and fight the launchpad bug there.

FWIW, the issue is that fdatasync wasn't actually doing anything on the some filesystems, so this now makes sure it does something. As always, if it looks free and too good to be true, it usually is.

Lots of other subtle lessons were learnt, but most of those are about human nature and communication.

Regards,

Matthew

[cowmix]

Well, it is a lot more complex than that.

After initial disbelief, the KVM team eventually accepted that it was a current version, reproduced it themselves.

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/41353


The "closure" of the bug was really just a knee jerk reaction by Anthony because he didn't want to look deeper into the issue because he did not believe it was valid.

In the end, cooler heads prevailed, they accepted there was an issue, created and applied the patch

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/41592

I didn't bother going back to re-open and fight the launchpad bug there.

FWIW, the issue is that fdatasync wasn't actually doing anything on the some filesystems, so this now makes sure it does something. As always, if it looks free and too good to be true, it usually is.

Lots of other subtle lessons were learnt, but most of those are about human nature and communication.

Regards,

Matthew

So will the fix make it into the final release of 9.10? I

[mtippett]

So will the fix make it into the final release of 9.10? I

Not sure, I would say it is probably too late (2 weeks before release is a bit late).

It's unclear if Dustin believed it was a concern or not.