Benchmarking virtualization solutions

[hungerfish]

Hi,

As you are probably most aware of, a multitude of virtualisation options are available these days.
Now, I don't own a huge server array onto which I want to deploy such software, I am but just a humble linux-desktop-user who wishes to have a few 'other' os'es around, but does not want to dual-boot.

I would really like to see a comprehensive overview of the different virtualisation solutions that exist for my use-case, and I think that PTS could really give some insight into how they compare performance wise.
Especially now that PTS is gaining Windows support, I think it would be great to compare different VS-solutions running on the same host, maybe with different guest-os'es.

I can imagine that such testing would require a lot of time to setup, but maybe I'm not alone in finding this sort of benchmarking extremely valuable?!

[birdie]

It would be useless and I will say why

1) People who run Linux almost never run Linux in a VM, so Phoronix Test Suit is not suitable here.

2) VM differ so much, you'd better choose 'em based on their features and price, rather than performance.

* VirtualBox is free but not very stable.
* VMWare WorkStation costs money but it's absolutely rock solid and allows Aero effects for Windows Vista/7.
* QEMU is not very usable for usual users.

3) There are billions of different workloads you may have in your VM, it's impossible to test even 1% of them.

4) There are tens of Operation Systems you may run.

[hungerfish]

1) People who run Linux almost never run Linux in a VM, so Phoronix Test Suit is not suitable here.

Yes, but PTS can run on many other OS'es not just linux, including even windows!

2) VM differ so much, you'd better choose 'em based on their features and price, rather than performance.

Well for me performance is the issue, not the price. And PTS measures 'real-world' performance, so I don't think that

3) There are billions of different workloads you may have in your VM, it's impossible to test even 1% of them.

is a true statement, because for my 'use-case'(desktop level VS) I'm interested in the same 'real-world' performance PTS measures on a 'native' os.

4) There are tens of Operation Systems you may run.

Yes, and quite a few of some are supported by PTS

So I don't think it would be useless, maybe not very accurate nor 'meaningful' nonetheless interesting though!

[birdie]

and

5) I'm quite sure you can test the performance on your own if you're interested in just a couple of applications.

6) If your applications are IO bound that no VM can be slower or faster than any other VM.

7) You still haven't told us what kind of workload you have

[hungerfish]

Hmm, I still think that this would make an interesting article for phoronix...but yes, you are of course right when you say

5) I'm quite sure you can test the performance on your own if you're interested in just a couple of applications.

except that I don't have the extra machine/drive-space to setup and test a multitude of options.

6) If your applications are IO bound that no VM can be slower or faster than any other VM.

Why so? What do you mean with IO bound?

7) You still haven't told us what kind of workload you have

Because it shouldn't matter, in my view it's the sole point of a benchmarking suite to determine 'performance across a standardized set of usage loads', which is what I'm interested in, NOT how well I can play solitare running windows 7 in a VM
Also, I can't really say, for me its just general 'desktop-related-stuff', for instance use a certain app for which no alternatives exist, or when I can't be bothered to hunt around for a 'linux-version'

[biriachan]

Hmm, I still think that this would make an interesting article for phoronix...but yes, you are of course right when you say

except that I don't have the extra machine/drive-space to setup and test a multitude of options.

Why so? What do you mean with IO bound?

Because it shouldn't matter, in my view it's the sole point of a benchmarking suite to determine 'performance across a standardized set of usage loads', which is what I'm interested in, NOT how well I can play solitare running windows 7 in a VM
Also, I can't really say, for me its just general 'desktop-related-stuff', for instance use a certain app for which no alternatives exist, or when I can't be bothered to hunt around for a 'linux-version'

I'm a computer technician and have been using virtualbox and ubuntu for years now (started with ubuntu 7.04).

I run at least 3 to 4 virtual machines at a time every day. Most of the machines are running Windows 2k3. Over the years I have developed a great love for virtualization. At some point I got sick of hacking around my Host OS and braking it. I needed the PC for work so braking it wasn't really an option, but then again, I love braking my OS!

So this is were VMs come into play. When I want start a new project such as Build an FTP server, want to test a different version of a distro, or for anything really, it goes into a virtual machine bottom line.


I don't think that it would be that hard to run some bench marks.

They just need to standardize on the Hardware of the Host and virtual hardware of the Guest.

Example:

HOST
Dual Core 2.5 GHz =<
4 GB RAM =<
72,000 SATAII Hard Drive
OS - Debian, ubuntu, Fedora, Suse (Pick one for testing)

Guest
Single Core 32 bit
1 GB RAM
10 GB Hard drive
OS - Debian, ubuntu, Fedora, Suse, (windows ??) (Pick a few for testing)

I think picking 1 HOST os, and 2 or 3 different Guest OS would make for a great start.

It would be nice to see performance between different Host File System and Guest interaction

Personally I think virtualization is great and I would like to see the day when the HOST os native starts all User space applications in their open private virtual environment.

[hungerfish]

Personally I think virtualization is great and I would like to see the day when the HOST os native starts all User space applications in their open private virtual environment.

I couldn't say it any better