Why does Phoronix use Ubuntu for Benchmarking ?

[MetalheadGautham]

I see every phoronix hardware review using Ubuntu. Often there are issues like the latest version of ubuntu NOT having the latest version of the linux kernel and phoronix using alpha or beta releases of future versions of ubuntu, which may give wrong results because of buggy software.

Why can't you guys use a Rolling Release distro or a distro with instant updates to the latest stable bleeding edge software like Gentoo or ArchLinux ?

[Zhick]

Because (A) the majority of linux-users use Ubuntu, so benchmarking on/with Ubuntu reflects the experience of most users and more importantly because (B) Michael is an Ubuntu-fanboy. :P

[RealNC]

And (C) Gentoo can be much slower than Ubuntu if you don't know how to set it up properly (Gentoo is, and tries to be only as smart as the person sitting in front of the keyboard.)

[fat_chris]

You would have to try pretty hard to make Gentoo slow, it's not easy to mess up an installation if you can follow instructions when needed.

[deanjo]

Because (A) the majority of linux-users use Ubuntu

Ubuntu may be the most popular but it's not what a majority of linux users use. A majority implies more then half of all linux users would be using ubuntu which is not the case at all.

Here is a interesting little break down of the various distro's popularity.

http://royal.pingdom.com/2008/08/21/linux-popularity-across-the-globe/

[MetalheadGautham]

and those who care for performance are often also those who use extreme distros like gentoo and arch.

[combuster]

Gentoo had the fastest boot times but as a matter of speed - every linux distro can be fast as the other if set propperly, some feature can be slower (packet management etc) then the other.

Phoronix perform their benchmark on Ubuntu, ok, I don't have anything against ubuntu, I've used it for two years, but when an article comes up stating that on 2.6.31 kernel with 2.8.0 intel drivers are slower and 100% unstable then the 2.6.30 and 2.7.0 then u got to ask yourself: is testing a feature on early alpha a smart move?

[energyman]

Phoronix should use a distri using KDE. Since the majority uses it.

Since kubuntu's kde is utterly broken, SLACKWARE or opensuse would be a good choice.

In fact, Slackware would be the best choice, because there are no patches that can screw up the results. Mean, lean, unpatched.

[germulvey]

I would welcome a review that was carried out on Ubuntu and another distro. Or at least one that uses say the rpm package system and also one of the more techie distros.
I have noticed that the results I see in rpm based distros for nvidia performance seems better. No idea why, but I definately get better video playback than on ubuntu in Mandriva and openSuSE.

[d2kx]

Phoronix should use a distri using KDE. Since the majority uses it.

Since kubuntu's kde is utterly broken, SLACKWARE or opensuse would be a good choice.

In fact, Slackware would be the best choice, because there are no patches that can screw up the results. Mean, lean, unpatched.

I have used Slackware some years ago, but like to try it out again. Is there anywhere a howto that helps with the first steps with Slackware? Can you set up slackware-current directly or do you have to use the release first? Do you still need to install slapt-get for good package management? Is there any special configuration needed like adding the user to specific groups like with Arch? Thank you:)

[germulvey]

This might help you out.

http://www.slackware.com/install/

:)

[germulvey]

This might be what your after!

http://www.slackware.com/install/

[energyman]

honestly: I don't know ;) I am using gentoo. Last time I installed slackware was 8.0 - and back then I didn't need howtos.... ir slap-get ;)

[val-gaav]

Since kubuntu's kde is utterly broken,

It is ? I'm not a great fan of ubuntu, but I am using kubuntu right now (ditched debian because I wanted kde 4.2 when it was out) and overall I am happy with it...

Well it would be nice though to see some desktop article here like some benchmarks tests of KDE4 around major distros and pointing out which is 'the best KDE4 distro out there' ... Most people consider ubuntu to be the best gnome distro, but as far as KDE goes opinions are very different...

[combuster]

That is why you should choose distribution where you can choose your DE, arch and gentoo are not limiting your choice, kde, gnome, xfce whatever you want - you can install.

[energyman]

well, lets see - reports of kde crashes in kubuntu - and seldomly somewhere else. KDE's localization is pretty much complete (and it is complete for languages like german or french), but not in kubuntu - because they 'destroy' the complete kde localization and replace it with the incomplete ubunutu localization.

[germulvey]

Linuxmint's KDE (gloria) was very impresive and one of the best kde implementations I have tried to date. Very usable and I know they fixed a lot of the Kubuntu issues along the way. Shame no 64 bit version.

[Apopas]

Well, I'm fan of gentoo and while I'musing for the last 5 years the same installation, it's always modern and fast, but such a distro is not the best choice for phoronix tests. While it could be faster than Ubuntu, OpenSUSE etc it needs work to install so is not comfortable when you have a piece of new hardware and want to setup, run and test an OS in a few minutes. So a binary based pre-configured distro is the best choice. So since we have to choose between one million dists the best option is the most popular one and that's Ubuntu.

[energyman]

but ubuntu does not use the most popular desktop and their kernel is heavily patched. Something closer to mainline would be better, because the results could pe transfered for easily.

[Apopas]

but ubuntu does not use the most popular desktop and their kernel is heavily patched. Something closer to mainline would be better, because the results could pe transfered for easily.

Well, nothing is perfect, Ubuntu uses one of the two most popular desktop environments but to use KDE or Gnome is not critical for benchmarking. Also, since Ubuntu has a lot of flavours like Kubuntu and Xubuntu you can be pretty sure that the results you see in phoronix.com will be the same between them.
Also, for the heavily patched kernel, well this is more common between the distros than the ice in the arctic and there is a reason for that. OpenSUSE does the same and Mandriva and Fedora etc... even gentoo's default kernel is patched. Also, don't forget that PTS compiles the benchmark apps by itself rather than using the distro's defaults.
The most important thing someone who runs benchmarks should keep in mind is the time and easy of install and setup of the OS he uses.

[energyman]

except that the DE has an influence. I tried it a few years ago. Everything based qt (kde, integrity) was a little bit faster than everything based gtk (gnome, xfce)

[Apopas]

except that the DE has an influence. I tried it a few years ago. Everything based qt (kde, integrity) was a little bit faster than everything based gtk (gnome, xfce)

But that's a personal opinion, you can't present results based on this, personally I've experienced quite the opposite, so I use Gnome in my desktop machine and XFCE for laptops :)

[energyman]

I don't have the numbers anymore, but:
glxgears was faster
ut2003 was faster
vegastrike was faster

with kde or integrity.

Not much, we are talking 2-3fps. But that in all situations.

[Apopas]

I don't have the numbers anymore, but:
glxgears was faster
ut2003 was faster
vegastrike was faster

with kde or integrity.

Not much, we are talking 2-3fps. But that in all situations.

Maybe something in your Gnome installation was using CPU. Pulseaudio for example. For me both KDE and Gnome are the sdame in speed but Gnome uses lower RAM.

[energyman]

back then pulseaudio didn't even exist. And I will never use a sounddaemon. I hate them so much.

about memory:
http://ktown.kde.org/~seli/memory/desktop_benchmark.html

I really don't think that gnome has changed for the better - because a complete rewrite would be necessary.

[Apopas]

back then pulseaudio didn't even exist. And I will never use a sounddaemon. I hate them so much.

about memory:
http://ktown.kde.org/~seli/memory/desktop_benchmark.html

I really don't think that gnome has changed for the better - because a complete rewrite would be necessary.

All these tests are very subjective. the plain desktop means nothing. For example was nautilus compiled with tracker or beagle support? Did gnome-panel used eds etc. Under my gentoo I can for example build a Gnome system which can use 40% lesser RAM or 80% more than the current one I use. The same for KDE. The matter is for the things I need my system to support, Gnome is a bit lighter. But as I said before that's just a personal experience and should stay like that when I have to present tests in the public. The DE should be relevant just when we benchmark desktop environments. When we run a general benchmark in Ubuntu and Mandriva, the matter is the performance with its default enviroments and not when we change Gnome to KDE in Ubuntu and KDE to Gnome in Mandriva.

[energyman]

look at the date. 2006. Was anybody using tracker or beagle back then? did pulseaudio even exist?

[Apopas]

look at the date. 2006. Was anybody using tracker or beagle back then? did pulseaudio even exist?
I mentioned some examples, I'm bored to check what options had Gnome and KDE back then. There are thousands of things in Linux that can make your desktop environments fly or crawl, from the compiler's flags to support of bluetooth and multimedia. In general you can make even XFCE to perform "heavier" than Vista (well, overexaggeration ofcourse, but you get the picture :) )
Come on, you're a gentoo user, you know that very well :)

[sabriah]

Phoronix probably uses Ubuntu for hardware benchmarking for a very good reason - they had to choose one. If they started to choose more, Phoronix might lose focus.

I am happy as I hope they will also stick with it. That way one can make comparisons over time. No, I have never run Ubuntu myself.

And, if they compare an alpha version with a stable? Well, as long as it is clearly stated in the article, I see no bad in that.

.

[frantaylor]

Many servers run at runlevel 3 so the desktop does not even matter. I want to see benchmarks of server performance: nfs, openldap, postgresql, samba.

I want benchmarks with tweaked configurations. Anyone who cares about performance is going to tweak, so benchmarks with default parameters are not very interesting.

I would like to see benchmarks for routing. I want to use el-cheapo hardware for routing and I want to know which is the best kernel to use. I want to see IPv4 performance versus IPv6 performance.

I want to see benchmarks for network cards. I want to know which cards will slurp up a saturated gigabit network without dropping packets.

I would like to see benchmarks for virtualization products. Centos 5 in VMware versus Centos 5 in qemu, for example.

I want to see benchmarks for supported products. I don't care about gentoo or opensolaris or opensuse because they do not have professional support. In my business the only two distributions that matter are RedHat and SuSE. This is not my choice but it is reality.

The desktop is more than fast enough for me already. I don't care if a text box draws 1.4% faster on one distribution than another.

[Apopas]

The desktop is more than fast enough for me already. I don't care if a text box draws 1.4% faster on one distribution than another.
But benchmarks have rarely practical use. Their existence is only to prove that my distro/OS is better than yours for no real reason :rolleyes:

[frantaylor]

But benchmarks have rarely practical use. Their existence is only to prove that my distro/OS is better than yours for no real reason :rolleyes:

Well that is just bogus because there are many issues. Some ethernet cards just suck and you cannot see for sure unless you benchmark. If you tweak on the block size for your disk partitions it will have impact, but you will not know unless you benchmark. Often there are multiple ways to write an algorithm and you cannot tell which is faster unless you benchmark. There is a bewildering array of optimizations for gcc and you cannot tell which will make your program run faster unless you benchmark.

[lordmozilla]

Centos 5 in VMware versus Centos 5 in qemu, for example.


Releasing Vmware benchmarks is illegal. sorry ;-)

Benchmarks on a rolling release such as Arch/gentoo would show more what unpatched linux does. The problem with ubuntu is the patching of patches which end up with for example Intel gfx issues.

But phoronix is doing a great job, and I'm sure using ubuntu gets a lot of readers in.

[Apopas]

Well that is just bogus because there are many issues. Some ethernet cards just suck and you cannot see for sure unless you benchmark. If you tweak on the block size for your disk partitions it will have impact, but you will not know unless you benchmark. Often there are multiple ways to write an algorithm and you cannot tell which is faster unless you benchmark. There is a bewildering array of optimizations for gcc and you cannot tell which will make your program run faster unless you benchmark.
If I didn't like benchmarks I wouldn't be here and their usefulness is obvious. I had in mind some desktop benchmarks like lame encoding which mainly differs by 1-2% between the dists when I made the post.

[frantaylor]

Releasing Vmware benchmarks is illegal. sorry ;-)

Benchmarks on a rolling release such as Arch/gentoo would show more what unpatched linux does. The problem with ubuntu is the patching of patches which end up with for example Intel gfx issues.

But phoronix is doing a great job, and I'm sure using ubuntu gets a lot of readers in.

Is it publishing the benchmark that is "illegal", or is the publishing of the RESULTS of the benchmark that is "a contractual violation"?

It would be nice to have a script that runs the benchmark for us and we can view the results for ourselves. I can't imagine that that would be "illegal", or even a "contractual violation"

I wonder if there is any way to procure a copy of VMware Player without agreeing to the EULA. In this case one could publish benchmarks, because one did not enter into any sort of contract.

[MetalheadGautham]

Releasing Vmware benchmarks is illegal. sorry ;-)

Releasing VMWare benchmarks is illegal ? Since when ? How ? Why ?

Benchmarks on a rolling release such as Arch/gentoo would show more what unpatched linux does. The problem with ubuntu is the patching of patches which end up with for example Intel gfx issues.

And what is wrong with showing what unpatched linux does ? And btw, Gentoo too can be easily patched.

[frantaylor]

Releasing VMWare benchmarks is illegal ? Since when ? How ? Why ?





It's not "illegal", it's a "contract violation" to publish benchmark results for many products. Read the EULA. Oracle is another example of this.

EMC would probably work their legal wiles and take down any web site that published VMware benchmark results. However, there is nothing wrong with publishing the benchmark program itself. The curious can run the benchmark themselves, and use the results for decision-making, as long as they don't publish the results.

The problem with benchmarks of big complex software is that the results are often dependent on hardware or software setup or whatever, and the results are not very useful unless you just happen to have exactly the same setup.

[deanjo]

Releasing VMWare benchmarks is illegal ? Since when ? How ? Why ?

It's been in the EULA for as long as I remember.

See here why:

http://blogs.vmware.com/virtualreality/2009/03/index.html

Look at the March 12th at the bottom of the page.

You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review.

[Kano]

Basically all VMs have got problems with timers. If you really want to benchmark then use a manual stop watch. Timers tend to run pretty much like the moon inside VMs ;)

[frantaylor]

Basically all VMs have got problems with timers. If you really want to benchmark then use a manual stop watch. Timers tend to run pretty much like the moon inside VMs ;)

There is nothing wrong with the timer on the bare metal machine in which the VMs are running. Processes inside of VMs can easily communicate with the host system, and vice versa.

The guests' real time clock in Vmware Workstation 6.5.2 is really quite accurate. It uses the timer in the host system. The clocks in my VMs stay accurate to within a second over days of use despite heavy loading of both the guest and host operating systems.

[BlackStar]

I wonder if there is any way to procure a copy of VMware Player without agreeing to the EULA. In this case one could publish benchmarks, because one did not enter into any sort of contract.

It's quite easy, really:*have your cat step on your keyboard while you are not looking.*Chances are, she will press the enter or spacebar keys before too long - done!

[deanjo]

It's quite easy, really:*have your cat step on your keyboard while you are not looking.*Chances are, she will press the enter or spacebar keys before too long - done!


Noooooo!!!! Cats download kiddie porn. http://www.fudzilla.com/content/view/14999/1/

[MetalheadGautham]

wait a sec, didn't the EULA say you could publish benchmarks IF you got permision from VMWare first ?

[deanjo]

wait a sec, didn't the EULA say you could publish benchmarks IF you got permision from VMWare first ?

Yes, and as long as the tests have been approved by them. That gives them the ability to say "We suck in this test so we won't approve this one".

[TwistedLincoln]

I wonder if there is any way to procure a copy of VMware Player without agreeing to the EULA. In this case one could publish benchmarks, because one did not enter into any sort of contract.

That wouldn't work. A copyright license (in this case the EULA) gives you the right to do things you don't already have the right to do. Without the license, you don't have the legal right to do anything with the software, including simply using it.

[BlackStar]

Yes, and as long as the tests have been approved by them. That gives them the ability to say "We suck in this test so we won't approve this one".

Which is a good reason to use one of the Free virtualization solutions. Sorry VMWare, but you won't get my business with those practices.

[deanjo]

Which is a good reason to use one of the Free virtualization solutions. Sorry VMWare, but you won't get my business with those practices.

Well while publishing benchmarks may be a big concern for you, simply being able to benchmark on my own, with my setups and equipment satisfies my needs as those benchmarks are going to be completely relevant to my needs. Keep in mind that even VBox is not fully featured unless the closed source version is used.

[energyman]

and the cs version, which you can also use for free, works very well. I don't see any reason to use vmware for my virtualization needs.

[deanjo]

and the cs version, which you can also use for free, works very well. I don't see any reason to use vmware for my virtualization needs.

Ya it's getting there. It still is missing some features that vmware does well and it's just been fairly recently that VBox started implementing some of the more advanced features such as multicore support.