Fedora, Debian, FreeBSD, OpenBSD, OpenSolaris Benchmarks

[F.Ultra]

I do not agree on this. I suspect you have not studied complexity theory for parallell computers. There are many problems that are P-complete.

Of course there are many, I just said that it did not apply to most, and with most I meant the kind of problems that affect most types of software in the IT industry; webbservers, databases, stock exchange trading systems, games etc.

I think it is wrong to extrapolate from 4 cpus up to hundreds of CPUs

Yes it is, and that is why nobody here has done that.

To me, the price is not important.

And we are not talking about you, we are talking about HP, their PR department and why they chose to enter with the setup they did. What IBM does or does not do is highly irrelevant since we are talking about HP.

For the benchmark, I suspect it is designed to not be DataBase bound. That would be a bad benchmark. Most probably the speed of the RAM is important, not the DB.

I agree with you, but that does not mean that the DB cannot affect the outcome of the benchmark. Even if the DB is only used once per 100k transactions in the benchmark it could still impact the numbers in the way that we see.

Because what we see is that the CPU utilization is not 99%, and this can be because of a number of reasons:

1. There is not enough incoming requests to max out the server. It could be that the number of SAP clients used is too small, that they have congested the network etc. This would fit nice with the fact that the HP is using CPUs that is 200MHz stronger.

2. The CPU is waiting for I/O, either there is writes/reads to disk or the DB is holding the CPU back.

3. HP is displaying the userspace CPU utilization and is not including the kernel CPU utilization.

4. SAP is deadlocking semaphores on Linux but not on Solaris for whatever reason.

We simply do not know why, unless we perform the benchmark ourselves and take some significant time to debug it we cannot say why HO got the result they got.

All I can say is that there are very few things involved with scaling to CPUs within the kernel that can lower the CPU utilization, in 99% of cases where you see low CPU utilization it is either due to being I/O bound or not having enough requests to process.

[kebabbert]

Yes it is, and that is why nobody here has done that.

Maybe I misunderstood you, but you claimed that you had no scalability problems on Linux up to 16 cores (4 cpus) for your special parallell workload and therefore you doubt Linux has scalability problems with many more cpus? I thought this was a kind of extrapolating?

Never mind. We both agree that Solaris is better for SAP for this very setup (and we saw that Solaris easily handled more clients than linux). We dont agree why we see this higher CPU utilization on Solaris, but you dont rule out that Linux can have scalability problems on SAP:

4. SAP is deadlocking semaphores on Linux but not on Solaris for whatever reason.

I agree we dont know why, maybe the SAP user had a head ache and turned on wrong parameters, or whatever. My interpretation is simply that Solaris lives up to it's old reputation of scaling well, and Linux lives up to it's old reputation of scaling not so well. Occam's razor. But that is MY interpretation, and your interpretation is different.

I agree that to be able to draw a full conclusion, we need to know more details and debug the SAP system and see exactly why Linux has problems with cpu utilization on almost identical hardware.

Maybe you should write to Phoronix and question all Linux vs OpenSolaris vs FreeBSD benchmarks? They should stop, because you want Phoronix to debug and explain exactly why one distro is slower? But then, if Phoronix found out the problem, they could just correct the source code so the system runs faster. But this takes good kernel programming skills to do - so it is maybe unfair to request this. But it is not unfair to request this, when Linux lags behind, it seems. No one objects when Linux is faster ("ah, such a good and fair benchmark". But when Linux is slower on, e.g., SAP benhcmarks, there are people questioning the SAP benchmark ("no, this can not be right, something is wrong with the benchmark"). I think this is not really fair, but hey, since when has Linux supporters been fair? They compare 800 MHz SPARC to 2.4GHz Intel Core Duo and thinks that is fair. If I question that, they insist it is fair. Only one year later, they MAY admit it was not fair. But maybe not.

Linux wins a benchmark - no problem. Everything is fine.

Linus lags behind - demand a full investigation with full debug information and dont accept this benchmark. "Something must be wrong, I refuse to accept this stupid stupid stupid SAP benchmark."

I still think Linux supporters are not fair, but that is the way life is: not fair. Nothing to do, just accept it.

Let us agree that we disagree and conclude this debate.

[F.Ultra]

I gave the numbers from the SAP benchmark a closer look.

HP is performing 20.46% worse than Sun. While HP has a CPU with 7.69% higher clockrate compared with Sun the utilization is 12.12% lower, if we combine these two numbers we get 19.81% total lower CPU utilization for HP compared with Sun, which matches the lower performance.

Also HP is using 19.76% less SAP users than Sun.

Thus HP is having less CPU utilization due to it performing less number of transactions with a faster CPU so that is why the CPU utilization is lower.

In essence: HP is feeding their system with roughly 20% less traffic and is getting roughly 20% less performance, this is close to 1:1.

The question is why HP decided to feed with 20% less users than Sun, it could very well be that Linux on DL785 and SAP scales worse than Sun so that they would get even less performance with 10000 users or it could be any other reason. We simply do not know, we have to ask the HP team why the chose 8022 users instead of 10000.

This all depends upon how SAP is programmed. Let's say that they use malloc() in the hot code path inside a critical section, here GNU libc is unfortunately not fully optimized which is why we Linuxprogrammers who create software that has to scale avoids doing just that. Solaris probably does not have this particular issue since it uses another libc (the BSDs for example does not have this particular problem). If this is the case it is no wonder that SAP works better on Solaris than on Linux. Calling malloc() in the hot code path is a deadly sin anyways

[F.Ultra]

Maybe I misunderstood you, but you claimed that you had no scalability problems on Linux up to 16 cores (4 cpus) for your special parallell workload and therefore you doubt Linux has scalability problems with many more cpus? I thought this was a kind of extrapolating?

No, all that I said was that I had no trouble to create software that scaled to 16 cores, I did never say anything about Linux with more cpus. My only point was that creating scalable software on Linux or Solaris might be completely different tasks.

but you dont rule out that Linux can have scalability problems on SAP

No I don't, we clearly see lower performance from HP, this cannot be denied. It would be interesting however to see the Solaris benchmark retested on the HP hardware with the MaxDB, and also the reverse. Not to prove a point, more that it would be interesting to see what would happen.

Maybe you should write to Phoronix and question all Linux vs OpenSolaris vs FreeBSD benchmarks? They should stop, because you want Phoronix to debug and explain exactly why one distro is slower?

Please don't be this silly. It was you who posted SAP benchmarks as proof that Linux does not scale well, I have never posted any benchmark result anywhere claiming that Solaris or any other OS is bad, so please do not go down that road, it's just silly.

There is zealots from every camp doing just that, but these people are idiots and we do not have to drag ourselves down to their level?

[F.Ultra]

No, all that I said was that I had no trouble to create software that scaled to 16 cores,

A little explanation perhaps could be in order. When first reading the SAP benchmarks I didn't notice that they used 8 6-core CPUs so I erroneously saw 16 cores used and not 48, which is why I thought that my experience was relevant.

[kraftman]

You claim that I FUD. But everything I claim, is backed up by relevant links and certified benchmarks and white papers in an academic way - then it is not FUD. On the other hand, YOU, have not provided any relevant backup for your claims. And you claim that I lie, but do not/can not point out my lies.

I won't even bother replying to your other bull. You're still lying and FUDing, because I pointed you were lying and FUDing and you're saying now you backed up everything you said by "relevant links and certified benchmarks...", but you didn't backup things I pointed and which are lies and FUD. Every time you say "But everything I claim, is backed up by relevant links" is another lie. I consider you didn't provided any relevant backup for your claims and I was repeating this since some time.

[kraftman]

I agree that to be able to draw a full conclusion, we need to know more details and debug the SAP system and see exactly why Linux has problems with cpu utilization on almost identical hardware.

It's not almost identical hardware and software is different too.

No one objects when Linux is faster ("ah, such a good and fair benchmark".

Another lie.

But when Linux is slower on, e.g., SAP benhcmarks, there are people questioning the SAP benchmark.

A sane person can't say Linux scales worse basing on SAP papers you posted and this is one of the reasons you're an idiot to me.

("no, this can not be right, something is wrong with the benchmark")

I see there's really something wrong with you.

I think this is not really fair, but hey, since when has Linux supporters been fair? They compare 800 MHz SPARC to 2.4GHz Intel Core Duo and thinks that is fair. If I question that, they insist it is fair. Only one year later, they MAY admit it was not fair. But maybe not.

This is exactly what you're doing here. You're showing two different papers with two different servers and you say Solaris scales better and you don't want to admit it's not fair. You're also accusing others exactly what you're doing.

Linus lags behind - demand a full investigation with full debug information and dont accept this benchmark. "Something must be wrong, I refuse to accept this stupid stupid stupid SAP benchmark."

This "stupid stupid stupid SAP benchmark" didn't benchmark Linux and Solaris, but two different servers. Phoronix benchmarks systems using their stock configurations (usually) and this is quite fair. Of course you can install newer packages, tweak some things etc. and results will be different. You will accept defaults are tested or you can cry like a baby.

I still think Linux supporters are not fair, but that is the way life is: not fair. Nothing to do, just accept it.

It's you who's not fair. You gave Bonwick (a SUN developer) PR talk to backup your claims, two different SAP papers which you treat like an OS benchmark etc.

You said in one of your comments something like: Linux is known it doesn't scale well. I can say Solaris is known of being slow thus the name Slowlaris.

[indezign]

Well after a long period of time, we now have FreeBSD 8.1 RC2 with high-performance ZFS. I wonder how it competes with Fedora 13 using EXT4?