Benchmarking ZFS On FreeBSD vs. EXT4 & Btrfs On Linux

[kraftman]

Now you are doing this again.

I remember when you compared a new Intel core duo 2.4GHz Linux vs an old 800MHz SPARC Solaris and you said it proves that Linux is faster. I rejected that benchmark, but you thought it was a fair and good benchmark. Even if Linux were indeed 30% faster than Solaris, then 800MHz Linux would not be faster than 2.4GHz Solaris. So you are unfair.

How is it different than yours comparisons? Got any proves?

Good for you. You know that I am not making things up, I just quote Linus Torvalds, Andrew Morton and other Linux kernel developers. Do you suggest they are trolls? I have always shown links to what I say. You on the other hand, have confessed you FUD sometimes.

You simply make things up and put them out of the contest.

Now you are doing this again. You are comparing the fastest x86 cpu ever made, to a several years old cpu. You are comparing Intel 32nm Sandybridge Westmere-EX 10-core vs AMD 45nm Opteron 6-core.

It turns out that the Intel Sandybridge Westmere-EX E7-4870 cpu, is roughly twice as fast as the AMD Opteron 8435 cpu.

Here we see that the Westmere-EX cpu is 50% faster than an AMD 12-core cpu. See picture
http://www.anandtech.com/print/4285

And a 6-core cpu should be half as fast as a 12-core cpu. This means that Intel E7 is 3x as fast as the AMD 6-core cpu.

How that's different from your comparisons? It didn't matter for you if slowlaris machine was using double amount of RAM compared to Linux one. I bet if slowlaris was running same machine as Linux it would still be slower. Sadly, you can't prove I'm mistaken here.

Is this fair? You are comparing a Intel cpu that is 3x as fast than an AMD Istanbul cpu. You are comparing Intel 40 cores of the x86 fastest cpu ever made, to 48 cores of the old AMD Opteron.

You showed machine with double amount of RAM, with probably faster DB and that machine was much more expensive, so there were much better efforts put to make it perform good. How is that fair comparing mentioned system to much less expensive one with much less memory and probably slower DB?

Only Linux fans thinks it is fair to compare a 3x faster server, to an old server. Great.

Only slowlaris fans thinks it's fair to compare different systems.

Also, we see that this Linux Westmere-EX server have 97% cpu utilization which is better than the 87% earlier Linux result. Earlier, the Linux server had 87% cpu utilization, which is bad. But this Westmere-EX server have 97% cpu utilization. Why is that? Answer: The reason the new Linux SAP have better cpu utilization is because Linux uses fewer cpus

Linux benchmarks:
4cpus - 97% cpu utilization. This Westmere-EX benchmark
6cpus - 87% cpu utilization. The earlier SAP benchmark where Linux used same cpus as Solaris (see below for link)

That's a damn shame for slowlaris! It shows how bloated it is. With utilization at 99% using 48cores it was performing much slower than Linux with utilization at 97% and 40 cores! You should now have a point where utilization was wasted on slowlaris - it's its bloat that wastes CPU power.

The more cpus Linux uses, the worse the cpu utilization gets. In other words, Linux scales bad on SMP servers. If Linux was using 8 cpus, I suspect cpu utilzation would drop below 80%. And if using 24 cpus, the cpu utilization would maybe drop below 50%. Linux scales bad on SMP servers. But on clusters, Linux scales very good.

Great, but where are the proves? What we saw was slowlaris wasting cores, because of its bloat.

For the record, Solaris had 99% cpu utilization on 6 cpus, because Solaris has run on 64 cpus and above, for decades.
http://download.sap.com/download.epd...11DE75E0922A14

Great, but Linux was running on even more cores.

On the other hand, let us compare hardware that is similar. Here we see that Linux uses AMD Opteron 8439 cpus 2.8 GHz
http://download.sap.com/download.epd...FCA652F4AD1B4C

which are faster than AMD Opteron 8435, 2.6GHz. And still Solaris is 23% faster.
http://download.sap.com/download.epd...11DE75E0922A14

The only way Linux can win, is if Linux is using hardware that is 3x as fast. Desperate, yes?

Not at all. As usual your showing menigless comparisons. This slowlaris machine uses twice amount of RAM more. It uses 48 cores, but Linux running on 40cores is faster. Amazing? Hell no, because Linux isn't bloated way the slowlaris is.

There are many benchmarks I know of, which shows 32 cpu Solaris being faster than 4 cpu Linux, but I never post them. Because that is not fair. Such benchmarks proves nothing. Why do you keep posting benchmarks which proves nothing? Do you think that a 3x faster server proves that Linux is faster?

You post benchmarks with different ammount of RAM. Why do you keep posting benchmarks which proves nothing? Do you think that a system with double amount of RAM proves that slowlaris is faster?

If Bonwick FUDs, then show me a big SMP Linux server. Show me links. Go ahead. I have asked this many times, but you have never showed me a big SMP server. Why? Because there are none! Linux does not scale on SMP servers. This is true. If this is false, then you can show me a big SMP server. I am still waiting, I have asked you this many times. Show me the link. Prove that Bonwick FUDs by showing a big SMP Linux server.

Stop lying. You got links many times. Remember SGI machines? Even if you're claiming those were blades connected with each other we have 4*256CPUs. 256CPU is a big SMP Linux server. It's funny your bagging for proves while you give me nothing.

http://www.phoronix.com/scan.php?pag...inux_games_bsd
New tests have revealed that the modern FreeBSD operating system (via PC-BSD 8.2) can actually outperform Linux when it comes to running OpenGL Linux game binaries.

This is hilarious. Even when running Linux software, FreeBSD is faster. Everybody is faster than Linux. Solaris is faster. FreeBSD is faster. I would not be surprised if even Windows was faster. Studies by Intel shows that Linux has dropped 10% performance. Linux is slowest in the league.

This just proves you're a troll. That test was simply messed up and meaningless. The same "prove" as yours. Bsd was running KDE with kwin that suspends compositions and Linux was running compiz that doesn't suspend compositions thus there's performance drop in games running compiz. You meant: everybody is faster than Linus, perhaps? When comes to Linux, slowlaris and bsd are slower. I bet windows is slower, too.

http://www.theregister.co.uk/2009/09/22/linus_torvalds_linux_bloated_huge/

"Citing an internal Intel study that tracked kernel releases, Bottomley said Linux performance had dropped about two per centage points at every release, for a cumulative drop of about 12 per cent over the last ten releases. "Is this a problem?" he asked.
"We're getting bloated and huge. Yes, it's a problem," said Torvalds."

Out of contest as usuall. If there's some drop it doesn't really matter while there are huge performance gains:

"A FFSB benchmark in a 48 core AMD box using a 24 SAS-disk hardware RAID array with 192 simultaneous ffsb threads speeds up by 300% (400% disabling journaling), while reducing CPU usage by a factor of 3-4"

I have no problems with Linux being faster on a single cpu in some benchmarks. It might be true, and I dont deny that. This guy runs a software on his desktop pc, probably it is a single cpu pc. I have no problems if BTRFS is faster than ZFS on a single disk.

I have no problem with slowlaris being faster on much more expensive hardware with double ammount of RAM and with more cores and faster database.

ZFS and Solaris is built for scale, for big SMP servers with 64 cpus and beyond. Linux is not.

When comes to slowlaris it's build for making money on not so smart customers. It mainly serves just for running Oracle DB. When comes to horizontal and vertical scaling there's Linux that matters.

When we compare as few as 16 SSD disks, then ZFS is faster than BTRFS because BTRFS is not built to scale, BTRFS is a desktop filesystem:
http://www.mail-archive.com/linux-bt.../msg05689.html

When "we" compare ZFS with unstable BTRFS then we find nothing amazing out there.

When we compare as few as 48 cores on SAP benchmarks, then Solaris is faster. If we go above 8 cpus, or even 16 cpus, then Solaris crushes easily because Linux is not built for big SMP servers.

Where? While there's 40 cores Linux machine kicking slowlaris ass in SAP. I'm asking, where's slowlaris faster? When we go above 8CPUs I can only imagine how CPU utilization is wasted on slowlaris (like above SAP benchmark showed).

As we have seen, Linux scales bad at 6 cpus in SAP benchmarks with a low 87% cpu utilization. Ext4 creator Ted Tso explains why. The reason is that Linux developers dont have access to big SMP servers, so Linux can not be improved on SMP servers:
thunk.org/tytso/blog/2010/11/01/i-have-the-money-shot-for-my-lca-presentation/

As we have seen, Linux scaled crap out of slowlaris and Linux didn't waste CPU power for bloat.

http://kernelnewbies.org/Linux_2_6_37

As you can see there's improvement on the SMP server. By Ted Tso btw.

There dont exist big SMP Linux servers. Only up to 8 cpus, which is bad, compared to 64 cpus and beyond.

Wrong as stated before. Get the facts.

And now you have to answer, Kraftman.

Yep and I won't stop.

[jalyst]

@kraftman & kebabbert

Pointless Nerd debate...

[kebabbert]

How is it different than yours comparisons? Got any proves?

Kraftmans comparisons:
-----------------------
You first compared a 2.4GHz Linux server to an old 800 MHz SPARC Solaris server. We see that Linux server is 3x faster here. 2.4GHz / 0.8Ghz = 3.

Now you compare the latest 10-core Intel Westmere-EX, the fastest x86 cpu in the world, to an old AMD Opteron with 6-cores. SAP benchmarks show that the Westmere-EX is 3x faster than the AMD Opteron.

Do you think these are fair comparisons? (As you said earlier: yes, these are fair).


Kebabberts comparisons:
-----------------------
I compare the SAP benchmarks from the same year. I do not compare the latest server to an several year old server.

Linux server has an advantage here, it has faster cpus, and faster memory sticks. Solaris is at disadvantage here, it uses slower hardware. And who has the highest cpu utilization? Who gets the highest benchmarks? You tell me.




Regarding if Bonwick FUDs or not about Linux scales bad on SMP servers, I asked you to post a link to a SMP server. You pointed to the SGI Altix server.

Here is another Linux server. It has 4.096 cores, up to 8.192 cores. It also runs a single Linux kernel image. Just as the SGI Altix server.
http://www.theregister.co.uk/2011/09..._amd_opterons/

The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit,"

A programmer writes:

I tried running a nicely parallel shared memory workload (75% efficiency on 24 cores in a 4 socket opteron box) on a 64 core ScaleMP box with 8 2-socket boards linked by infiniband. Result: horrible. It might look like a shared memory, but access to off-board bits has huge latency.

Stop lying. You got links many times. Remember SGI machines? Even if you're claiming those were blades connected with each other we have 4*256CPUs. 256CPU is a big SMP Linux server. It's funny your bagging for proves while you give me nothing.

The SGI Altix server that goes up to many thousands of cores:
http://www.sgi.com/products/servers/altix/uv/

Support for up to 16TB of global shared memory in a single system image enables Altix UV to remain highly efficient at scale for applications ranging from in-memory databases to a diverse set of data and compute-intensive HPC applications.

http://www.hpcprojects.com/products/...product_id=941

CESCA, the Catalonia Supercomputing Centre, has chosen the SGI Altix UV 1000 for its HPC system

Thus, you are wrong again. The SGI Altix server is HPC, just as I said all the time.

There are Linux SMP servers on the market, they typically have 4cpus, 6cpus or 8cpus. Then there are Linux HPC servers running a single Linux image, they typically have 4.096 cores or even more. There are not Linux servers in between. Either they have 4-8 cpus, or they have thousands of cores. Why are there no Linux servers with 16 cpus? Because such servers are SMP servers.

Now, please post links to a Linux SMP server. Not a 4 cpu Linux server. But one with 16 cpus, or 32 cpus. Let me tell you: there does not exist such SMP Linux servers on the market. For a reason: Linux can not handle 16 cpus on SMP machines. Nor can Linux handle 32 cpus.

So, once more I ask you to post links to a Linux SMP server which has 16 or 32 cpus. You will not find any such links. Bonwick was right all the time; Linux scales bad on SMP servers. Up to 6-8 cpus are the maximum. On HPC servers everybody knows that Linux scales well.

That's a damn shame for slowlaris! It shows how bloated it is. With utilization at 99% using 48cores it was performing much slower than Linux with utilization at 97% and 40 cores! You should now have a point where utilization was wasted on slowlaris - it's its bloat that wastes CPU power.

The Linux server is using 3x faster cpus. Of course Linux is faster. If they compared Windows on a 3x faster server, I promise you that Windows would be faster than Linux, too.

This just proves you're a troll.

Thanks. At least I do not FUD, as you do. You confessed you FUD. This proves you FUD a lot.

"A FFSB benchmark in a 48 core AMD box using a 24 SAS-disk hardware RAID array with 192 simultaneous ffsb threads speeds up by 300% (400% disabling journaling), while reducing CPU usage by a factor of 3-4"

What is your point? Linux does great speed ups, because it is so slow. What is your point? That Linux is slow?

I have no problem with slowlaris being faster on much more expensive hardware with double ammount of RAM and with more cores and faster database.

The benchmarks show that Solaris is faster.

When "we" compare ZFS with unstable BTRFS then we find nothing amazing out there.

This is funny. You reject this ZFS vs BTRFS benchmarks, because ZFS used many disks and therefore scaled better. Your objection? That BTRFS is unstable, that is why you reject BTRFS.

Now, do you remember when Phoronix benchmarked OpenSolaris vs Linux, and Linux won? But OpenSolaris was the alfa version of Solaris 11 (OpenSolaris was not even beta). Why do think it is fair to compare unstable OpenSolaris to Linux? Why do you think it is not fair to compare ZFS to unstable BTRFS?

When Linux wins, it is fair and good. When Solaris and ZFS wins, it is unfair. Why is this? One rule book for Linux, and another rule book for others?

Where? While there's 40 cores Linux machine kicking slowlaris ass in SAP. I'm asking, where's slowlaris faster? When we go above 8CPUs I can only imagine how CPU utilization is wasted on slowlaris (like above SAP benchmark showed).

Again, the Linux server used 3x faster CPUs.

As we have seen, Linux scaled crap out of slowlaris and Linux didn't waste CPU power for bloat.

I missed this. The lastest SAP benchmark used 4 cpus. That is not good scaling. Linux did not scale crap out of Solaris, by using 4 cpus. Let me tell you, if Linux used 2 cpus, then I suspect Linux would have good cpu utilization.

As you can see there's improvement on the SMP server. By Ted Tso btw.

Ted Tso said that Linux kernel devs did not have access to as much as 32 core servers earlier. Ted Tso says the opposite of what you say, he agrees with me.

Yep and I won't stop.

You wont stop FUDing?

[kebabbert]

@kraftman & kebabbert

Pointless Nerd debate...

I would not call this a debate. Kraftman has nothing reasonable to say. He compares 3x faster servers to old Solaris servers. He says the SGI Altix server is SMP server, but on the SGI website it says it is HPC. He also confessed he FUDs and calls me "Idiot, Troll" etc. His debate technique could be improved, dont you think?

[kraftman]

I would not call this a debate. Kraftman has nothing reasonable to say. He compares 3x faster servers to old Solaris servers. He says the SGI Altix server is SMP server, but on the SGI website it says it is HPC. He also confessed he FUDs and calls me "Idiot, Troll" etc. His debate technique could be improved, dont you think?

It's not a debate, because you don't understand obvious things. You're comparing system with double amount of RAM with faster DB to another, much less expensive one.

Altix UV 1000 - For maximum scalability, Altix UV 1000 ships as a fully integrated cabinet-level solution with up to 256 sockets (2560 cores, 4096 threads) and 16TB of shared memory in four racks. Altix UV 1000 delivers up to 24.6 teraflops of compute power in a single system image.

Do you claim SMP servers cannot be HPC?

[kebabbert]

HPC servers are simple to build. SMP servers are difficult to build. I am claiming HPC can not be SMP.

Ok, the discussion continues here:
http://phoronix.com/forums/showthrea...508#post238508

[kraftman]

Kraftmans comparisons:
-----------------------
You first compared a 2.4GHz Linux server to an old 800 MHz SPARC Solaris server. We see that Linux server is 3x faster here. 2.4GHz / 0.8Ghz = 3.

Now you compare the latest 10-core Intel Westmere-EX, the fastest x86 cpu in the world, to an old AMD Opteron with 6-cores. SAP benchmarks show that the Westmere-EX is 3x faster than the AMD Opteron.

Do you think these are fair comparisons? (As you said earlier: yes, these are fair).

When Kebbaberts compares different systems it's good, but when others do this it's bad?

Kebabberts comparisons:
-----------------------
I compare the SAP benchmarks from the same year. I do not compare the latest server to an several year old server.

Linux server has an advantage here, it has faster cpus, and faster memory sticks. Solaris is at disadvantage here, it uses slower hardware. And who has the highest cpu utilization? Who gets the highest benchmarks? You tell me.

You compare more powerful hardware to much less expensive one with less amount of RAM, slower memory sticks and slower database. Is this fair? Even slowlaris that wastes CPU cycles, because of its bloat can win benchmark when it's running on much better hardware. That CPU utilization and much lower score just means slowlaris is bloated, don't you think? Do you have something to backup your claims regarding Westmere-EX giving Linux huge advantage on 40core machine compared to 48core one? As a single CPU comparison it mayb be three times faster in some things, but do you have something to backup your claims that it will still have such advantage on 40core machine?

Regarding if Bonwick FUDs or not about Linux scales bad on SMP servers, I asked you to post a link to a SMP server. You pointed to the SGI Altix server.

Here is another Linux server. It has 4.096 cores, up to 8.192 cores. It also runs a single Linux kernel image. Just as the SGI Altix server.
http://www.theregister.co.uk/2011/09..._amd_opterons/

A programmer writes:




The SGI Altix server that goes up to many thousands of cores:
http://www.sgi.com/products/servers/altix/uv/


http://www.hpcprojects.com/products/...product_id=941

Thus, you are wrong again. The SGI Altix server is HPC, just as I said all the time.

Being HPC doesn't mean it's not SMP. Try again, because it's SMP system.

http://www.pcworld.com/businesscente...ll_street.html

"The trading shops saw that the lowest-latency solutions would only be possible with Linux," Lameter said. "The older Unixes couldn't move as fast as Linux did."

The wallstreet migration to Linux proves only Linux is fast enough and that's why they abandoned solaris. It's nothing amazing in this decision since Linux isn't bloated the way solaris is. It seems having 30% slower binaries is too much for critical workloads and that's why they replaced solaris. Damn shame.

There are Linux SMP servers on the market, they typically have 4cpus, 6cpus or 8cpus. Then there are Linux HPC servers running a single Linux image, they typically have 4.096 cores or even more. There are not Linux servers in between. Either they have 4-8 cpus, or they have thousands of cores. Why are there no Linux servers with 16 cpus? Because such servers are SMP servers.

Just lies. You saw 64CPUs Big Tux.

Now, please post links to a Linux SMP server. Not a 4 cpu Linux server. But one with 16 cpus, or 32 cpus. Let me tell you: there does not exist such SMP Linux servers on the market. For a reason: Linux can not handle 16 cpus on SMP machines. Nor can Linux handle 32 cpus.

I gave it you few times. Just google for Big Tux. It was 64CPUs (not cores) machine and afaik it has been upgraded lately.

So, once more I ask you to post links to a Linux SMP server which has 16 or 32 cpus. You will not find any such links. Bonwick was right all the time; Linux scales bad on SMP servers. Up to 6-8 cpus are the maximum. On HPC servers everybody knows that Linux scales well.

Kebbabert's lying as usual. Like I said, Big Tux.

The Linux server is using 3x faster cpus. Of course Linux is faster. If they compared Windows on a 3x faster server, I promise you that Windows would be faster than Linux, too.

The slowlaris server with double amount of RAM running n much more expensive hardware with faster database is simply faster. Even Windows would be faster on such hardware, I guess. However, I can't promise that.

Thanks. At least I do not FUD, as you do. You confessed you FUD. This proves you FUD a lot.

You FUD and lie like it was proven.

What is your point? Linux does great speed ups, because it is so slow. What is your point? That Linux is slow?

No, it's fast and becomes even faster. The point is it's being faster and faster in general.

The benchmarks show that Solaris is faster.

The benchmarks show Linux is faster.

This is funny. You reject this ZFS vs BTRFS benchmarks, because ZFS used many disks and therefore scaled better. Your objection? That BTRFS is unstable, that is why you reject BTRFS.

I don't reject this test, because ZFS used many disk, but because it was tested against unstable btrfs.

Now, do you remember when Phoronix benchmarked OpenSolaris vs Linux, and Linux won? But OpenSolaris was the alfa version of Solaris 11 (OpenSolaris was not even beta). Why do think it is fair to compare unstable OpenSolaris to Linux? Why do you think it is not fair to compare ZFS to unstable BTRFS?

In this case it should be fair, because solaris has very few devs compared to Linux, so it can't be so fast and stable thus it doesn't really matter if you benchmark unstable or stable slowlaris edition. Afaik Phoronix benchmarked "stable" (open)slowlaris editions, too.

When Linux wins, it is fair and good. When Solaris and ZFS wins, it is unfair. Why is this? One rule book for Linux, and another rule book for others?

That's your domain.

Again, the Linux server used 3x faster CPUs.

I don't have your point. It used less cores. Is it a fair comparison when slowlaris has hardware advantage? You were showing me slowlaris that used double amount of RAM, faster DB and much more expensive hardware. Is that fair?

I missed this. The lastest SAP benchmark used 4 cpus. That is not good scaling. Linux did not scale crap out of Solaris, by using 4 cpus. Let me tell you, if Linux used 2 cpus, then I suspect Linux would have good cpu utilization.

Linux doesn't waste CPU cycles for bloat thus it's faster even with less utilization.

Ted Tso said that Linux kernel devs did not have access to as much as 32 core servers earlier. Ted Tso says the opposite of what you say, he agrees with me.

Surprisingly there are kernel devs that has access to much bigger machines. In example SGI sent patches for Linux and there are other devs that has access to 48core systems and more. Ted didn't agree with you. Maybe he just didn't have access to big servers once, but he improved performance later. That's a fact. IBM, Fujitsu, HP, Oracle have access to big servers and they're working actively on the Linux kernel.

You wont stop FUDing?

I won't stop replying to your FUD. Will I get a price, too? Kebb, what do you think? Is there anyone so generous as sun was? ;>

[kraftman]

HPC servers are simple to build. SMP servers are difficult to build. I am claiming HPC can not be SMP.

Ok, the discussion continues here:
http://phoronix.com/forums/showthrea...508#post238508

What does HPCwire means saying
SGI Altix SMP System?

[kebabbert]

I will answer in this thread
http://phoronix.com/forums/showthrea...508#post238508

[jalyst]

Blah, blah, blah, blah...
Ok, the discussion continues here:
http://phoronix.com/forums/showthrea...508#post238508

Thank God, take the e-Peen flashing elsewhere folks.
No one gives a shit how big your e-Peens are.