File System Benchmarks in Linux

[petabyte]

As Michael requested: a suggestion for Linux benchmarking:

I would really like to see some file system comparisons next time you
test a (stable) Linux kernel. Benchmarking Linux kernels is fine, but aside from the new features, significant performance gains are unlikely (especially when testing with games). It seems we know much less about the performance of the array of Linux file systems we have. The last good file system benchmarks I saw were fefe's: http://bulk.fefe.de/scalability/, but it seems he's ceased his testing.

I know it is difficult to control file system benchmarks in a fair way, but the guidance within fefe's extensive tests should surely get you started. I'd like to see reiserfs3, reiserfs4, ext3, ext4, xfs, jfs, and zfs in a line-up. I have lots of ideas so please reply with any questions.

[Jade]

As Michael requested: a suggestion for Linux benchmarking:

I would really like to see some file system comparisons next time you
test a (stable) Linux kernel.....

I'd like to see reiserfs3, reiserfs4, ext3, ext4, xfs, jfs, and zfs in a line-up. I have lots of ideas so please reply with any questions.
Funny, you should mention filesystem tests:

Smaller is better:


-------------------------------------------------------------------------------------------------
| Filesystem | Copy Sources | Disk Usage | Copy Sources | Tar & Gzip | Untar & Unzip | Delete All |
| | Across | Usage | Within | Sources | Sources | (seconds) |
| | Partitions | (MB) | Partition | (seconds) | (seconds) | |
| | (seconds) | | (seconds) | | | |
-------------------------------------------------------------------------------------------------
| REISER4 | 148 | 692 | 55 | 68 | 27 | 52 |
| NTFS-3g | 1328 | 772 | 1348 | 600 | 776 | 73+ |
| NTFS | 781 | 779 | 173 | X | X | X |
| REISER3 | 184 | 793 | 100 | 87 | 61 | 11 |
| XFS | 221 | 799 | 173 | 120 | 91 | 57 |
| JFS | 290 | 806 | 547 | 87 | 1248 | 89 |
| EXT2 | 204 | 816 | 81 | 73 | 40 | 24 |
| EXT3 | 181 | 816 | 77 | 77 | 46 | 26 |
| FAT32 | 252 | 988 | 162 | 126 | 88 | 19 |
-------------------------------------------------------------------------------------------------


REISER4 wins (by far) in ALL the important categories.

It is not clear why NTFS preformed so poorly.

Everyone knew FAT32 was a dog.

The reiser4 filesystem clearly outperforms all the other filesystems, in almost all the measured categories. Using reiser4, rather than ext3, saves you a massive 816 - 692 = 124 MB of disk space (a 15% saving, mainly, from eliminating block alignment wastage). Not only do you save 124 MB, but your copy is finished significantly quicker.

As you can see, REISER4 is a truly remarkable filesystem.

This is the real reason that REISER4 has not been included in the Linux kernel.
This is the real reason that Hans Reiser languishes in an Oakland prison cell at this time.

From http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,

http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

ALSO SOME BONNIE++ RESULTS


---------------------------------------------------------------------------------
Version 1.93c Sequential Create Random Create
SuSE 10.0 Create Read Delete Create Read Delete
files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
---------------------------------------------------------------------------------
REISER4 128:131072:0 779 21 893 19 1554 9 126 4 65 1 781 4
REISER3 128:131072:0 288 7 72 1 382 2 283 7 64 1 138 1
notail 128:131072:0 279 7 71 1 343 2 293 7 64 1 140 1
XFS 128:131072:0 222 3 525 8 914 5 280 4 71 1 111 1
EXT2 128:131072:0 263 53 764 13 766 2 265 53 83 8 133 11
EXT3 128:131072:0 224 5 87 1 172 1 224 6 86 1 167 1
FAT32 128:131072:0/5 49 93 94 82 294 97 72 89 34 30 94 95
NTFS-3g 128:131072:0 70 0 77 0 4744 3 59 0 38 0 435 0
JFS 128:131072:0 57 0 421 6 37 0 19 0 70 1 22 0
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
Version 1.93c Sequential Output Sequential Input Random
Concurrency 1 Per Chr Block Rewrite Per Chr Block Seeks
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
---------------------------------------------------------------------------------
REISER4 1G 247 97 57710 12 28317 10 405 99 63709 10 210.3 6
REISER3 1G 541 98 52162 14 27133 6 1566 91 63820 7 218.0 3
notail 1G 518 99 53709 14 25820 5 1565 83 58242 6 219.0 3
XFS 1G 753 99 58613 11 27075 5 1242 95 63546 5 197.7 2
EXT2 1G 1103 98 54209 8 27425 5 1475 90 62867 5 215.3 2
EXT3 1G 399 99 48357 16 27124 5 1459 92 63209 6 213.9 3
FAT32 1G 966 98 50540 20 24478 10 1553 93 63883 15 195.1 5
NTFS-3g 1G 26 6 49060 7 20383 3 1629 93 57937 4 169.5 0
JFS 1G 1141 98 56314 9 27866 4 1582 92 63839 5 220.2 2
---------------------------------------------------------------------------------

[petabyte]

Thats excluding ext4 and zfs. If the phoronix people get around to testing newer versions of these with a newer kernel...That would be cool.

[1c3d0g]

Jade: you also forgot to mention that Reiser4 is unstable as hell. Sure, it performs nicely, no doubt about that. But would you really trust your data on a fs that can f*ck itself over any second? Not me, thanks. :)

[Rob Williams]

Jade: you also forgot to mention that Reiser4 is unstable as hell. Sure, it performs nicely, no doubt about that. But would you really trust your data on a fs that can f*ck itself over any second? Not me, thanks. :)

I was thinking the same thing. It's fast, maybe because it doesn't handle thing securely. I'd be interested in seeing ext4 performance as well.

[StringCheesian]

Reiser4's performance is awesome. I'd use it if its latency was better and it was stable.

This is the real reason that REISER4 has not been included in the Linux kernel.
No. Reiser4 has it's own built in replacement for kernel code that the other linux filesystems use. That replacement would be redundant - the kernel devs would rather reiser4 and the other file systems share as much code as possible. They also said it was too unstable. That's why.

This is the real reason that Hans Reiser languishes in an Oakland prison cell at this time.
Name one person/entity with the motive to do what you're implying. Let me guess - it's gotta be either Elvis or Big Foot, right? No wait, it's probably that massive government conspiracy with the aliens. Reiser4 protects you from being kidnapped and probed, and they can't have that can they?

[Jade]

1c3d0g said: "you also forgot to mention that Reiser4 is unstable as hell."

May I politely suggest that you are mistaken.

If not, please produce the studies that show the purported Reiser4 instability.

I have seen ZERO evidence that Reiser4 is unstable.

I have noted that Reiser4 has been deliberately sabotaged in all of Andrew Morton's recent Linux kernels (linux-2.6.20-rc1-mm1 thru rc4), and even though many complaints can be found on the internet, no one has bothered to respond to, or fix, the problem(s).

StringCheesian said: "Name one person/entity with the motive to do what you're implying."

Hans Reiser is a Jew who is being slandered and set up by Jews.

His problem is that his product Reiser4, is much better than NTFS.

This is a problem for Microsoft and Microsoft owners.

The visible face of these people is "the Jews" (the secret society, not the religion, or having a Jewish mother (although, of course, the last are closely related to the first)).

The GPL prevents the usual tactic of buying him out (and turning Reiser4 into NTFS2).

Without the GPL, Reiser might be a rich man, and Microsoft might be extolling the virtues of it's new filesystem.

[Michael]

Jade,

I have already warmed you about your anti-semitic remarks on the Phoronix Forums. These comments will not be tolerated and any further rants will result in your banning on the forums. I ask that you kindly refrain from making these comments, thank you.

[joshuapurcell]

This is unbelievable. Did anyone ever see that Mel Gibson movie where he was worried everyone was watching him, and he had his house locked down and had all the conspiracy theories? Can't remember the name of the movie (probably Conspiracy something or other), but Jade missed his calling.

Stop posting this crap here... find some foil collector's community to work out the details of your as-yet-to-be discovered plot for Jewish world domination.

[siti]

This is unbelievable. Did anyone ever see that Mel Gibson movie where he was worried everyone was watching him, and he had his house locked down and had all the conspiracy theories? Can't remember the name of the movie (probably Conspiracy something or other), but Jade missed his calling.
It's Conspiracy Theory.

[petabyte]

Amazing, I never would have imagined that a file system thread could mutate into a religious discussion. Yes, the movie you are looking for is "Conspiracy Theory", and yes, there is much Zionism in America. There is, however, no evidence to show that Reiser's jailing is a result of Microsoft's (and the Jews') actions. Reiser's jailing is much more likely the result of our flawed judicial system (or, unfortunately, but less likely, a very large mistake on his part).

Furthermore, if the exclusion of reiserfs4 from the mainline kernel is an attempt to keep us from a revolutionary file system, then why aren't ext4 and zfs being censored, and their creators being thrown in jail. Therefore, it is much more likely that the reiserfs4 file system was excluded due to laziness, technicalities, or misunderstandings.

I agree that ext4 is no more ready for the kernel that reiserfs4. The solution to quicker maturity, of course, is to include both in the kernel to allow easier testing. I hope reiserfs4 is soon accepted by the kernel developers for this reason. Until then we will have to do the work of patching the new kernels ourselves in order to test the latest and greatest file systems against those which we are accustom to.

[Fragadelic]

You guys better be careful, I hear the Men In Black coming - lmao.

[Jade]

petabyte said:"if the exclusion of reiserfs4 from the mainline kernel is an attempt to keep us from a revolutionary file system, then why aren't ext4 and zfs being censored"

Gee,.. you ask easy questions:

1) ext4 is not even close to being as good, or as stable, as Reiser4 (I ran some tests, ext4 sux).
2) zfs isn't part of Linux (maybe you could help and port it from Solaris).

To make things explicit: Why censor things that people will not (want to) use anyway?

petabyte said:"I agree that ext4 is no more ready for the kernel that reiserfs4."

WRONG. REISER4 is easily ready for inclusion. EXT4 is NOT.

Here are the results for EXT4, with 2.6.20 kernel (drivers), and SuSE 10.0.


-------------------------------------------------------------------------------------------------
| Filesystem | Copy Sources | Disk Usage | Copy Sources | Tar & Gzip | Untar & Unzip | Delete All | kernel
| | Across | (MB) | Within | Sources | Sources | (seconds) |
| | Partitions | | Partition | (seconds) | (seconds) | |
| | (seconds) | | (seconds) | | | |
-------------------------------------------------------------------------------------------------
| REISER4 | 148 | 692 | 55 | 68 | 27 | 52 | 2.6.13-15-default
| tails | 83 | 673 | 91 | 91 | 38 | 82 | 2.6.20-rc6-mm3
| EXT2 | 204 | 816 | 81 | 73 | 40 | 24 | 2.6.13-15-default
| EXT3 | 181 | 816 | 77 | 77 | 46 | 26 | 2.6.13-15-default
| EXT4 | 251 | 806 | 125 | 109 | 93 | 47 | 2.6.20
| extents | 223 | 793 | 196 | 178 | 86 | 118 | 2.6.20
-------------------------------------------------------------------------------------------------


The comparison is not too accurate as different kernels were used.

Strangely, my customized 2.6.20 kernels ran consistently slower (about 7%) than the 2.6.13-15-default SuSE 10.0 kernel (even though 2.6.13-15 was compiled only with generic x86-64 support and the 2.6.20 had AMD Athlon64 support, etc, etc).

There seems to be a fair amount of randomness in the figures.

When I get time I might run the script ten, or so, times and average out the results.

[Jade]

So, what is the accepted way of stopping other processes running (and effecting the results) while doing these filesystem tests?

[Rob Williams]

So, what is the accepted way of stopping other processes running (and effecting the results) while doing these filesystem tests?

I guess not using a DE for one thing would be the first step, and them making sure you don't have a bunch of system services at boot time. Shouldn't be that difficult, really.

[petabyte]

petabyte said:"if the exclusion of reiserfs4 from the mainline kernel is an attempt to keep us from a revolutionary file system, then why aren't ext4 and zfs being censored"

Gee,.. you ask easy questions:

1) ext4 is not even close to being as good, or as stable, as Reiser4 (I ran some tests, ext4 sux).
Hearsay.

2) zfs isn't part of Linux (maybe you could help and port it from Solaris).
I didn't suggest it is part of linux. I was only comparing potentially great file systems.

To make things explicit: Why censor things that people will not (want to) use anyway?

More hearsay.

petabyte said:"I agree that ext4 is no more ready for the kernel that reiserfs4."

WRONG. REISER4 is easily ready for inclusion. EXT4 is NOT.

Please do not act as if your corrections are the absolute fact; colorful font and the information the colorful font conveys are two different things entirely. You've presented some questionable performance tests, and thats it. This is unrelated to my claims. Maturity and stability are much more valid when analyzing what should and shouldn't be included in the kernel. I'm a big reiserfs fan too, but please don't perpetuate trolliness.

[Jade]

http://linuxhelp.150m.com/resources/fs-benchmarks.htm


.----------------------------------------------------------------------------------------------------.
| Filesystem | Copy Sources | Disk Usage | Copy Sources | Tar & Gzip | Unzip & Untar | Delete All |
| Kernel | Across | (MB) | Within | Sources | Sources | (seconds) |
| | Partitions | | Partition | (seconds) | (seconds) | |
| | (seconds) | | (seconds) | | | |
.----------------------------------------------------------------------------------------------------.
|REISER4 2.6.13 | 148| 692| 55| 67| 25| 56|
|REISER4 2.6.20 | 151| 692| 85| 81| 32| 65|
| tails 2.6.13 | 148| 673| 63| 78| 33| 65|
| tails 2.6.20 | 155| 673| 99| 81| 41| 79|
|NTFS-3g 2.6.13 | 1333| 772| 1426| 585| 767| 194+|
|NTFS-3g 2.6.20 | 1153| 772| 1488| 597| 844| 195+|
|NTFS WIN XP | 781| 779| 173| X| X| X|
|REISER3 2.6.13 | 184| 793| 98| 85| 63| 22|
|REISER3 2.6.20 | 182| 793| 103| 81| 65| 23|
|XFS 2.6.13 | 220| 799| 173| 119| 90| 106|
|XFS 2.6.20 | 210| 799| 159| 109| 89| 88|
|JFS 2.6.13 | 228| 806| 202| 95| 97| 127|
|JFS 2.6.20 | 226| 806| 216| 95| 100| 133|
|EXT4 2.6.20 | 174| 816| 70| 74| 42| 50|
| extent 2.6.20 | 162| 806| 55| 69| 36| 32|
|EXT3 2.6.13 | 182| 816| 74| 73| 43| 51|
|EXT3 2.6.20 | 177| 816| 62| 76| 41| 47|
|EXT2 2.6.13 | 201| 816| 82| 73| 39| 67|
|EXT2 2.6.20 | 172| 816| 72| 72| 37| 52|
|FAT32 2.6.13 | 253| 988| 158| 118| 81| 95|
|FAT32 2.6.20 | 207| 988| 105| 97| 88| 57|
.----------------------------------------------------------------------------------------------------.


http://linuxhelp.150m.com/resources/fs-benchmarks.htm

[1c3d0g]

Desktop: ext4.
Server: XFS, and perhaps one day when it's ported, ZFS.

Don't waste your time or energy with other useless FS unless you run some exotic setup like clusters etc.

[Jade]

Don't waste your time or energy with anything other than REISER4 (with gzip or lzo compression).

SMASHES the competition.

[joshuapurcell]

Don't waste your time or energy with anything other than REISER4 (with gzip or lzo compression).

SMASHES the competition.

I'm all for using the best filesystem if it is available, but this type of black and white (or should we say red) argument you are attempting to make (rather poorly) is not the way to sway people over to your way of thinking. Even if you've done tests that show reiserFS is the be-all-end-all of filesystems, other people have assuredly done more involved tests that show ext does better than reiser under certain circumstances.

Two things:
1) there is no greatest filesystem that is better than all others
2) lose the red font color

[Kano]

Funny test, but NTFS-3g should be restested with 1.913, released on September 13, 2007. Speed was increased dramatically - especially on fragmented partitions.

[thoemy]

What about the cpu usage? The last thing I remember is, that reiserfs was the overall fastest but also the most CPU consuming file system.

[Kano]

reiser4 has too many problems in long term use. Also when you install grub into that partition you can get into real trouble...

[cjcox]

http://www.ntlug.org/Presentations/FilesystemBenchmarks2

[Pseus]

Some of you might be interested in this short article I found a while ago while peeping through btrfs' mailing lists:

http://www.csamuel.org/articles/emerging-filesystems-200709/

It's a fairly recent comparison of most emerging (and old too) Linux filesystems. Bear in mind that some of them have made a lot of progress since then, particularly btrfs, at least according to the mailing lists.

[Lykos]

I'm a little bit confused...
I had always thougt, reiser4 was unstable.
And I really need a stable FS, a stable Hardware, I have enough trouble with unstable Software, I want to concentrate on other Tings.
But of course I would like to see the reiser4 FS in the Linux kernel. But not if I'd have to loose stability, performance in other parts of the OS and much time.

[Pseus]

I'm a little bit confused...
I had always thougt, reiser4 was unstable.
And I really need a stable FS, a stable Hardware, I have enough trouble with unstable Software, I want to concentrate on other Tings.
But of course I would like to see the reiser4 FS in the Linux kernel. But not if I'd have to loose stability, performance in other parts of the OS and much time.

IMHO, reiser4 suffers from the lack of a group of committed developers. I know at least one guy from Namesys kept working on it, but progress has been slow. I think it will get merged to mainline someday, just don't expect that to happen soon.

If you want stability, pick any of the other filesystems according to your needs. There's no be-all and end-all solution when talking about filesystems. At least not now.

[Jade]

HANS REISER AND HIS FILESYSTEMS.

The HANS REISER Murder Trial. Timeline and Analysis.

http://linuxhelp.150m.com/politics/ReiserTrialSummaryAnalysis.htm
http://linux.50webs.org/politics/ReiserTrialSummaryAnalysis.htm

DISCUSS IT HERE:

http://www.phoronix.com/forums/showthread.php?t=7544

REISER4 HOWTOS.

Some Amazing Filesystem Benchmarks. Which Filesystem is Best?
http://linuxhelp.150m.com/resources/fs-benchmarks.htm

Compiling yourself a 2.6.23 Kernel (with Reiser4 support). (2.6.24 Kernel Patch)
http://linuxhelp.150m.com/installs/compile-kernel.htm

Installing your favorite Linux Distro on Reiser4.
http://linuxhelp.150m.com/installs/install-on-reiser4.htm

Installing GRUB on a Reiser4 Partition.
http://linuxhelp.150m.com/installs/grub-reiser4.htm

AND OTHER GOOD STUFF AT:

http://linuxhelp.150m.com/
http://linux.50webs.org/

[l00l]

I would strongly recommend against using reiser4 for real world use. I tried it and after several weeks it ate my filesystem and I was never able to recover data. Fortunately, that was on an experimental non-critical system.

Also note that in several benchmarks it is the only filesystem to crash and fail to complete the benchmark. For instance, from here:

http://www.csamuel.org/articles/emerging-filesystems-200709/

"The first attempt with reiser4 ended rather unhappily with a crash part way through the testing which killed the filesystem."

Which matches my personal experience wherein I found it very unstable.

I think there is potential to this filesystem, but it simply is not stable enough for production or desktop use at the moment. I think it's fine to play with on a non-critical experimental system, but based on my experience with it, I would urge caution if a stable FS is required.

[Jade]

Yes, the Reiser4 filesystem has been deliberately sabotaged on occasions. If you use one of Morton's older patches, you are likely to run into trouble,...

See, the Linux Kernel SABOTEURS @

http://www.phoronix.com/forums/showthread.php?t=9509

[lockin]

Did anyone ever see that Mel Gibson movie where he was worried everyone was watching him, and he had his house locked down and had all the conspiracy theories? Can't remember the name of the movie , but Jade missed

[Jade]

I have fixed the broken links (they should be fine till the conspiracy forces the next site closed (so far http://linux.coconia.net and http://linuxhelp.150m.com have been forced closed)

HANS REISER AND HIS FILESYSTEMS.

REISER4 HOWTOS.

Some Amazing Filesystem Benchmarks. Which Filesystem is Best?
http://m.domaindlx.com/LinuxHelp//resources/fs-benchmarks.htm

Compiling yourself a 2.6.23 Kernel (with Reiser4 support). (2.6.24 Kernel Patch)
http://m.domaindlx.com/LinuxHelp//installs/compile-kernel.htm

Installing your favorite Linux Distro on Reiser4.
http://m.domaindlx.com/LinuxHelp//installs/install-on-reiser4.htm

Installing GRUB on a Reiser4 Partition.
http://m.domaindlx.com/LinuxHelp//installs/grub-reiser4.htm

DISCUSS THE REISER4 FILESYSTEM HERE:

Linux FILESYSTEM BENCHMARKS (includes Reiser4).
http://www.phoronix.com/forums/showthread.php?t=12865

The HANS REISER Murder Trial. Timeline and Analysis.

http://m.domaindlx.com/LinuxHelp/politics/ReiserTrialSummaryAnalysis.htm
http://linux.50webs.org/politics/ReiserTrialSummaryAnalysis.htm

DISCUSS IT HERE:

http://www.phoronix.com/forums/showthread.php?t=7544

AND OTHER GOOD STUFF AT:

http://m.domaindlx.com/LinuxHelp//
http://linux.50webs.org/

[Hephasteus]

So if resier is getting modified so it stops screwing itself in the ear and causing problems and it slows it down its being "sabotaged".

So the car companies are deliberately making automotive engines slow and less powerful by using adequate material in connecting rods but not using adequate material in the connecting rods caused them to slam the pistons through the cylinder head.

Am I really missing something here?

[dacresbu]

Thats excluding ext4 and zfs. If the phoronix people get around to testing newer versions of these with a newer kernel...That would be cool.

comparing ZFS with raidz performance vs ext4+lvm or xfs+lvm. does reiserFS handle powercuts yet, then maybe I'll consider it.?

[squirrl]

Ext3, Reiser and JFS yet still even XFS all handle power cuts.

Bad ribbon cables, faulty harddrives, well that is another thing.

[martdj]

Hi all,
I'm in the process of migrating my server from Windows 2003 x64 with 3 VMWare VMs to Linux with the VMs in KVM. I thought as I'd like to try KVM that Fedora 12 would be a good host to use. Fedora by default uses Ext4, but from all these tests I wonder if I should not just use Ext3 for all my filesystems. I'll have a LVM of 2 disks which will contain movies, music, software etc and one separate disk (7200 RPM, while the others are 5400) for my VMs. What would you guys use as the filesystem?
One extra remark. I never see NTFS taken along in the benchmarks. I thought that the current NTFS driver for Linux performs quite well, or doesn't it? If NTFS would be a good choice, than that one would have my preference as then I don't have to move any data around (currently all drives have NTFS as filesystem).

[dacresbu]

these benchmarks are about 2 kernel versions old I THINK. Check the kernel versions and do some performance testing of your own before you go with my response. EXT4 is supposed to scale better with large files, note how, even then, the read and write speeds of lage files gets better. Also, seeking on large files should also be better just because its an extends based filesystem. You can put everything else on ext3 but that media should be better served on ext4