Ubuntu 12.10 File-Systems: Btrfs, EXT4, XFS

[jabl]

Side note: Why is Btrfs so good at threaded IO? Like what the f*ck? Thats a huge jump up from ext4 and xfs. Is it because of BTRFS design or what?

COW "converts" a bunch of random writes into sequential writes; the downside is that the files will become more fragmented than with a traditional update-in-place filesystem.

[jabl]

Unfortunately, I am still stuck with the perf/regression testing, and still have to write a CBA report for the directors. The other option is to wait for our vendor (RHEL or OEL) to adopt the new FS as a default/recommended option and implement it during the next cycle.

ext4 is the default fs in RHEL6. If you're stuck on an older release, well, sucks to be you..

(Another big advantage of ext4 over ext3 in addition to the ones mentioned previously is much faster fsync, since ext3 cannot really fsync an individual file but rather the entire file system is synced when fsync() is called.)

[Zplay]

Michael, could you benchmark EXT4 with different journal size ?

From this article (2 years half old), it can make a big changes with small files : http://www.linux-mag.com/id/7666/

[Yfrwlf]

Dpkg is incredibly, horribly, painfully slow with btrfs due to fsync calls. Would have been nice to see that added to the benchmarks.

[dnebdal]

I'm 99,9% sure that's the case simply because physically the HDD in this test would never be able to achieve such high random write values. Even the fastest consumer HDD (Velociraptor 1TB) wouldn't be able to achieve even 1/10 of those 35MB/s.

Like jabl said, the way btrfs works effectively converts random writes into linear writes - and with that in mind, 35 MB/sec is fairly modest.

[highlandsun]

You should have also tested JFS. In my database tests, JFS outperforms all the others. Also, you should do a test with two HDDs (or a combination of HDD and SSD) where the journal for the main filesystem is stored on the 2nd device. Again, in my tests this can make a huge difference in overall throughput.

My results are posted here:
http://highlandsun.com/hyc/mdb/microbench/july/#sec11

[jabl]

You should have also tested JFS. In my database tests, JFS outperforms all the others

Sure, a filesystem which doesn't support barriers (such as JFS or ext2) will obviously outperform one which does (e.g. ext4, btrfs, xfs) on a test which tests synchronous writes (fsync()). In order to avoid an apples to oranges comparison, you need to either

- Disable the disk write cache when using a filesystem without barrier support (slower but safer).

- Disable barriers on the filesystems with barrier support (mount with barrier=0) (fast but unsafe).

- Or even better, use a device with a non-volatile write cache (e.g. a RAID card with battery backed cache) (fast AND safe).

[highlandsun]

Sure, a filesystem which doesn't support barriers (such as JFS or ext2) will obviously outperform one which does (e.g. ext4, btrfs, xfs) on a test which tests synchronous writes (fsync()). In order to avoid an apples to oranges comparison, you need to either

- Disable the disk write cache when using a filesystem without barrier support (slower but safer).

- Disable barriers on the filesystems with barrier support (mount with barrier=0) (fast but unsafe).

- Or even better, use a device with a non-volatile write cache (e.g. a RAID card with battery backed cache) (fast AND safe).

Hm, thanks for pointing that out. Yeah, I probably need to retest with disk write cache disabled.

Of course, in a dedicated database deployment, I would just have a single DB residing in a dedicated filesystem, and preallocate all of the space for the DB file(s). At that point, metadata updates are irrelevant, they would only be occurring for the mtime stamps and not for any structural changes, so FS structural corruption would be impossible.

[Dami55an]

I believe that fsck is as essential as it was with the previous generation of filesystems.

[jabl]

Of course, in a dedicated database deployment, I would just have a single DB residing in a dedicated filesystem, and preallocate all of the space for the DB file(s). At that point, metadata updates are irrelevant, they would only be occurring for the mtime stamps and not for any structural changes, so FS structural corruption would be impossible.

Yes, in such a situation a database engine can get away with using fdatasync() instead of fsync(). The filesystem still needs to have barrier support in order to provide data integrity guarantees when used on a device with a volatile write cache, though.