Linux 5.10 Btrfs Hitting A Performance Regression But Improving With Linux 5.11

[phoronix]

Phoronix: Linux 5.10 Btrfs Hitting A Performance Regression But Improving With Linux 5.11

Linux 5.10 as a Long-Term Support (LTS) kernel has been off to a rocky start after an immediate point release due to a RAID issue, some reporting AMDGPU problems, and also a staggering Btrfs performance regression hitting some users...

http://www.phoronix.com/scan.php?page=news_item&px=Linux-5.10-Btrfs-Regression

[polarathene]

These patches bring the performance up to around 40% higher than baseline. In the meantime we'll probably push this partial revert into 5.10 stable so performance isn't sucking in the meantime.

I indeed tested the linux-btrfs for-5.11 and found the performance some 50% better. I would hope that can be brought back to 5.9 levels sometime soon

What is the baseline here? Was the 2nd quote finding 5.11 BTRFS performed 50% better than 5.10 or 5.9 kernel?

I assume they're referring to baseline as 5.9, and that the partial revert they mention is to remove the regression in performance, pushing any perf improvement to 5.11?

[gavron]

Git rocks. Bisect changes rocks. Glad they found it and are backporting it!

Merry Christmas! If you don't celebrate, grab some gifts and happy New Year

E

[AJSB]

What is the baseline here? Was the 2nd quote finding 5.11 BTRFS performed 50% better than 5.10 or 5.9 kernel?

I assume they're referring to baseline as 5.9, and that the partial revert they mention is to remove the regression in performance, pushing any perf improvement to 5.11?

Sadly, baseline is 5.10 , so, its still way slower than 5.9 witch was already crippled compared with previous Kernels.

All in all, devs were making a terrible and sloppy job with BTRFS, a File system that, IMHO, is MANDATORY in terms of speed and reliability if you want to install Linux in any Mass Storage device that uses Flash Memory be it a SSD, NVMe, USB or MicroSD (YES, it works very well in a MicroSD in terms of reliability although speed is then equivalent to around HDD CMR in a MicroSD A1...do NOT use a A2) device....and even in HDD SMR if you want to use them only for Storage.


I hate VMs, they are the CANCER of program developing (including of Games)...devs are sloppy ( , to say the least, ) trusting on them.

[chrys87]

Nah, baseline is 4.19, follow the mailing list and don’t spread wrong information.

[matteo]

...MicroSD (YES, it works very well in a MicroSD in terms of reliability although speed is then equivalent to around HDD CMR in a MicroSD A1...do NOT use a A2) device....

Why not use A2? Aren't A2 cards faster than A1?

[AJSB]

Why not use A2? Aren't A2 cards faster than A1?

In theory , YES, in practice, NO.

The thing is Linux doesn't have the correct support for A2 (as opposite to Android OS) and as result, if you use uSD A2 cards, no matter File System, you will have worse performance than with A1 cards, specially with small/fragmented files.

This was several times tested and proved so.

NEVER use A2 uSD cards with Linux no matter with witch adapter and no matter what uSD A2 card manufacturer says.

PS:

*IIRC*, same problem happens with Windows.

[AJSB]

Nah, baseline is 4.19, follow the mailing list and don’t spread wrong information.

Thanks for clarification but the way was written previously in the threat was not clear that was in relation to 4.19....i wonder how 5.4 compared with 4.19 Kernel (FWIW, only with Kernel 5.4 i started to use BTRFS so i have no experience before that).

[polarathene]

The thing is Linux doesn't have the correct support for A2 (as opposite to Android OS) and as result, if you use uSD A2 cards, no matter File System, you will have worse performance than with A1 cards, specially with small/fragmented files.

This was several times tested and proved so.

Usually good to backup such claims with some sources?

TKaiser has put together a document last updated in Oct 2018 regarding the issue here:

https://github.com/ThomasKaiser/Knowledge/blob/master/articles/A1_and_A2_rated_SD_cards.md

Cache and Command Queuing require host (driver) support since the host needs to activate those new features first. The cache feature on A2 rated cards makes use of volatile RAM on the card requiring the host to learn new commands to issue flushing the cache

The Extreme Pro is even slightly slower than the Extreme Plus and only now I started to understand that A2 cards can only show their performance if the host is also 'A2 compliant' needing updated MMC drivers to deal with these new cards. It seems at the time of this writing this is still missing in Linux (tested with 4.19.0-rc4).

TKaiser is a name you'd recognize if you get involved with some communities or a popular embedded news site equivalent to Phoronix (CNX-Software). He often chimes in about I/O related things, sometimes it's good information, but sometimes IIRC it's not always correct or has become outdated.

---

Next up there is a more recent July 2019 article, it features someone sharing some insights/benchmarks against A1 vs A2 and complaining that A2 perf is terrible. This article has a few social platforms like reddit and hacker news discussing it. Interestingly the article lacks much background information about the test environment, no clue what kernel was used for example.

https://www.jeffgeerling.com/blog/2019/raspberry-pi-microsd-follow-sd-association-fools-me-twice

This was added back in Dec 2017 for the 4.16 kernel, CQE support, not sure about the cache bit.

https://github.com/torvalds/linux/commit/1e8e55b67030c6a2fef893d428bdcd611f73705c#diff-a1e2dabd8702d1d008f5a39b0d460d2e49f7ae4f59f756a0c1e31ee52051c0af

A2 products didn't start coming out until 2018 apparently, and from what I've read you would ideally have hardware A2 supported controllers, much like USB 3.0 gen1x1 isn't magically going to perform better with USB 3.0 gen2x2 capable device. There was mention that these features can work on compatible controllers by the OS directly, but those which are bridged over USB quite like USB-to-SATA bridge controllers may not passthrough commands properly (sometimes they do on Windows but not Linux, or require you to setup a udev rule, was especially common for TRIM and some SMART stuff).

And in May 2020, for the 5.7 kernel we received a host software CQE support apparently:

https://github.com/torvalds/linux/commit/511ce378e16f07b66ab78118587b7cc6ac197364#diff-a1e2dabd8702d1d008f5a39b0d460d2e49f7ae4f59f756a0c1e31ee52051c0af

we can see the software queue can improve some performance with 4K block size, increasing about 16% for random read, increasing about 90% for random write, though no obvious improvement for sequential read and write.

A 90% random 4K write improvement if your device supports offloading to the host, similar to some SMR based HDDs.

---

So... from when and what are your bad experiences with A2 based on? Support for new HW features take time, Linux as you should know can lag behind a bit on that front. I am not familiar with the RPi4 HW support for A2, if it lacks you may have to wait until a future revision or next iteration/version product release I guess. To say never bother with A2 just seems silly.