Blosc Compressor Adds AVX-512 Bitshuffle: 20% Faster Compression For Zen 4 X3D CPU

[phoronix]

Phoronix: Blosc Compressor Adds AVX-512 Bitshuffle: 20% Faster Compression For Zen 4 X3D CPU

Blosc (c-blosc2) is a high performance compressor focused on binary data for efficient storage of large binary data-sets in-memory or on-disk and helping to speed-up memory-bound computations...

https://www.phoronix.com/news/c-blosc2-2.11-AVX-512

[geearf]

Could this be of any use for compressed FS?

[timofonic]

Could this be of any use for compressed FS?

I was going to ask the same...

[kieffer]

Could this be of any use for compressed FS?

Nop. Blosc is a meta-compressor: it mixes some pre-processor like bitshuffle (or many others) with one "classical" compressor like LZ4 or zlib, zstd ... It works great for data structured in block with values correlated. It is unlikely that bitshuffle improves the compression on a disk volume.

[geearf]

Nop. Blosc is a meta-compressor: it mixes some pre-processor like bitshuffle (or many others) with one "classical" compressor like LZ4 or zlib, zstd ... It works great for data structured in block with values correlated. It is unlikely that bitshuffle improves the compression on a disk volume.

Why would the shuffling not help in that situation but in another?

[kieffer]

Bit-shuffling works when you have integer values stored with several leading zeros. After bit-shuffling, bits of same "weight" are packed together and the most significant bits are all zero, thus the better compression rate. With general data you don't have any guaranty about the size of the objects stored.
Please, have a look at the publication: https://arxiv.org/pdf/1503.00638.pdf
Blosc is, in a sense, a generalization of this with different prefilters and compressors.