Ubuntu Still Unsure On Using XZ Packages

[phoronix]

Phoronix: Ubuntu Still Unsure On Using XZ Packages

While Fedora has been using XZ-compressed packages for their RPMs for a while now with having a greater compression ratio than Gzip, Ubuntu developers remain unsure of switching to using XZ compression for the Ubuntu 13.04 release...

http://www.phoronix.com/vr.php?view=MTIxOTY

[Sidicas]

Right now, just from watching the Debian mailing lists.
It looks like Debian is going to be using XZ compression on a very small number of packages for the sole purpose of cramming more content onto installation media that is targeted for Desktops.

It was discussed at DebConf that unpacking XZ files eats a lot of system memory (RAM) and if you don't have the RAM, the unpacking could take several times longer than expected. This may make using XZ compression a bad idea for tablets and cell phones which is important to Ubuntu right now (See the videos of Ubuntu running on cell phones / tablets at Ubuntu's UDS Youtube feed).

If you have the RAM (2GBs or more) then you're golden for XZ compression as it really isn't noticeably slower than the alternative.

Debian recognizes that a lot of hardware out there still has less than 2GBs of RAM which means switching entirely to XZ compression could be a *very* bad idea. So they're not going to do it for the Wheezy release for the overwhelming majority of packages and likely won't do it for Wheezy+1 either, but it's still up in the air.

Using XZ everywhere would be talking about bumping the RAM requirements of Debian from 64MB to 2GBs... That's one heck of a leap.

[ChrisXY]

You're not going to run a bloaty distribution like ubuntu on something with low ram... I hope.

Meanwhile archlinux has used xz packages for a while and is still unsure about using lrzip.

[brent]

The memory usage and performance problems may be tackled simply by reducing the compression setting. xz level 0 or 1 use very little memory and should decompress much faster than e.g. bzip2, with much better compression ration than old gzip.

[Shnatsel]

I'm not sure about RAM usage, but as far as decompression speed on x86 and x86-64 goes, LZMA/XZ are second in speed only to LZOP/LZ4. Well, for files filled with zeroes at least.
Here's some research I did back in the day, improving on some wildly inaccurate benchmarks I found in Wikipedia: http://www.webupd8.org/2011/07/free-...utilities.html

[grotgrot]

If Ubuntu/Canonical actually care about bandwidth then they'd fix their servers. It took me several attempts (and several regressions) to get them to reliably set etags. They were using the Apache defaults which include using the inode number so that meant each server gave a different etag which then prevented any kind of caching since each server was essentially claiming the same file was different. (DNS round robin meant you would hit different servers when downloading packages.) They eventually fixed this.

I then tried to get them to set correct HTTP caching headers. Package files can't change so they could be set to be forever cacheable. However the Canonical people refused to set such headers which means that even if there are http caches, they have to keep contacting the servers to double check the package files are still valid.

[jhansonxi]

How does XZ compare to 7zip or bzip2?

I've never liked tar/Gzip archives. Too slow to open large archives because they have to be completely decompressed first. I can get a directory listing of the contents of a an equivalent 7z archive much faster. The deb files are created with Gnu ar then Gzip, also requiring a two-stage extraction process FWICT.

[curaga]

You're not going to run a bloaty distribution like ubuntu on something with low ram... I hope.

Meanwhile archlinux has used xz packages for a while and is still unsure about using lrzip.

lrzip for packages? Dear God, unpacking one of those can take 8gb of ram. What the heck is their target audience?

[J___]

How does XZ compare to 7zip or bzip2?

I've never liked tar/Gzip archives. Too slow to open large archives because they have to be completely decompressed first. I can get a directory listing of the contents of a an equivalent 7z archive much faster. The deb files are created with Gnu ar then Gzip, also requiring a two-stage extraction process FWICT.

xz is just compression, in the same way as gzip - you generally still use tar as the archiver (or whatever archiver you want).

Where exactly are the memory concerns coming from? The xz manpage says:

The following table summarises the features of the presets:

                     Preset   DictSize   CompCPU   CompMem   DecMem
                       -0     256 KiB       0        3 MiB    1 MiB
                       -1       1 MiB       1        9 MiB    2 MiB
                       -2       2 MiB       2       17 MiB    3 MiB
                       -3       4 MiB       3       32 MiB    5 MiB
                       -4       4 MiB       4       48 MiB    5 MiB
                       -5       8 MiB       5       94 MiB    9 MiB
                       -6       8 MiB       6       94 MiB    9 MiB
                       -7      16 MiB       6      186 MiB   17 MiB
                       -8      32 MiB       6      370 MiB   33 MiB
                       -9      64 MiB       6      674 MiB   65 MiB

So using the default, 6, you only need 9MiB RAM to decompress.

As for speed, I'm sure they're right to look further into it, but pacman uses xz and is arguably the fastest package manager (and certainly the fastest I've ever used).

[strcat]

lrzip for packages? Dear God, unpacking one of those can take 8gb of ram. What the heck is their target audience?

https://bbs.archlinux.org/viewtopic.php?id=137918&p=1

It doesn't seem like it's really being considered at all for building packages yet (see the responses there from the devs), just talk of possibly adding it as another supported format in pacman through libarchive, once the work in the library is done.