Gentoo's Portage 3.0 Stabilized With Much Faster Dependency Calculations

[phoronix]

Phoronix: Gentoo's Portage 3.0 Stabilized With Much Faster Dependency Calculations

Gentoo's Portage 3.0 package manager has been stabilized and ready to shine...

http://www.phoronix.com/scan.php?page=news_item&px=Gentoo-Portage-3.0

[ed31337]

That's a nice improvement, but 3 minutes still sounds like a long time to me. Maybe it would be a good idea to ditch Python for such heavy tasks?

[yoshi314]

it's not the fault of python. it's the complexity of how dynamic the dependency tree is. in gentoo, you have plenty of variables that affect it.

there is a c++ package manager that follows similar principles, called paludis. it used to work with gentoo (maybe it still does, but i doubt it) and it was not a whole lot faster.

the package format can be improved, quite a bit of time is spent in text parsing when calculating deps.

[Brane215]

Typical Python use case. Spare 2 weeks of coding so that you can waste 20 years on making it fast enough.

[reavertm]

That's a nice improvement, but 3 minutes still sounds like a long time to me. Maybe it would be a good idea to ditch Python for such heavy tasks?

It was for:
emerge -uDvpU --with-bdeps=y @world which has some extra non-default options (like upgrading build deps as well down the dependency tree) enabled. He could have added -N for good measure as well.
As someone said here, there is already portage compatible package manager written modern C++ and it's not much faster if at all. I think its author, Ciaran, former Gentoo dev (later he founded Exherbo - Gentoo alike distro), did a thesis on dependency resolution.
Gentoo tracks USE flags, slots (multiple versions of the same package), subslots (used to track ABI), KEYWORDS (marker to tell what CPU architecture is package available for + stability flag for that package in that arch). And there is also version-aware blacklist for packages and/or USE flags. All this makes dependency calculation whole lot more expensive than in basic binary package manager like rpm or dpkg.

[ermo]

it's not the fault of python. it's the complexity of how dynamic the dependency tree is. in gentoo, you have plenty of variables that affect it.

there is a c++ package manager that follows similar principles, called paludis. it used to work with gentoo (maybe it still does, but i doubt it) and it was not a whole lot faster.

the package format can be improved, quite a bit of time is spent in text parsing when calculating deps.

Paludis is the default package manager of Exherbo Linux.

IF you ask me, Exherbo is -- in many respects -- a better gentoo; yet at the same time, it's also its own thing so don't expect 1:1 compatibility.

[Serafean]

A bit of background, why I believe I have something to say about paludis and portage:
I have a couple of Gentoo installs, running for already 6+ years, maintained and updated. Some of these installs have been migrated to paludis, managed for about a year using paludis, and migrated back. I believe this gives me a bit of experience to compare the two...

So, about paludis:
It is more like a meta package manager. You can teach it new formats, new ways to install... It actually is a pretty neat idea. There was an experiment to create a perl (I think, maybe ruby) backend, which ultimately failed because the language's package manager was using (and still is) malformed yaml. Paludis could be the central place to install and manage packages, replacing pip, cargo, npm...

It currently doesn't work with Gentoo, because of unsupported EAPI 7, although there is a WIP branch to support that.
The migration back happened preciesly because of EAPI7.

I didn't perceive much of a speedup, but the configuration was much more sane than portage. Most of these features (repository configuration, use flag globs...) are bubbling back into portage.
Exherbo is a completely different beast, and I love the philosophy of "many repositories", but ultimately Gentoo has the manpower.

[grigi]

Dunno, a full emerge calculation run takes 22 seconds on my 4-year old i5 skylake notebook, depending on if there is conflicts or what.
That's fine for what it does.
It does depend on what you have installed, I have 1265 packages when I timed that (which is more than I expected, ah well)

[marios]

Using a scripting language for computationally complex tasks is a always a bad idea.
I am using Gentoo for some time and using python for package management is the only major flaw I have found.

Of course portage has to perform more complex calculations than other package manager but using a slow interpreted language just multiplies the time taken for the task.
In fact writing a simpler package manager in python would not be closely as bad as the blunder of writing a package manager as complex as portage in python.

I will still keep using Gentoo, since it is the only distro that meets my demands. However i strongly believe that writing portage in a scripting language is a blunder, that can only be fixed with a rewrite in a compiled language. I will also still keep cursing python every time I emerge while I watch the | spinning for minutes...

[caligula]

it's not the fault of python. it's the complexity of how dynamic the dependency tree is. in gentoo, you have plenty of variables that affect it.

there is a c++ package manager that follows similar principles, called paludis. it used to work with gentoo (maybe it still does, but i doubt it) and it was not a whole lot faster.

the package format can be improved, quite a bit of time is spent in text parsing when calculating deps.

Some food for thought:
- at least in the beginning, the c++ tool was a one man proof of concept. MAYBE it isn't as optimized as a mature multi-year high profile project maintained by a popular distro and lots of maintainers
- the c++ tool may use a different set of algorithms and metadata so it's not comparable at all
- if a computation isn't I/O bound, it's CPU bound (unless there's some deadlock or something). Parsing pretty much is CPU bound. If not, how about changing the granularity of package metadata storage

Python is significantly slower:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/python3-gcc.html