Issue #2478: Topological installation order

[rbtcollins]

This fulfils the request in Issue #2478 - install dependencies in topological order.

The finished order is:

• anything install_require'd before the requiring thing
• as given for user supplied requirements where compatible with the install-requiring constraint

Examples:
pip install foo bar, where foo depends on bar will install bar, then foo.
pip install foo bar, where foo depends on baz will install baz, foo, bar.

[xavfernandez]

I don't understand why we need a topological ordering of installs since it has been agreed that setup_requires won't end up in the target environment ?

[qwcode]

per your readme update, the 3rd item should be "TopoRequires", right?

[rbtcollins]

nuts,yes.

[rbtcollins]

@xavfernandez this has nothing to do with setup_requires. Topological sorting is an existing feature request predating my work on setup_requires, but which increases the reliability of building and installing things - because there are setup.py's where egg_info can be run, but install/bdist_wheel etc can fail if the dependencies have not been installed. Because we can't have nice things. This is why folk have had to tweak install orders in the past, because pip was installing in 'first-found' or 'last-found' order, neither of which solve this generally.

[xavfernandez]

there are setup.py's where egg_info can be run, but install/bdist_wheel etc can fail if the dependencies have not been installed.

This is the goal of setup_requires.

I'm afraid implementing topological ordering will send this message to users: "You don't need to declare setup_requires dependencies, pip should figure it for you".

[rbtcollins]

I don't dispute that. But if the pip committers didn't want topological sorting they would have closed that issue WONTFIX, right?

[dholth]

+1 on toposort. No one should have to manually order their dependencies.

[xavfernandez]

+1 on toposort. No one should have to manually order their dependencies.

No one has. People only have to use the tools correctly (i.e. use setup_requires if their setup.py requires something special).

This is even less useful/desirable with wheel-caching coming: it will be much less painful to have numpy as setup_requires and install_requires, since it will only be built once as wheel.

[dholth]

Topological sorting only means that all of a package's requirements get installed before that package is installed. It does not have anything to do with setup requirements.

[qwcode]

It does not have anything to do with setup requirements.

practically speaking, it does. to say again, this PR intends to resolve #2478, which was motivated by a user wanting a topo sort for the example of "h5py" needing "cython" to run a build extension. A topo install is not needed if "h5py" setup-requires "cython". Although much of the conversation for "setup-requires" has focused around wanting to do module level imports in setup.py itself, this scenario of needing requirements in place for build extensions is something that is literally mentioned in the setuptools docs: " [setup_requires] is needed if you are using distutils extensions as part of your build process".

[pfmoore]

To put it another way: Any given pip installation should be seen as atomic - everything requested (and their dependencies) gets installed at once, or not at all. The order pip does the tasks it needs to to achieve that is an implementation detail, and should not matter. If it does, that is a bug in the packages and how they have specified their requirements. All the user should have to care about is that when they do pip install X Y Z, something happens and when it is finished, X, Y and Z are available and work (in terms of all their dependencies are present). Or there's an error and nothing gets installed.

[pfmoore]

... which is not to say that this change won't help users deal with broken package specs, but that's all it does.

[rbtcollins]

Ok, so I think I've figured out some of the cognitive dissonance here. Coming from a distro background it didn't even occur to me until I got up this morning. Package managers like apt and yum (and their deeper plumbing, dpkg and rpm) try very hard to avoid leaving the users system in a broken state. pip also has this goal - see for instance @dstufft 's requirement that we don't install anything until we know we can resolve all the dependencies satisfactorily.

The reason for topological sorting then, is not so that buggy setup_requires will work, but correctness: we should not have packages installed without their dependencies. topological sorting is a first approximation to this goal.

A second, closer approximation would involve attempting to maintain axioms across all installed packages e.g. when A and B both have versioned deps on C, don't allow an upgrade or install or B if that would break the versioned dep A holds, even transiently.

This can be made easier by a third thing - atomic replacement of packages, which requires making the installation and removal of packages atomic, which its not today. But I don't want to get into perfection here; acting in topological order is clearly much closer to 'all axioms maintained at all times' than installing in either first-discovered or last-discovered orders.

[dstufft]

In particular, a problem that happens is if you do pip install thing-that-depends-on-foo, and you install thing-that-depends-on-foo before you install foo, then if foo fails to install for whatever reason (perhaps missing dev headers?) you cannot repeat the instal command to get it to install everything else because thing-that-depends-on-foo is already installed.

There are of course other ways of handling that, like making pip install <whatever> re-evaluate the dependencies to make sure there are no missing dependencies (which isn't itself a bad idea), but installing things in this order just makes sense I think.

An option to be more "atomic", is to have the other PR that @rbtcollins is working on (the auto build wheel PR) build all the wheels first, before installing any wheels. That would move us into a situation where we ensure we can build everything before we attempt to install anything. That isn't particularly relevant to this PR though, which I think is an entirely reasonable thing to do.

[rbtcollins]

@dstufft actually we re-evaluate the whole dep chain 'pip install thing-with-missing-dep' will install the missing dep. But its better if thing-with-missing-dep not be installed at all, of course :)

[xavfernandez]

In particular, a problem that happens is if you do pip install thing-that-depends-on-foo, and you install thing-that-depends-on-foo before you install foo, then if foo fails to install for whatever reason (perhaps missing dev headers?)

Thanks, that's indeed a good justification for this PR. I thought pip was able to deal with that in an atomic way and rollback the whole thing.

Indeed the solution to build all the wheels first before installing anything seems like an ideal goal 👍

[qwcode]

a problem that happens is if you do pip install thing-that-depends-on-foo,
and you install thing-that-depends-on-foo before you install foo,
then if foo fails to install for whatever reason (perhaps missing dev headers?)
you cannot repeat the instal command to get it to install everything
else because thing-that-depends-on-foo is already installed.

what? pip handles this now. pip will see thing-that-depends-on-foo is already installed, but still proceed to install foo

[dstufft]

I'm remembering some of the details wrong, the issue was #2151 but I don't recall what exactly it was now... but I remember that reversing the installation order so that dependencies got installed before the thing that depended on them solved it.

[qwcode]

we should not have packages installed without their dependencies

more generally, we shouldn't partially fulfill a user's install request. all or nothing, i.e. support rollbacks.

but I agree, this is an approximation, and like I said, I don't dislike the topo sort, I'm just still not a fan of the wording in the docs and docstrings. it's going to promote people thinking the sort has to be this way just due to the dep chain itself (i.e. that pip can't install A before B if A install-requires B)

[rbtcollins]

But pip shouldn't install A before B if A install-requires B, because that isn't a valid install of A. What exact prose would you like in the docstring - I don't want to try and guess at what will be acceptable to you.

[qwcode]

as noted in IRC, I'll log another PR with ideas on changing the docs and comments