'pip.commands' entry point for pip command plugins

[qwcode]

this adds support for external projects adding pip commands using a new 'pip.commands' entry point.
pretty simple. more could be done with the help formatting.
see the test for an example project using the entry point.
#1061 mentions doing this with "git style redirection" (with an example gist link). the example was about execution. it was unclear to me how it provided a solution similar to entry points, but maybe I was missing something.

[dstufft]

I'd much rather not use entry points for this. The benefit of the git style redirection is that it doesn't add further reliance on setuptools/pkg_resources. Right now our use of pkg_resources can be removed as it's all internal stuff, but this change cements a dependency on setuptools/pkg_resources as a public API.

[pfmoore]

I agree with @dstufft. Given that we're slowly working on removing pip's dependency on setuptools, adding a public API that relies on it seems like a backward step.

[jezdez]

I also agree with @dstufft.

git-style command execution actually means two things:

• some code to discover commands like shown in https://gist.github.com/dstufft/5a12c2fd7194ee7100bb
• a stable API of the Command class and a simple way to hook it up with a CLI tool like optparse.

So I wonder if we should make other parts of pip public, other than the Command, e.g. a "simple" runner for users to hook the command class into?

[pfmoore]

Having said that, I'm not keen on the "git-style" approach of exec-ing a new command for each subcommand (or even just for non-core commands. It's not a typical pattern on Windows, where process creation is slow, and there is no exec (so you have to spawn a subprocess and wait for it).

I'd much prefer an in-process solution, which means (following the split @jezdez suggests) that the code to discover commands needs to introspect the importable Python modules - pkgutil.walk_packages is available for Python 2.7+. I think it also existed (but was undocumented) for 2.6, but I don't have a copy to hand to check.

[qwcode]

hmm, entry points are pervasive. if we feel like we have to run from it (and there's no way we could transition over, when there's an alternative later), what does that tell all the other projects who are using it?

like paul, I'm not so keen on a bunch of "pip-*" scripts.

code to discover commands needs to introspect the importable Python modules

this is my question with the "git style execution" idea.
we either have to walk code to find command classes, or offer some kind of registration api? right?
what's the alternative to one of those two?

[dstufft]

I'm not sure what pkgutil.walk_packages would do in this regard? The goal is to allow anyone to write pip commands yea? In order to do that we need some method of lettings users register arbitrary commands. Afaik pkgutil.walk_packages would require knowing ahead of time what packages you want to walk.

I'm not saying we need to use the git style redirection but if we do something to allow folks to register commands I don't think pkgutil.walk_packages helps in that.

[dstufft]

Other projects are in a different situation than pip. They can depend on setuptools and use pip to install it (in the long term). Whereas our dependency on setuptools has been a cause of constant problems and makes the bootstrapping of pip harder. Ideally we'll be moving to be more self contained by eliminating the setuptools dependency.

[qwcode]

Afaik pkgutil.walk_packages would require knowing ahead of time what packages you want to walk

I imagine @pfmoore was combining this with the pip-* idea where you would know what to walk, but I'm not sure.
I assumed registration would be the non-entry-points way to handle it.

So I wonder if we should make other parts of pip public, other than the Command

Minimally, a public Command api, and a registration api. To do anything useful, many commands would want to use req.* classes and index.* classes, but I'm not sure we're ready to start maintaining a guaranteed api for all that..

[dstufft]

Git style redirection doesn't involve walking any packages, pip install simply invokes pip-install. So instead of having a single entry point you have multiple entry points and pip itself becomes more library like with commands calling into the library to do the actual work.

A registration API doesn't work exactly like that, since you need some method to discover the things to register. Entrypoints uses project metadata and git style redirection uses the systems PATH variable to discover things. Whichever thing we pick we need some method of discovery/redirection and IMO it shouldn't rely on anything that can't be implemented in pip or vendored in pip._vendor.

[qwcode]

Other projects are in a different situation than pip ...
Whereas our dependency on setuptools has been a cause of constant problems

to process entry_points, we just need pkg_resources, and we've been talking about vendorizing that into pip.

Git style redirection doesn't involve walking any packages, pip install simply invokes pip-install.
git style redirection uses the systems PATH variable to discover things

redirection alone is not a full solution, right? In order to put together help etc... pip has to know about all the command classes. So, it has to involve a combination of discovery, and then some kind of introspection to find the command classes. the latter is not what I'm clear on in your idea?

[pfmoore]

I'm not sure what pkgutil.walk_packages would do in this regard?

My (half thought out) idea was to walk all the available packages looking for ones with a specific naming convention. Alternatively (and equally not thought through), we could use namespace packages and have the pip command xxx import something like pip-contrib.commands.xxx and run its main() function.

The main thing I want to avoid is looking for commands on PATH and running subcommands as separate processes. That sucks on Windows.

Registration is obviously better than discovery, insofar as it doesn't rely on naming conventions, but I'm not clear where a registration API would get called, or where it would store the information on registered commands.

In all honesty, setuptools entry points are a perfect fit for this type of problem. As with many setuptools features, though, the feature itself is a good idea, but the implementation is sufficiently controversial that it becomes a real stumbling block. The exports extension to Metadata 2.0 (see PEP 459) effectively replaces setuptools entry points - I'd hate to tie ourselves to "old-style" setuptools features that are being phased out in Metadata 2.0, and yet Metadata 2.0 is clearly not yet ready for us to rely on. The problem is that presently there's not even a clear view on what the migration path will be from one to the other :-(

[dstufft]

So essentially

a) We can use entry_points and tie ourselves to a legacy system with an as of yet undefined migration path (Works Today)
b) We can work on the Metadata 2.0 exports and get that ready and use that (Works in the Future)
c) We can defer this feature until a later time.

I actually realizes that using git style redirection will be a footgun because unlike git where you have typically 1 installation, with pip you have one installation per virtualenv so you'd need to install each plugin into each virtualenv. With git style redirect pip command-1 could operate in the virtualenv and pip command-2 could operate outside of the virtualenv.

[qwcode]

btw, my motivation for this is 2 things:

1. To push out "pip bundle" into a set of 3rd party command extensions, before deleting it from the core.
2. I have a number of pip ideas, and it would be easier to do them first as extensions, vs trying to make them core commands

the idea that we would wait for Metadata 2.0 is depressing.

sending email now to distutils-sig to better understand what the migration path might be for entry_points.

[dstufft]

That being said, it's not quite as nice but you could implement them as pip-bundle and pip-experiment which means when they move in and out of core the invocation changes would be minimal for the end users (pip-bundle vs pip bundle). That's what pip-tools does https://github.com/nvie/pip-tools

[qwcode]

That being said, it's not quite as nice but you could implement them as pip-bundle and pip-experiment

yes, that's my fallback, if we don't go with entry_points. I don't want to mess around with anything else.
It seems to me since we're advocating setuptools as the "now" solution, we should be ok with eating that dog food, and just know that the few hardy folks that might implement a pip extension, will be able "migrate" to PEP459 later, however that turns out.

[pfmoore]

I would be OK with going for entry points as a temporary solution provided:

1. We vendor pkg_resources at the same time so that we don't depend on an external setuptools
2. We clearly document this as a temporary solution, and any 3rd parties using the pip extension mechanism must be prepared to switch when pip does (i.e., we won't guarantee any sort of compatibility or deprecation period).

Actually, having a concrete use case for Metadata 2.0 exports could help push progress...

[qwcode]

must be prepared to switch when pip does

from the distutils-sig thread on this, it's sounding like there should be no fear of transition and compatibility.

let's consider 2 cases

1. someone publishes an sdist, PipPlugin-1.0.tar.gz up on pypi, that uses "entry_points" keyword
2. someone publishes a wheel, PipPlugin-1.0.whl that has an old-style "entry_points.txt" baked into it.

and then PEP426 goes online, and pip/setuptools refactor themselves to be compliant, i.e. to generate and consume metadata 2.0 (when it's present in a distribution)

for #1, the new setuptools would build out the entry points metadata (during the pip install) to be PEP459 compliant.

for #2, the pip installer would be responsible for converting it during the install (or maybe just forcing a rebuild when it detects metadata < 2.0)

vendor pkg_resources at the same time so that we don't depend on an external setuptools

why is that strictly necessary?

[pfmoore]

vendor pkg_resources at the same time so that we don't depend on an external setuptools
why is that strictly necessary?

Mainly because I don't want to have use cases that currently don't need setuptools to suddenly start needing it. Here, I'm mostly thinking of pip install <wheel> which works without setuptools (and so can be used to install a setuptools wheel). I'm also not sure how much of the mechanisms virtualenv uses to bootstrap pip will work if pip needs setuptools for basic command parsing.

[qwcode]

don't want to have use cases that currently don't need setuptools to suddenly start needing it.
Here, I'm mostly thinking of pip install which works without setuptools

I don't follow. pip currently requires setuptools in all cases (even just installing wheels), due to pkg_resources imports all over the place.

but as for vendoring pkg_resources, independent of this issue, I'm all for it. let's do it. we just need a convention on where to get it from, and how to mark which version we have.

[pfmoore]

Here, I'm mostly thinking of pip install which works without setuptools

I don't follow. pip currently requires setuptools in all cases (even
just installing wheels), due to pkg_resources imports all over the place.

Maybe I'm misremembering something. My apologies if so.

[qwcode]

assuming that:

1. we think plugins can make it easy for people to add features and work on pip
2. we'd ultimately like to use 2.0 exports for plugins (http://www.python.org/dev/peps/pep-0459/#the-exports-extension)
3. we don't want to wait for 2.0
4. there's a decent migration path from entry_points to exports (and not for any other home-grown plugin system)

I think that leads to merging this...
worse case scenario, it never catches on, and it's an easy removal later. it's a small change.

[pfmoore]

OK, I checked. Indeed, even pip.main cannot be run without pkg_resources (1.5rc3). My mistake.

I guess that means there's no real reason to object to this feature. At the moment it's undocumented, so worrying about it being a "public" API using setuptools features is probably premature. When (if) we do document it, let's just make sure then that it's documented to avoid locking us into setuptools.

[qwcode]

after much fretting, deciding not to do this for now. maybe later if demand arises.
staying conservative, and will make do with subclassing and "pip-*" console scripts.