How to create plugin for PIP

[vit1251]

• Pip version: HEAD
• Python version: 2.7
• Operating System: Windows

Description:

How to create plugin to extends PIP with additional command for example I want to make more extendable requirement.txt format. How to do that?

What I've run:

I just only ask ;-)

[pfmoore]

There's no official API for that. We've had some discussions on the possibility (search the issues for details) but there's nothing at the moment.

[vit1251]

@pfmoore and what prevents to make public API? I can see lot of commands can be called from the module pip it very nice for automation tools. Or problem more deep and stoped by require improve the internal quality of the product?

[xavfernandez]

I'd say this is duplicate of #3121

[vit1251]

@xavfernandez no sure that you right. There I talk about plugin but public API is depends problem.

[pfmoore]

@vit1251 It's simply that there is no API for pip at the moment. Not for being called from other programs, nor for plugins. We're not against either of these in principle, but there are a number of significant issues that would need to be thrashed out first (check out the various related issues for details). And in all honesty, it's not a high priority for the limited available time the devs have.

I hope that answers your question, even if it's not the answer you'd hoped for.

[upendra-k14]

@pfmoore @ncoghlan I would like to dedicate my time for developing pip API. However, I would need some guidance on what should be structure of API. I worked on tkinter front-end for pip and had lot of difficulties using pip.main(). Some of the problems I faced were :

1. The GUI application needs to be restarted every time in order to reflect changes in the list of installed packages ( possibly due to cache issues, the python interpreter needs to be restarted).

2. I was able to direct the real-time output of pip install/uninstall command to tkinter page layout using subprocess module. However, using subprocess I was unable to capture the real-time information about the installation progress.

3. I tried other ways to extend pip, like inheriting the private classes like SearchCommand, InstallCommand and others. They worked well till the GUI application was single-threaded. However, when I tried to make the application multi-threaded using multiprocessing module, it turns out that all of the pip commands' classes are not thread-safe. ( I really don't remember the exact error )

4. pip doesn't has any specified format of output which is consistent with previous versions. It makes it difficult to parse output of pip commands when called through subprocess module. I think a standard JSON format will be very helpful.

[pfmoore]

Some very quick thoughts (sorry, not much time at the moment).

Your item 1 is not a pip issue, it's how the Python import system works. Installing new packages into the site-packages of a running interpreter is not safe - I'd be inclined to say that Python doesn't support it, but I don't think there's any explicit statement on this, so that's just my view. This is fundamentally the biggest issue with a "pip API" - any such API is designed to allow users to install packages into a running interpreter, and when things go wrong as a result, they will be reported as pip issues, when in fact they are simply cases of "you shouldn't do that". I don't have a good suggestion about how to resolve that, in the absence of support from the Python import system for changing site-packages on the fly.

Your item (2) is interesting, in that direct access to read pip's logs as they are produced would help here. But equally, realtime access to subprocess output streams is a standard issue, and a general "run a subprocess and capture its output as it is produced" routine for tkinter would address this without needing any change to pip. And should be perfectly possible - I'm sure I've seen other applications that manage it. (Although stdio buffering might hinder this in the general case - python -u -m pip solves it specifically for pip.) So while this could be presented as a use case for a pip API, there is a perfectly good workaround which uses pip as a subprocess.

Making pip thread-safe is likely to be a major undertaking. My immediate thought would be that any pip API should not promise thread safety in the first instance. Maybe if we had a stable API, carefully adding thread safety guarantees would be possible, but we're a long, long way from that state at the moment.

For (4), I don't see any reason we couldn't add some sort of machine-readable output option (much like we have pip list --format=json). A lot of the actual output would just be an unstructured blob of data, of course (the build output comes from setuptools, and so we can't structure that) so it's not clear how much benefit this would be in practice, but there's no reason in principle why other pip commands couldn't gain a --format=json option.

[dstufft]

FTR I agree with @pfmoore with the one correction that caching of installed packages (in the PyPI sense) as far as pip is concerned is not exactly a Python problem but a pkg_resources problem. However, caching of packages (in the import sense) is a Python project.

[dstufft]

I'm going to close this, we're unlikely to add a plugin API to pip.

[vit1251]

You can not go around the directory and look for the .py files and attach them to the pip entry point? This is an elementary task! Look this implementation:

for name in os.listdir('...'):
    if name.endswith(.py):
        m = load_source(name)
        self._plugins.append(m.pip_plugin_entry)

Plugins directory registry may be inside user home directory ~/.pip_plugins.

[vit1251]

PIP API is key element of most automatic deploying system. Every application before start should check environment. Every installation script and scenarios should uninstall and install package via API. Now most of this tools invoke os.system pip but it very stupid style for upgrading and update environment. Also PIP may provide pypi search and other similar utilites.

[ncoghlan]

@vit1251 The library APIs you are looking for are distlib and packaging:

• packaging: assorted utility APIs covering the formally approved interoperability standards
• distlib: assorted utilities for working with Python packaging, including interacting with the PyPI API and handling "de facto standard" formats not covered by approved PEPs

As the internal capabilities of pip stabilise into a supportable API, they get factored out into one or the other of these libraries depending on whether they're part of complying with an approved interoperability standard or not

As far as closing the issue goes, I agree - any build-time extensions will be based on pyproject.toml rather than being pip-specific, while install-time extension mechanisms should also be at least nominally tool independent (more akin to RPM file triggers than anything else).

[devxpy]

Disclaimer: This is not a pip python API (because of which this issue was closed). It's simply invokes scripts from within pip.

---

Following the discussion from #6081, I've successfully created a plugin system for pip that seems to work fine with existing packages.

Here is the branch I'm working on (see - plugin.py)

Why?

Well it's really because plugins are quite convinient. It allows users to extend pip in a very meaningful way.

cargo has had a very successful story with subcommands, and I would love to see this with pip. Even npm has scripts, which are considered almost essential to devel workflow in JS land.

This really has the potential to improve python devel workflow. pip plugins won't require any intervention from pip developers once the functionality is merged into master.

Plus the cross-virtualenv capabilities that this plugin system brings, means that us developers don't have to keep reinstalling common pip utilites on our virtualenvs!

And frankly, I don't feel like this brings substantial baggage to the pip codebase, to not consider having it.

I also presume that a lot of issues like #4551 can be just resolved using plugins, instead of modifying pip codebase, which I understand, could be a challenging task.

How it works?

• The search_plugin_scripts() function looks for executables prefixed with PIP_PLUGIN_SCRIPT_PREFIX (currently "pip-plugin-"). So it catches executables like pip-plugin-compile.
• It lists them in a 3rd party plugin section at --help. Auto-complete also works as of now.
• When invoking the script, it provides a single JSON encoded string, which helps the plugin determine the environment in which pip is running (see here).
• From there on, the plugin is free to do anything with this information.

This architecture allows for plugins that are independent of the actual package, which I think is quite nice.

It also solves the major recurring problem related to cross-virtualenv execution. Any plugins written with this architecture can be made so that they only need to be installed once on the global python, and then used from within a virtualenv.

I have also added a --disable-plugins switch.

How to actually write plugins?

plugingen is a small utility for generating a plugin. It allows exposing an existing python CLI as a pip plugin.

Also, Here is a proof of concept that contains some popular packages wrapped as plugins.

---

It's obviously not ready to be merged.

I need to run the tests (and write new ones).
There is no documentation either.

I would be happy to continue this work if you guys think it's a viable thing to do.

[devxpy]

Also, now that I think about it, this could also potentially solve @upendra-k14's use cases, since the plugin system allows overriding built-in commands.

Like He could just add a simple pre-install hook, and then reissue pip install with --disable-plugins

I should probably look into how this would result in security issues.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.