Allow pip install to only install optional dependencies

[mwtoews]

What's the problem this feature will solve?

It seems there is a need to allow pip to install optional dependencies (or extras) independent from a package installation. For instance, packages use stand-alone requirements files that can simply be installed e.g.:

pip install -r requirements-test.txt

However, if these optional dependencies are migrated into a static pyproject.toml file following PEP 621, there is no equivalent to install only test dependencies.

Describe the solution you'd like

Unsure what solution would be best, but here are some suggestions:

# using a command option
pip install mypackage[test] --only-extras

# using syntax
pip install !mypackage[test]

Alternative Solutions

Install/uninstall method

Using the above example, install the package + test, then uninstall the package with two commands:

pip install mypackage[test]
pip uninstall mypackage

But there is obvious overhead to install the package (+ dependencies) then uninstall the package (without dependencies).

Copy/paste method

This method needs access to the sdist or source repository. Locate the source file where optional dependencies are declared, which may vary within pyproject.toml, python.cfg, python.py or dynamically loaded from another file. Copy and paste entries into a pip install ... command. This is often the quickest solution where there are only a few dependencies.

Duplicate requirements files method

Projects may duplicate their optional dependencies in requirements files. This can be manually maintained to mirror the information, or (e.g.) kept in sync with create a pre-commit script to generate requirements files.

Additional context

N/A

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

It seems there is a need to allow pip to install optional dependencies (or extras) independent from a package installation.

Can you provide some reference for this? Who needs it? Why is it needed?

[mwtoews]

It's perhaps not a wide-spread need, but I came across this today numpy/numpydoc#481 (comment)

[uranusjr]

If I understand the context correctly, the perceived need for the feature comes from the desire to put all requirements in pyproject.toml. But as the latter comment in the thread indicates, putting those requirements in a separate file instead of eliminate the need of the feature entirely, while satisfying the same needs. What is the motivation behind moving the requirements in the first place? What issues does the currently working solution have and can be resolved by moving the requirements into pyproject.toml, thus introducing the need of this new feature?

[mwtoews]

What is the motivation behind moving the requirements in the first place?

To provide a single pyproject.toml file with static optional dependencies, and not relying on tool-specific dynamic features to read other files. This could benefit 3rd party tools that don't follow these tool-specific behaviors.

What issues does the currently working solution have and can be resolved by moving the requirements into pyproject.toml, thus introducing the need of this new feature?

The current working solution for that specific project is to maintain a template pyproject.toml.in file and two requirements files, with some custom pre-commit logic to ensure the "do not edit" pyproject.toml is properly synced to the template. This is awkward. A better solution for the project is to have only one pyproject.toml file to specify optional dependencies.

---

Another scenario that I routinely come across is installing a project once, then later deciding that one or more of the extras is needed. This example is:

# build and editable install
pip install -e .
# later determine that "test" extras is needed
pip install -e .[test]

Both "install" commands may take a bit of time to do, for instance if Cython modules need to be created and compiled. However, if there was an option to only install optional dependencies, the second install would be much quicker.

[uranusjr]

I am definitely missing some context here, since the relation between templating pyproject.toml and being able to install extras without installing the original package seems ortogontal to me, and should be solved separately.

Both "install" commands may take a bit of time to do, for instance if Cython modules need to be created and compiled.

I would say this seems like a deciciency in how the build backend functions. Installing an already-installed editable package should only require minimal building work aside from metadata preparation (which will still be needed even if the title feature is implemented). If the above requires a lot of rebuilding, I would direct efforts into finding why the build backend cannot re-use existing built artifacts and has to do the work from scratch and attempt to find improvements there instead.

[mwtoews]

Perhaps too many details of an example PR are being interpreted here, as this feature request has nothing directly to do with pyproject.toml, so the context shouldn't be relevant to the core feature in this request.

---

Take an example "mypackage", which was packaged with core metadata containing:

Metadata-Version: 2.1
Name: mypackage
...
Requires-Dist: somethingelse
...
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'

• A basic pip install mypackage would install {mypackage, somethingelse}
• An extra pip install mypackage[test] would install {mypackage, somethingelse, pytest}
• This feature request is to enable a way to only install the "test" extra packages, or simply {pytest} in this example

[pfmoore]

The feature request is clear. What isn’t clear is the justification for it. Extras in pyproject.toml are not a replacement for requirements files, and there is no reason to assume that there is any need to migrate from the latter to the former.

As a result, your request reads like “we are trying to do this thing we don’t need to do, and it doesn’t work”. What you need to do is explain why you can’t continue to use a requirements file, if you want to justify this feature request.

[pradyunsg]

Previously discussed in https://discuss.python.org/t/adding-a-non-metadata-installer-only-dev-dependencies-table-to-pyproject-toml/20106 (IIUC).

[sigma67]

Requirements files are not published with the distribution, optional-dependencies metadata is.

How would you ship a requirements.txt?

[mwtoews]

What you need to do is explain why you can’t continue to use a requirements file, if you want to justify this feature request.

Some projects repeat the optional dependencies in one or more requirements files. These files need to be synchronized; here is an example of a pre-commit script to do that. This feature idea would not require duplication or synchronization, as they would be kept in one place and accessible via pip install command option. So it's a DRY thing.

How would you ship a requirements.txt?

There are lots of ways, unfortunately. E.g. setuptools uses a MANIFEST.in template:

recursive-include requirements *.txt

---

There are some complications with this feature idea. Upon reflection, I don't think using syntax (e.g. !mypackage[test] shown above) is good, since this is not part of PEP 508.

This idea would naturally open the door for other requests, e.g. to only install build-system.requires or project.dependencies from a pyproject.toml file. Maybe also useful ideas, but not directly what this feature idea is being discussed.

[ml31415]

It's painful to see, how after literally hundreds of people in at least three different issues ( #11440 #12100 #4783 ) requesting this feature, commenting and upvoting, some maintainers of this project keep trying to tell those people, that their use case is invalid and keep closing one issue after another on that topic.

It's 2023. Lots of people want to switch their projects to pyproject.toml. It's natural, that you want to get rid of requirements.txt, requirements-dev.txt and so on all together and use this new nice and shiny format. Instead of appreciating this change and making life easy for users, that were used to install requirements in that way, what we see is a litany of denial and education attempts.

Just to give you an example, how popular it was to install dev dependencies using pip install -r requirements-dev.txt for developers: A quick github search finds 87k mentions of this! Now you can either reeducate 87k package maintainers and developers and tell them that they were stupid to use pip in such a way, or you just accept, that pip might have been used in a way that you didn't intend, but worked well anyways, because it was easy and straight forward, and allow that to work with pyproject.toml files as well.

[pradyunsg]

Lots of people want to switch their projects to pyproject.toml.

The maintainers of this projects are also authors/approvers/implementers for the PEPs that led to the creation of pyproject.toml as well as the optional-dependencies features (and it's underlying mechanisms + implementation) -- we know about this user need/desire. :)

some maintainers of this project keep trying to tell those people, that their use case is invalid and keep closing one issue after another on that topic.

It's more that we don't want to have the same discussion in multiple places -- all the issues that you've linked to have been closed to consolidate them, or because the person who filed them closed it themselves.

This is something we've agreed on a design for, but at the end of the day, this is an extremely under-resourced open source project and there's limited amount of maintainer time available to help get this implemented. We have agreed on a design over on that issue, and help is welcome to implement this -- subject of course to the give-and-take of engaging with an open source project. :)

To that end, I'm actually also going to close this one to consolidate this into #11440, which is the current tracking issue for the feature being requested here.