Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No package inspection routines outside pkg_resources #317

Open
jaraco opened this issue Feb 8, 2020 · 9 comments
Open

No package inspection routines outside pkg_resources #317

jaraco opened this issue Feb 8, 2020 · 9 comments

Comments

@jaraco
Copy link
Member

jaraco commented Feb 8, 2020

One of the goals of importlib.resources and .metadata was to replace Setuptools' pkg_resources. There's still a big gap that pkg_resources supplies that these packages do not, and that's inspection of the environment for the presence of a package. In particular, the user wants to know:

  • Given a package specification (requests>=2), is that package installed and if so, what package is it?
  • Given a package specification, what are all of the dependencies of that package as installed?
  • Given a set of requirements, are those requirements satisfied?

For example, pip-run relies on pkg_resources to determine if a package is installed at a given spec.

When trying to re-implement these behaviors using importlib_metadata, I find it's clumsy and complicated, because it requires substantial implementation by the client but also because it usually depends on the third-party packaging project, which because it isn't available in the stdlib, means it's not possible to have this behavior in the stdlib. The objects and models in packaging.version and packaging.specifiers and packaging.requirements are essential to performing some of the desired operations without setuptools.

Put simply, it's difficult to find a home for the functionality which necessarily overlaps the functionality in packaging and metadata.

What's to be done about this?

I've been mulling this over and every time I think about it, I come to the same conclusion (proposal):

  • importlib.metadata should be ported to packaging.metadata
  • packaging should grow another module for inspecting environments, such as .inspector or .environments.
  • to retain the stdlib support for metadata, packaging should be included in stdlib

That third aspect is certainly debatable, and maybe the conclusion is that this behavior remains third-party indefinitely.

@warsaw Do you have any instinct on this matter? @dstufft I know you have had some thoughts on this matter; what would you recommend?

Is there a promising alternative to this issue that I haven't considered?

@pradyunsg
Copy link
Member

(initial reactions based on reading this and thinking about this for a bit before dinner)

tl;dr: (1) let's avoid (2) let's do it (3) let's not do it

packaging.metadata

Is there any reason to avoid making packaging depend on importlib.metadata? If not, I'd much rather that packaging.environments build on top of the work already done in importlib.metadata.

packaging.environments

This sounds like a good idea to me and I'm on board for adding something like this. We'd definitely want to spend some time figuring out a good design for this functionality; much like we did for packaging.tags and importlib.metadata.

That third aspect is certainly debatable, and maybe the conclusion is that this behavior remains third-party indefinitely.

I strongly prefer that packaging stays out of the standard library, mostly due to the "locked at the CPython release cadence" aspect of improvements to it. I'd prefer we keep the the people-are-using-an-older-version related problems contained to the "big fish" like CPython, pip, setuptools, etc.

With my pip maintainer hat on, it is much easier for us to evolve + fix issues, with packaging being outside of the standard library. e.g. during the roll out of packaging.tags in pip 20.0, we needed to fix a fairly nasty bug. We were able to make a new packaging release containing the bugfix, updating the version vendored in pip; making no other pip-code changes and making pip release -- all in the span of a less than a week, with most of the week spent waiting on PR review for the initial bugfix.

With my packaging maintainer hat on, I like the low overhead and simple workflows we have for the project right now. Doing a transition + adding complexity in our usage patterns/adoption, for not-much-benefit isn't something I am really excited for.

@uranusjr
Copy link
Member

uranusjr commented Feb 9, 2020

I’m wondering, what exactly should go into (say) packaging.environments? My instinct to finding whether a distribution is satisfied is no more than a few lines of glue code:

from importlib.metadata import distribution as get_distribution
from packaging.specifiers import SpecifierSet
from packaging.version import parse as parse_version

def pkg_installed(name: str, spec: SpecifierSet):
    if (dist := get_distribution(name)) is None:
        return False
    return parse_version(dist.version) in spec

(Note: there is a functional difference between this and pkg_resources.require() due to pkg_resources also recursively checks whether dependencies are satisfied. I am personally not fond of the behaviour, but it’s not difficult to replicate that as well.)

(Minor grumble: I really wish SpecifierSet.__contains__() could accept a str instance and parse it automatically. -- Edit: Apparently it already does, I must be confusing it to something else.)

I would find it hard to justify making packaging hard-depend on importlib.metadata if this is the case, and suggest maybe putting the sample somewhere for implementors to reuse instead.

@jaraco
Copy link
Member Author

jaraco commented Feb 10, 2020

Consider also this use-case, where retrieving the list of dependencies of a given package is far from trivial. I'd really like to avoid recommending that users copy/paste whole segments of code, especially when those segments have multiple imports.

@uranusjr
Copy link
Member

O markers and extras, the bane of us all. Thanks, that makes perfect sense.

(Minor grumble 2: The whole things seem to be complicated a lot by all the if req.marker is None checks. I wonder if it’d make sense to allow Marker('') or add an EmptyMarker() stub that always return True for evaluate().)

I still wonder whether it is possible to avoid coupling importlib.metadata with packaging. My mental distinction between the two is:

  • importlib.metadata parses package metadata.
  • packaging makes sense of the parsed metadata.

I would definitely be +1 to include such logic as packaging.environments or similar if hard dependency can be avoided (my feeling is it’s possible if we use str and EmailMessage as the interfacing types).

@jaraco
Copy link
Member Author

jaraco commented Feb 16, 2020

Another area of interaction is between pep517 and importlib.metadata, illustrated by importlib_metadata MR 113 (loading metadata for an unbuilt package). Does that belong in packaging also?

@jaraco
Copy link
Member Author

jaraco commented May 26, 2020

In pypa/setuptools#2116, a user has this case:

  • for a package, enumerate its extras and determine which extras are satisfied (requirements installed at their required versions)

@ncoghlan
Copy link
Member

Regarding the stdlib-inclusion question, I think packaging would make a lot of sense as something for "ensurepip" to explicitly install as a public guarantee.

@pfmoore
Copy link
Member

pfmoore commented May 27, 2020

If we're going down that route, maybe the importlib.resources backport (on Python versions that don't already have it) would be a good idea too?

@jaraco
Copy link
Member Author

jaraco commented Apr 18, 2023

Another use-case requested in python/importlib_metadata#450:

# an exception is thrown if any transitive dependency is not met
pkg_resources.require(['Werkzeug>=0.6.1', 'Flask>=0.9'])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants