Make Distribution._dep_map into a public attribute

[Jackenmen]

Would it be possible to make Distribution._dep_map from pkg_resources API public attribute? It seems quite useful to be able to know which extra has what requirements, for example - to determine what extras of the package are installed:

pkg = pkg_resources.get_distribution("MyPackage")
installed_extras = []
for extra, reqs in pkg._dep_map.items():
    if extra is None:
        continue
    reqs = [x.name for x in reqs]
    try:
        pkg_resources.require(reqs)
    except DistributionNotFound:
        pass
    else:
        installed_extras.append(extra)

[jaraco]

I'd recommend instead of using pkg_resources, to use importlib.metadata for this need. Something to do with importlib.metadata.requires('MyPackage'). You'll still need to call pkg_resources.require. Let me know if that helps.

[Jackenmen]

importlib.metadata.requires only gives raw strings of all requirements as given in Requires-Dist - I would have to manually parse the markers and somehow figure out what belongs to each extra and that doesn't seem trivial.

I guess I could just use output of Distribution.requires(extras=[extra_name]) which might be slower (I haven't measured so no idea if it would actually be significant) because of bigger amount of requirements to "require" but I wouldn't have to use the internal _dep_map then:

pkg = pkg_resources.get_distribution("MyPackage")
installed_extras = []
for extra in pkg.extras:
    try:
        pkg_resources.require(pkg.requires(extras=[extra]))
    except DistributionNotFound:
        pass
    else:
        installed_extras.append(extra)

I guess I'm just surprised it's not really possible (in a supported way that is) to only get requirements of extra without requirements of main package.

[jaraco]

Totally fair. I'd recommend passing the results of requires() to packaging.requirements.Requirement to inspect the contents.

I do agree, it's annoying there isn't something more sophisticated to achieve what you need, but that's the world we live in for now.

[Jackenmen]

I'd recommend passing the results of requires() to packaging.requirements.Requirement to inspect the contents.

Problem with using packaging.requirements.Requirement is that the only thing telling you about the extra is packaging.markers.Marker in .market attribute which only allows you to evaluate it, not inspect it which makes it a little more complicated (though doable):

import importlib.metadata
import packaging.markers
import packaging.requirements
import pkg_resources

dist = importlib.metadata.distribution("MyPackage")

# funnily, ↓ this line isn't part of stable API as it's a method of email.Message
extras = dist.metadata.get_all("Provides-Extra")

raw_reqs = dist.requires
parsed_reqs = list(
    filter(
        lambda req: req.marker is not None,  # if no marker, it can't be extra req
        (packaging.requirements.Requirement(raw) for raw in raw_reqs)
    )
)
installed_extras = []
for extra in extras:
    extra_reqs = []
    for req in parsed_reqs:
        try:
            req.marker.evaluate()
        except packaging.markers.UndefinedEnvironmentName:
            # if above doesn't raise an error, it means it isn't extra req
            if req.marker.evaluate(environment={"extra": extra}):
                extra_reqs.append(req.name)
    try:
        pkg_resources.require(extra_reqs)
    except pkg_resources.DistributionNotFound:
        pass
    else:
        installed_extras.append(extra)

I tested a lot of stuff before making this issue 😄 Anyway, I think it's great that all those APIs for packaging exist in Python, even if I can't do everything with them in a way I want to.

I currently use the Distribution._dep_map even though I know it can break (I found out we were already using it for something else so that convinced me to use it, besides the fact it makes handling of extra reqs very easy), but this is actually the first time I wrote down what it would take to handle this with packaging (previously I just left it at "it's too complicated to bother") so I might think of switching just to avoid potential point where my package could break in future. I would have to test out how it performs first but there's hope 👍

[jaraco]

Yikes. That's clumsy. I'd recommend exposing that functionality in importlib.metadata, except that it's a low-level interface and so doesn't have access to the classes in packaging. Similarly, I don't think packaging has an interest in having a dependency on importlib.metadata, so it seems there's a need for a package that sits above those and provides interfaces that consume both. In pypa/packaging-problems#317, I'm trying to resolve this problem.

Short answer is that I'm trying to supplant the functionality in pkg_resources, so I'd rather not expose more functionality to deprecate, but this use-case helps the PyPA understand the need and hopefully there will be a better solution soon.