PyPIContents is an application that generates a Module Index from the Python Package Index (PyPI) and also from various versions of the Python Standard Library.

PyPIContents generates a configurable index written in JSON format that serves as a database for applications like pipsalabim. It can be configured to process only a range of packages (by initial letter) and to have memory, time or log size limits. It basically aims to mimic what the Contents file means for a Debian based package repository, but for the Python Package Index.

This repository stores the index. The actual application lives in a different repository.

For more information, please read the full documentation.

Getting started

About the Module Index

In the pypi.json.xz file you will find a dictionary with all the packages registered at the main PyPI instance, each one with the following information.

{
    "pkg_a":{
        "version":[
            "X.Y.Z"
        ],
        "modules":[
            "module_1",
            "module_2"
        ],
        "cmdline":[
            "path_1",
            "path_2",
            "..."
        ]
    },
    "pkg_b":{
        "version":[
            "X.Y.Z"
        ]
    }
}

This index is generated using Github Actions. This is done by executing the setup.py file of each package through a monkeypatch that allows us to read the parameters that were passed to setup(). Check out pypicontents/api/process.py for more info.

Use cases

• Search which package (or packages) contain a python module. Useful to determine a project's requirements.txt or install_requires.

import json
import lzma
from urllib.request import urlopen
from pprint import pprint

pypic = 'https://raw.githubusercontent.com/LuisAlejandro/pypicontents-build/master/pypi.json.xz'

with urlopen(pypic) as f:
    pypicontents = json.loads(lzma.open(f.read()).read())

def find_package(contents, module):
    for pkg, data in contents.items():
        for mod in data['modules']:
            if mod == module:
                yield {pkg: data['modules']}

# Which package(s) contain the 'django' module?
# Output:
pprint(list(find_package(pypicontents, 'django')))

Hint: Check out Pip Sala Bim.

Known Issues

1. Some packages have partial or totally absent data because of some of these reasons:

   1. Some packages depend on other packages outside of stdlib. We try to override these imports but if the setup heavily depends on it, it will fail anyway.
   2. Some packages are broken and error out when executing setup.py.
   3. Some packages are empty or have no releases.

2. If a package gets updated on PyPI and the change introduces or deletes modules, then it won't be reflected until the next index rebuild. You should check for the version field for consistency. Also, if you need a more up-to-date index, feel free to download this software and build your own index.

Getting help

If you have any doubts or problems, suscribe to our Discord server and ask for help. You can also ask your question on StackOverflow (tag it pypicontents) or drop me an email at luis@collagelabs.org.

License

Copyright 2016-2022, PyPIContents Developers (read AUTHORS.rst for a full list of copyright holders).

Released under a GPL-3 License.

Made with 💖 and 🍔

Web luisalejandro.org · GitHub @LuisAlejandro · Twitter @LuisAlejandro