Skip to content

Commit

Permalink
add section: File resolution (DavHau#434)
Browse files Browse the repository at this point in the history
  • Loading branch information
milahu authored Apr 25, 2022
1 parent 4dde445 commit 0901d64
Showing 1 changed file with 55 additions and 0 deletions.
55 changes: 55 additions & 0 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Table of Contents
* [Why nix?](#why-nix)
* [How does mach-nix work?](#how-does-mach-nix-work)
* [Dependency resolution](#dependency-resolution)
* [File resolution](#file-resolution)
* [Generating a nix expression](#generating-a-nix-expression)
* [Contributing](#contributing)
* [Limitations](#limitations)
Expand Down Expand Up @@ -203,6 +204,60 @@ As core for the resolving resolvelib is used: https://github.com/sarugaku/resolv

Mach-nix supports multiple providers to retrieve python packages from. The user can specify which providers should be preferred. Packages from different providers can be mixed.

### File resolution
With `pypi-deps-db`, we have built a dependency graph, where each package is defined by name and version, for example `pillow-9.1.0`.

To download a package, we need it's URL and hash. This is where [nix-pypi-fetcher](https://github.com/DavHau/nix-pypi-fetcher) comes in.
`nix-pypi-fetcher` is another database, which allows us to resolve packages to their URL and hash.
For example, the declaration "package pillow + version 9.1.0 + python 3.9 + linux" resolves to
```json
"Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": [
"gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=",
"cp39"
],
```
The wheel URL is constructed in [pypi-crawlers/src/crawl_wheel_deps.py](pypi-crawlers/src/crawl_wheel_deps.py)
```py
def construct_url(name, pyver, filename: str):
base_url = "https://files.pythonhosted.org/packages/"
return f"{base_url}{pyver}/{name[0]}/{name}/{filename}"
```
So in this example, the full URL would be
```
https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
```
Now we can call Nix's `fetchurl` like

```nix
builtins.fetchurl {
url = "https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=";
}
```

... and we get

```
$ unzip -l /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | head
Archive: /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Length Date Time Name
--------- ---------- ----- ----
0 04-01-2022 10:08 PIL/
0 04-01-2022 10:08 Pillow.libs/
0 04-01-2022 10:08 Pillow-9.1.0.dist-info/
7311 04-01-2022 10:08 PIL/PdfImagePlugin.py
141328 04-01-2022 10:08 PIL/_imagingcms.cpython-39-x86_64-linux-gnu.so
46872 04-01-2022 10:08 PIL/_imagingtk.cpython-39-x86_64-linux-gnu.so
1513 04-01-2022 10:08 PIL/GribStubImagePlugin.py
```

### Generating a nix expression
After all python dependencies and their providers have been determined by the dependency resolver, mach-nix will generate a nix expression defining your python environment.

Expand Down

0 comments on commit 0901d64

Please sign in to comment.