How are packages being built for the apk repository?

[mil1i]

So I would like a better understanding of how the packages are being built for the apk repository that chainguard.dev is serving.

https://github.com/orgs/wolfi-dev/repositories
I can see the yaml files here with their definitions, but the reason I ask this is that from testing on the wolfi-base image and installing several packages they don't appear to work properly or if they're a binary that reports the GitTree State they report as "Dirty".

For instance:

Poetry doesn't seem to work correctly.

$ apk add python-3.12 poetry

$ poetry -V
Poetry (version 1.8.3)
$ poetry list

No module named 'poetry_plugin_export'

If I uninstall the apk installation method, and install using the official installer it works as expected.

$ curl -sSL https://install.python-poetry.org | python3 -
...
Installing Poetry (1.8.3): Done
....

$poetry -V
Poetry (version 1.8.3)

$ poetry list
Poetry (version 1.8.3)

Usage:
  command [options] [arguments]

Options:
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.

Helm / ArgoCD report a GitTreeState of "Dirty" indicating that something was modified before the build.

$ apk add helm argo-cd
(1/2) Installing argo-cd-2.11 (2.11.0-r0)
(2/2) Installing helm (3.14.4-r3)
OK: 271 MiB in 79 packages

$ argocd version
argocd: v2.11.0+d3f33c0.dirty
  BuildDate: 2024-05-13T21:03:09Z
  GitCommit: d3f33c00197e7f1d16f2a73ce1aeced464b07175
  GitTreeState: dirty
  GoVersion: go1.22.3
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified

$ helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"dirty", GoVersion:"go1.22.3"}

[amouat]

Apologies for the slow reply.

Full build logs are available under the actions tab, so you can see exactly what happens e.g. here is a recent caddy build. The core build is just running melange on the YAML files.

We can open an issue for poetry, as that doesn't look right.

Regarding the "dirty" git status, I believe there is a good reason for that -- I'll come back with an answer shortly.

[amouat]

As a workaround, poetry self add poetry-plugin-export should help.

[amouat]

As @xnox writes below, the dirty state is caused by the way we do git checkout in the melange pipeline. We do have an action to improve this. Note that you will always get a dirty state if we have to apply a patch or bump a mod version, which is relatively common.

You can see the melange git checkout pipeline here: https://github.com/chainguard-dev/melange/blob/main/pkg/build/pipelines/git-checkout.yaml

[xnox]

During build git-checkout or fetch of tarball is done in builders $HOME.
It may have some settings overlayed - i.e. setting safe directory in ~/.gitconfig.
Then we may apply patches (see yaml file) which would cause tree to be dirty.
We may upgrade golang dependencies with go/bump which changes go.mod and go.sum files, which would case tree to be dirty as well.

[NeilJed]

I'm also having some problems with poetry and wanted to add some additional context.

I'm building a base image for python 3.10, 3.11 and 3.12 all with the same packages and just mutating the versions in the package list:

packages:
  - wolfi-base
  - build-base
  - python-<VERSION>
  - py<VERSION>-pip
  - poetry

In the case of the Python 3.10 and 3.11 images, poetry can't be run giving the error:

Traceback (most recent call last):
  File "/usr/bin/poetry", line 5, in <module>
    from poetry.console.application import main
ModuleNotFoundError: No module named 'poetry'

On 3.12 I get the error:

No module named 'poetry_plugin_export'

Having looked at the build output I can see that while I am specifying for example Python 3.10, the poetry dependencies align with 3.12:

2024/07/31 19:31:40 INFO installing py3.12-tomli (2.0.1-r9) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.12-installer (0.7.0-r7) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.12-pyparsing (3.1.2-r4) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.12-packaging (24.1-r2) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.12-build (1.2.1-r4) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.12-trove-classifiers (2024.7.2-r4) arch=aarch64
2024/07/31 19:31:40 INFO installing py3-poetry (1.8.3-r1) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.10-setuptools (72.1.0-r0) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.10-pip-base (24.2-r0) arch=aarch64
2024/07/31 19:31:40 INFO installing py3.10-pip (24.2-r0) arch=aarch64

The upshot is that these packages end up in /usr/lib/python3.12/site-packages. This is the cause of the first error as Python 3.10 is looking for packages in /usr/lib/python3.10/site-packages.

As @mil1i states, the only way to get around this is to install poetry using pip or the official installer.

So it appears that poetry doesn't take the actual system python into account and is also missing the poetry_plugin_export dependency.

[xnox]

It is correct that py3-poetry-core.yaml py3-poetry.yaml packages currently only build for a single latest version of python. And builds are not currently provided for all supported versions of python. Thus indeed using py3.*-pip and installing poetry with a matching pip is the currently available workaround.