'error: externally-managed-environment' when building source distribution

[srwi]

Recently I started getting the following error when attempting to build the source distribution:

2024-10-03T20:32:27.7458984Z ##[group]Run PyO3/maturin-action@v1
2024-10-03T20:32:27.7459359Z with:
2024-10-03T20:32:27.7459578Z   command: sdist
2024-10-03T20:32:27.7460071Z   args: --out dist
2024-10-03T20:32:27.7460503Z   token: ***
2024-10-03T20:32:27.7460732Z   sccache: false
2024-10-03T20:32:27.7460961Z ##[endgroup]
2024-10-03T20:32:27.7995307Z Found maturin version requirement maturin>=1.0,<2.0 specified in pyproject.toml
2024-10-03T20:32:28.1948506Z Found maturin release from manifest: v1.7.4
2024-10-03T20:32:28.1959114Z ##[group]Install Rust target
2024-10-03T20:32:28.1961023Z ##[endgroup]
2024-10-03T20:32:28.1962771Z ##[group]Install maturin
2024-10-03T20:32:28.1963290Z Installing 'maturin' from tag 'v1.7.4'
2024-10-03T20:32:28.5706980Z [command]/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/377b9b2b-e09d-4e78-822e-fc1408b20297 -f /home/runner/work/_temp/d8eff9e8-5f16-41a3-9511-43461629a92a
2024-10-03T20:32:28.7430228Z Installed 'maturin' to /home/runner/work/_temp/377b9b2b-e09d-4e78-822e-fc1408b20297/maturin
2024-10-03T20:32:28.7433023Z [command]/home/runner/work/_temp/377b9b2b-e09d-4e78-822e-fc1408b20297/maturin --version
2024-10-03T20:32:28.7474469Z maturin 1.7.4
2024-10-03T20:32:28.7496213Z [command]/usr/bin/python3 -m pip install cffi
2024-10-03T20:32:31.9410356Z error: externally-managed-environment
2024-10-03T20:32:31.9411006Z 
2024-10-03T20:32:31.9411972Z × This environment is externally managed
2024-10-03T20:32:31.9412862Z ╰─> To install Python packages system-wide, try apt install
2024-10-03T20:32:31.9413591Z     python3-xyz, where xyz is the package you are trying to
2024-10-03T20:32:31.9414092Z     install.
2024-10-03T20:32:31.9414353Z     
2024-10-03T20:32:31.9414823Z     If you wish to install a non-Debian-packaged Python package,
2024-10-03T20:32:31.9415608Z     create a virtual environment using python3 -m venv path/to/venv.
2024-10-03T20:32:31.9416344Z     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
2024-10-03T20:32:31.9417034Z     sure you have python3-full installed.
2024-10-03T20:32:31.9417438Z     
2024-10-03T20:32:31.9417924Z     If you wish to install a non-Debian packaged Python application,
2024-10-03T20:32:31.9418635Z     it may be easiest to use pipx install xyz, which will manage a
2024-10-03T20:32:31.9419362Z     virtual environment for you. Make sure you have pipx installed.
2024-10-03T20:32:31.9420122Z     
2024-10-03T20:32:31.9420587Z     See /usr/share/doc/python3.12/README.venv for more information.
2024-10-03T20:32:31.9421012Z 
2024-10-03T20:32:31.9422260Z note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
2024-10-03T20:32:31.9423620Z hint: See PEP 668 for the detailed specification.
2024-10-03T20:32:31.9902125Z [command]/usr/bin/python3 -m pip install patchelf
2024-10-03T20:32:32.2784126Z error: externally-managed-environment
2024-10-03T20:32:32.2786290Z 
2024-10-03T20:32:32.2786803Z × This environment is externally managed
2024-10-03T20:32:32.2787645Z ╰─> To install Python packages system-wide, try apt install
2024-10-03T20:32:32.2788603Z     python3-xyz, where xyz is the package you are trying to
2024-10-03T20:32:32.2789260Z     install.
2024-10-03T20:32:32.2789582Z     
2024-10-03T20:32:32.2790476Z     If you wish to install a non-Debian-packaged Python package,
2024-10-03T20:32:32.2791436Z     create a virtual environment using python3 -m venv path/to/venv.
2024-10-03T20:32:32.2792314Z     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
2024-10-03T20:32:32.2793083Z     sure you have python3-full installed.
2024-10-03T20:32:32.2793552Z     
2024-10-03T20:32:32.2794120Z     If you wish to install a non-Debian packaged Python application,
2024-10-03T20:32:32.2794948Z     it may be easiest to use pipx install xyz, which will manage a
2024-10-03T20:32:32.2795786Z     virtual environment for you. Make sure you have pipx installed.
2024-10-03T20:32:32.2796966Z     
2024-10-03T20:32:32.2797488Z     See /usr/share/doc/python3.12/README.venv for more information.
2024-10-03T20:32:32.2798029Z 
2024-10-03T20:32:32.2799695Z note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
2024-10-03T20:32:32.2801727Z hint: See PEP 668 for the detailed specification.
2024-10-03T20:32:32.3377043Z Error: The process '/usr/bin/python3' failed with exit code 1
2024-10-03T20:32:32.3378217Z     at ExecState._setResult (/home/runner/work/_actions/PyO3/maturin-action/v1/dist/index.js:1702:25)
2024-10-03T20:32:32.3379192Z     at ExecState.CheckComplete (/home/runner/work/_actions/PyO3/maturin-action/v1/dist/index.js:1685:18)
2024-10-03T20:32:32.3380428Z     at ChildProcess.<anonymous> (/home/runner/work/_actions/PyO3/maturin-action/v1/dist/index.js:1579:27)
2024-10-03T20:32:32.3381098Z     at ChildProcess.emit (node:events:519:28)
2024-10-03T20:32:32.3381539Z     at maybeClose (node:internal/child_process:1105:16)
2024-10-03T20:32:32.3382059Z     at ChildProcess._handle.onexit (node:internal/child_process:305:5)
2024-10-03T20:32:32.3409885Z ##[error]The process '/usr/bin/python3' failed with exit code 1

It looks like the cause is the update of the ubuntu-latest runner image to ubuntu-24.04 which now ships with python 3.12. One possible fix would be to add --break-system-packages when installing the packages, though creating a virtual environment would probably be the cleaner solution.

In the meantime the problem can be worked around by using the ubuntu-22.04 image explicitly.

Sources:

• https://github.com/actions/runner-images?tab=readme-ov-file#available-images
• 'This environment is externally managed' python error actions/runner-images#8615

[dbarnett]

Can someone explain how this worked in 22.04?

I do see in my previous passing runs there was a message Defaulting to user installation because normal site-packages is not writeable. But I don't understand how the exact installed python version factors into that logic...

[konstin]

As I understand it, github actions shipped a change that the default Python is now a system Python, hence not allowing installation into its environment anymore. Does using @actions/setup-python with an explicit Python version work for you? Otherwise you need to create a venv (e.g. virtualenv .venv or python -m venv venv). --break-system-packages is not an option, since we specifically don't want to corrupt the system wide /usr/bin/python3 that is also used by the operating system.

[dbarnett]

As I understand it, github actions shipped a change that the default Python is now a system Python

So it's not purely the ubuntu version and which python version it ships with? The breakage definitely tracks with the ubuntu-24.04 image and can indeed be worked around by downgrading from -latest to -22.04.

[konstin]

They seem to have made that change with the ubuntu-24.04 image, this breaking python change is apparently also the reason the rollout is now on hold: actions/runner-images#10636

[danking]

We encountered this on a macos-latest runner today. Action logs here. We definitely have the latest version of Maturin-action:

Download action repository 'PyO3/maturin-action@v1' (SHA:ea5bac0f1ccd0ab11c805e2b804bfcb65dac2eab)

We're definitely not using the --user flag:

Install maturin
  Installing 'maturin' from tag 'v1.7.4'
  /usr/bin/tar xz -C /Users/runner/work/_temp/cc8a46b9-5ec9-44bd-82c0-107ab367408a -f /Users/runner/work/_temp/d785f0a3-c0bf-4f99-b894-0a2aa5d56550
  Installed 'maturin' to /Users/runner/work/_temp/cc8a46b9-5ec9-44bd-82c0-107ab367408a/maturin
  /Users/runner/work/_temp/cc8a46b9-5ec9-44bd-82c0-107ab367408a/maturin --version
  maturin 1.7.4
  /opt/homebrew/bin/python3 -m pip install cffi
  
  Notice:  A new release of pip is available: 24.2 -> 24.3.1
  Notice:  To update, run: python3.13 -m pip install --upgrade pip
  error: externally-managed-environment
  
  × This environment is externally managed

AFAICT, Maturin Action only uses --user for docker builds. Maybe it should also use --user for host builds? I don't have any other explanation for why we would encounter this error. This is what the GHA job looks like:

  macos:
    needs: release-plz-release
    if: ${{ needs.release-plz-release.outputs.releases_created == 'true' }}
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/cleanup
      - uses: ./.github/actions/setup-rust
      - uses: spiraldb/actions/.github/actions/setup-uv@0.2.0
      - name: rust-toolchain
        shell: bash
        run: echo "version=$(cat rust-toolchain.toml | grep channel | awk -F'\"' '{print $2}')" >> $GITHUB_OUTPUT
      - name: Build wheels - x86_64
        uses: PyO3/maturin-action@v1
        with:
          rust-toolchain: ${{ steps.rust-toolchain.version }}
          working-directory: pyvortex
          target: x86_64
          args: --release --interpreter python3.10 --sdist

maturin-action/src/index.ts

Line 881 in 78d2b0f

await exec.exec('python3', ['-m', 'pip', 'install', 'cffi'])