Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

sdb9696 · 2024-04-15T10:48:32Z

Description:

The recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install.

Because pipx install poetry happens before setup-python and By default, Poetry will try to use the Python version used during Poetry’s installation to create the virtual environment for the current project.

Maybe the solution here is to update the docs to make sure that users know to run poetry env use ${{ steps.setup-python.outputs.python-path }}python prior to running poetry install. I have not tested that works however.

Action version:

v5

Platform:

Ubuntu
macOS
Windows

Runner type:

Hosted
Self-hosted

Tools version:

All versions of python affected except the default version installed on the github runner.

Repro steps:

Follow the steps outlined in advance-usage.md for caching poetry dependencies.

Run poetry install

Expected behavior:

poetry install should use the version of python set by setup-python

Actual behavior:

poetry install uses the default version of python from the runner os.

The text was updated successfully, but these errors were encountered:

HarithaVattikuti · 2024-04-15T22:04:24Z

Hello @sdb9696
Thank you for creating this issue. We will investigate it and get back to you as soon as we have some feedback.

andwaredev · 2024-04-18T17:37:37Z

Hello @sdb9696 – my team ran into this same issue recently. We solved it by pinning the python version in our pyproject.toml file, under [tool.poetry.dependencies]. See example below.

[tool.poetry.dependencies]
python = "3.11.4"

sdb9696 · 2024-04-18T17:54:41Z

Thanks @andwaredev but we run our CI against all supported python versions to check there are no issues so not sure this workaround will help.

silverwind · 2024-04-19T20:20:44Z

I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before setup-python. It can not be assumed that python is present before setup-python has run because some runner images will not have python pre-installed.

This is what I would like to see working:

- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- run: pipx install poetry

evilhamsterman · 2024-04-24T18:37:32Z

It's always struck me as a bad chicken/egg problem that poetry is the only cache type that requires the environment manager to be installed before it can be used. Even pipenv is not required to be installed

setup-python/src/cache-distributions/pipenv-cache.ts

Line 20 in 82c7e63

    
           // because pipenv is not preinstalled on hosted images and virtualenv is not created:

it just uses the default pipenv location.

The Poetry default cache location is well known and I bet in CI the vase majority of projects are using that case. The big difference appears to be that the Poetry option is trying to support the fact that Poetry can install virtual environments in the project. Pipenv can also do this but it's just ignored for that option. I'd recommend just caching the default location like Pipenv, or you could have an option for enabling caching in project virtual environments it people use it.

evilhamsterman · 2024-04-24T18:57:44Z

@silverwind You can tell poetry to use a specific version of python prior to running install poetry env use <python executable> then run poetry install

https://python-poetry.org/docs/managing-environments#switching-between-environments

silverwind · 2024-04-24T19:00:27Z

@evilhamsterman doesn't really help me if there is simply no python installed before setup-python ran. Of course one could do workarounds by running setup-python twice, but I rather skip on enabling cache than doing such hacks.

adminy · 2024-05-02T14:52:34Z

I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before setup-python. It can not be assumed that python is present before setup-python has run because some runner images will not have python pre-installed.

This is what I would like to see working:
- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- run: pipx install poetry

This also makes the assumtion that you have pipx which I do not :)

I'd highly recommend to install poetry as part of this python version fetching if cache is set to poetry.

Chances are everyone is going to need poetry if they are using poetry cache, so maybe get the specific python version with poetry pre-installed. This seems to be the only viable solution.

silverwind · 2024-05-02T15:08:32Z

This also makes the assumtion that you have pipx which I do not :)

I think setup-python does install pip and its variants currently, right?

silverwind · 2024-05-02T15:13:09Z

This also makes the assumtion that you have pipx which I do not :)

I think setup-python does install pip and its variants currently, right?

Apparently, it doesn't so to install poetry without pip and having a poetry cache, this would be the ideal:

- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- uses: snok/install-poetry@v1

LaurensBosscher · 2024-05-17T10:31:20Z

This is indeed inconvenient. We install Python and Poetry in the following way:

      - name: Install Poetry
        run: |
          python -m pip install poetry
          
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: 3.12
          cache: poetry
          cache-dependency-path: projects/investment_decision_support_system/poetry.lock
          
      - name: Set Poetry environment
        run: |
          python -m pip install poetry
          cd projects/investment_decision_support_system/
          poetry env use 3.12

This triggered the following error:

  - Installing xmltodict (0.13.0 9c3ec3c)
  ChefBuildError
  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
      main()
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 128, in get_requires_for_build_wheel
      backend = _build_backend()
                ^^^^^^^^^^^^^^^^
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 70, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
      import setuptools.version
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
      import pkg_resources
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module>
      register_finder(pkgutil.ImpImporter, find_on_path)
                      ^^^^^^^^^^^^^^^^^^^
  AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
  
  at /opt/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Do note how the installation is failing because Poetry is using Python 3.10 while the specified Python version is 3.12.

This took a while to chase down. I'm not sure what exactly depends on Poetry but perhaps it would be possible to integrate those functions into setup-python instead of having a hard dependency on the whole package?

Alternatively, if there's a workaround it would be great if the documentation could be updated.

evilhamsterman · 2024-05-17T17:00:10Z

Ultimately we're considering just using actions/cache directly to cache poetry rather than setup-python's built in support

till · 2024-05-21T16:31:08Z

I just ran into this issue as well. I used an example in the docs that triggers this bug beautifully. Maybe time to ramp up the tests for this action?

till · 2024-05-21T17:21:36Z

I am running the following workflow which works:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - run: pipx install poetry==1.8.3
    - run: echo "/root/.local/bin" >> $GITHUB_PATH
    - uses: actions/setup-python@v5
      with:
        python-version: '3.11'
        cache: poetry
    - run: pipx install poetry==1.8.3
    - run: poetry install
      working-directory: app
    - run: poetry run flake8
      working-directory: app

pipx is smart enough not to install poetry again, and somehow it ends up in the correct environment without any hacks.

What I had to do was add /root/.local/bin to the $GITHUB_PATH, because otherwise it wouldn't be able to find poetry anywhere and fail at actions/setup-python:

[pr/lint]   💬  ::debug::Found 1 files to hash.
[pr/lint]   💬  ::debug::followSymbolicLinks 'true'
[pr/lint]   💬  ::debug::implicitDescendants 'true'
[pr/lint]   💬  ::debug::matchDirectories 'true'
[pr/lint]   💬  ::debug::omitBrokenSymbolicLinks 'true'
[pr/lint]   💬  ::debug::Search path '/Users/till/Documents/workspaces/project/pdf-test'
[pr/lint]   💬  ::debug::Processing Poetry project at /Users/till/Documents/workspaces/project/pdf-test/app
[pr/lint]   ❗  ::error::Unable to locate executable file: poetry. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
[pr/lint]   ❌  Failure - Main actions/setup-python@v5
[pr/lint]   ⚙  ::set-env:: Python_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: Python2_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: Python3_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: LD_LIBRARY_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib
[pr/lint]   ⚙  ::set-env:: pythonLocation=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: PKG_CONFIG_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
[pr/lint]   ⚙  ::set-output:: python-version=3.11.8
[pr/lint]   ⚙  ::set-output:: python-path=/opt/hostedtoolcache/Python/3.11.8/x64/bin/python
[pr/lint]   ⚙  ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64/bin
[pr/lint] exitcode '1': failure

gowridurgad · 2024-06-03T06:06:12Z

Hi @sdb9696 , It is the expected behaviour, if the setup python tries to install the python version which was not specified in pyproject.toml file, under [tool.poetry.dependencies], the poetry install use the python version used during poetry installation from runner. Here's the screenshots for your reference.

[tool.poetry.dependencies]
python = "^3.10.9"
numpy = "^1.22.3"
tqdm = {version = "4.66.4", optional = true }
pandas = "^2.2.2"

gowridurgad · 2024-06-18T12:10:16Z

Hi @sdb9696, Just a gentle reminder regarding this issue, We have updated the document. If you have any questions or need further assistance, Please let us know.

sdb9696 · 2024-06-18T12:59:38Z

Hi @sdb9696, Just a gentle reminder regarding this issue, We have updated the document. If you have any questions or need further assistance, Please let us know.

Hi @gowridurgad, apologies for the delayed reply I have been away.

I'm not completely sure I follow the expected behaviour point or the point about matching the python version in pyproject.toml.

Your example above is doing poetry env use "${{ steps.setup-python.outputs.python-path }}". Is that something you added specifically to your test or has it been added to the setup-python codebase since I opened the original issue? My suggestion was to update the docs to include that suggestion and I can see from above that it will work if that's done.

w.r.t the setup-python interpreter not matching the toml spec that is not the issue I was reporting. Our toml specifies python = ">=3.8.0" so this isn't the issue I was reporting. The issue is that poetry will not use the setup-python python version unless the user runs poetry env use "${{ steps.setup-python.outputs.python-path }}" before running install.

gowridurgad · 2024-06-24T12:31:13Z

Hi @sdb9696, we haven't use the command poetry env use "${{ steps.setup-python.outputs.python-path }}" in workflow file. Poetry seems to be automatically using the Python version set by setup-python, regardless of whether the command is used. However, The Poetry fails to use the setup-python Python version only when there's a mismatch between this version and the one specified in the pyproject.toml file. In such cases, using the poetry env use "${{steps.setup-python.outputs.python-path }}" actually causes the build to fail. here's the screenshots for your reference.
screenshot1: with poetry env use "${{ steps.setup-python.outputs.python-path }}" .

screenshot2: without poetry env use "${{ steps.setup-python.outputs.python-path }}"

screenshot3: poetry env use "${{ steps.setup-python.outputs.python-path }}" causes the build to fail if python version didn't
match version specified in pyproject.toml file.

andreipetre · 2024-06-28T11:37:39Z

I ran into the same issue. The existing workflow stopped working recently, without any changes to it.

I fixed it by changing how poetry is installed.

steps:
  - uses: actions/checkout@v4

  - name: Install poetry
    run: curl -sSL https://install.python-poetry.org | python3 -

  - uses: actions/setup-python@v5
    with:
      python-version: "3.12.3"
      cache: "poetry"

  - name: Install dependencies
    run: poetry install

gowridurgad · 2024-07-03T11:35:58Z

Hi @andreipetre, Could you please attach link to the build or public repository to reproduce the issue ?
If you are referring to the workflow mentioned in the document, It is working correctly. The workflow outlined in the document and the one you mentioned in your previous comment are functioning identically.

sdb9696 · 2024-07-04T14:00:55Z

@gowridurgad maybe you're missing the issue because you're fixing the python version in your pyproject.toml. Have you tried this with a loose python specifier of >=3.8? Then poetry will use whichever version it was pipx installed with and not the version installed by setup_python

gowridurgad · 2024-07-09T05:39:48Z

Hi @sdb9696, We tried using a loose Python specifier of >=3.8 and Poetry is using the version installed by setup_python. Here is a screenshot for reference.

can you please share your updated workflow link to check and assist further if anything missed as we are not seeing the issue from our end.

[tool.poetry.dependencies]
python = ">=3.8"
requests = "^2.25.1

sdb9696 · 2024-07-09T09:25:43Z

Hi @gowridurgad, I don't have a workflow link to point you to as I ended up using actions/cache directly in the end. Can you point me to the PR/branch you've used to generate the output above and I'll see what the difference is?

gowridurgad · 2024-07-12T05:42:09Z

Hi @sdb9696, Here is the link to the repository
https://github.com/setup-actions-demo/test-setup-python/actions/runs/9868764478/job/27251296773
Please take a look and let us know if you find any differences or have any questions.

gowridurgad · 2024-07-24T13:00:00Z

Hi @sdb9696, Just a gentle reminder regarding this issue, If you have any updates, Please let us know.

gowridurgad · 2024-08-05T12:49:24Z

Hi @sdb9696, Just a gentle reminder regarding this issue, If you have any updates, Please let us know.

sdb9696 · 2024-08-05T13:35:23Z

Hi, apologies for the delay. I tried to recreate the issue and couldn’t anymore. It’s possible something has fixed it so I’d say good to close with the doc updates. Many thanks for your time with this.

gowridurgad · 2024-08-09T06:03:54Z

Hello @sdb9696 , The PR has been merged and the documentation has been updated. For reference, you may visit https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages. I am proceeding with closing the issue. Please feel free to contact us in case of any further concerns.
Thank You !

sdb9696 added bug Something isn't working needs triage labels Apr 15, 2024

HarithaVattikuti removed the needs triage label Apr 15, 2024

silverwind mentioned this issue Apr 19, 2024

Enable poetry cache in setup-python go-gitea/gitea#30607

Closed

gowridurgad self-assigned this May 3, 2024

gowridurgad mentioned this issue May 16, 2024

Updated recommendations for caching poetry dependencies #869

Closed

2 tasks

gowridurgad mentioned this issue Jun 4, 2024

Documentation update for cache #873

Merged

2 tasks

MrLixm mentioned this issue Jun 25, 2024

poetry install on GitHub actions fails with python version mismatch python-poetry/poetry#9515

Open

gowridurgad mentioned this issue Jul 9, 2024

Documentation update for caching poetry dependencies. #907

Closed

2 tasks

gowridurgad mentioned this issue Jul 9, 2024

Documentation update for caching poetry dependencies #908

Merged

2 tasks

gowridurgad closed this as completed Aug 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

sdb9696 commented Apr 15, 2024

HarithaVattikuti commented Apr 15, 2024

andwaredev commented Apr 18, 2024

sdb9696 commented Apr 18, 2024

silverwind commented Apr 19, 2024 •

edited

Loading

evilhamsterman commented Apr 24, 2024

evilhamsterman commented Apr 24, 2024 •

edited

Loading

silverwind commented Apr 24, 2024 •

edited

Loading

adminy commented May 2, 2024

silverwind commented May 2, 2024

silverwind commented May 2, 2024

LaurensBosscher commented May 17, 2024 •

edited

Loading

evilhamsterman commented May 17, 2024

till commented May 21, 2024

till commented May 21, 2024

gowridurgad commented Jun 3, 2024

gowridurgad commented Jun 18, 2024

sdb9696 commented Jun 18, 2024

gowridurgad commented Jun 24, 2024

andreipetre commented Jun 28, 2024

gowridurgad commented Jul 3, 2024

sdb9696 commented Jul 4, 2024

gowridurgad commented Jul 9, 2024

sdb9696 commented Jul 9, 2024

gowridurgad commented Jul 12, 2024

gowridurgad commented Jul 24, 2024

gowridurgad commented Aug 5, 2024

sdb9696 commented Aug 5, 2024

gowridurgad commented Aug 9, 2024 •

edited

Loading

Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

Comments

sdb9696 commented Apr 15, 2024

HarithaVattikuti commented Apr 15, 2024

andwaredev commented Apr 18, 2024

sdb9696 commented Apr 18, 2024

silverwind commented Apr 19, 2024 • edited Loading

evilhamsterman commented Apr 24, 2024

evilhamsterman commented Apr 24, 2024 • edited Loading

silverwind commented Apr 24, 2024 • edited Loading

adminy commented May 2, 2024

silverwind commented May 2, 2024

silverwind commented May 2, 2024

LaurensBosscher commented May 17, 2024 • edited Loading

evilhamsterman commented May 17, 2024

till commented May 21, 2024

till commented May 21, 2024

gowridurgad commented Jun 3, 2024

gowridurgad commented Jun 18, 2024

sdb9696 commented Jun 18, 2024

gowridurgad commented Jun 24, 2024

andreipetre commented Jun 28, 2024

gowridurgad commented Jul 3, 2024

sdb9696 commented Jul 4, 2024

gowridurgad commented Jul 9, 2024

sdb9696 commented Jul 9, 2024

gowridurgad commented Jul 12, 2024

gowridurgad commented Jul 24, 2024

gowridurgad commented Aug 5, 2024

sdb9696 commented Aug 5, 2024

gowridurgad commented Aug 9, 2024 • edited Loading

silverwind commented Apr 19, 2024 •

edited

Loading

evilhamsterman commented Apr 24, 2024 •

edited

Loading

silverwind commented Apr 24, 2024 •

edited

Loading

LaurensBosscher commented May 17, 2024 •

edited

Loading

gowridurgad commented Aug 9, 2024 •

edited

Loading