Installing packages in a python 3.6 environment fail due to PEP 517

[bearprevent]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 22.0.1

• Poetry version: 1.2.0

• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

A dependency uses MarkupSafe and our project must support python ^3.6.2. Since I updated to poetry 1.2.0, I can not use the python 3.6 environment anymore.

Installing MarkupSafe manually works, when I disable pep 517:

> poetry run python -m pip install --no-use-pep517 markupSafe==2.0.1

However, there is no option to disable pep 517 in poetry.
Is there any other way to install such packages?

Errors

> poetry env use 3.6
> poetry -vvv install
Using virtualenv: /tmp/foo/.venv
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals

  • Installing markupsafe (2.0.1): Pending...
Creating new session for pypi.org
  • Installing markupsafe (2.0.1): Installing...
  • Installing markupsafe (2.0.1): Failed

  Stack trace:

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1469 in _run
      1467│                 )
      1468│             else:
    → 1469│                 output = subprocess.check_output(
      1470│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1471│                 )

  1  /usr/lib/python3.10/subprocess.py:420 in check_output
       418│         kwargs['input'] = empty
       419│ 
    →  420│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       421│                **kwargs).stdout
       422│ 

  CalledProcessError

  Command '['/tmp/foo/.venv/bin/python', '/home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/tmp/foo/.venv', '--no-deps', '/home/local/.cache/pypoetry/artifacts/09/a9/a6/7e309619221c03e9a693897779a3fa9b06b4c71edb005b6919654585d3/MarkupSafe-2.0.1.tar.gz']' returned non-zero exit status 1.

  at /usr/lib/python3.10/subprocess.py:524 in run
       520│             # We don't call process.wait() as .__exit__ does that for us.
       521│             raise
       522│         retcode = process.poll()
       523│         if check and retcode:
    →  524│             raise CalledProcessError(retcode, process.args,
       525│                                      output=stdout, stderr=stderr)
       526│     return CompletedProcess(process.args, retcode, stdout, stderr)
       527│ 
       528│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:49 in pip_install
       47│ 
       48│     try:
    →  49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
       51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1433 in run_pip
      1431│         cmd = pip + list(args)
      1432│ 
    → 1433│         return self._run(cmd, **kwargs)
      1434│ 
      1435│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1722 in _run
      1720│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1721│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1722│         return super()._run(cmd, **kwargs)
      1723│ 
      1724│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/foo/.venv/bin/python', '/home/local/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/tmp/foo/.venv', '--no-deps', '/home/local/.cache/pypoetry/artifacts/09/a9/a6/7e309619221c03e9a693897779a3fa9b06b4c71edb005b6919654585d3/MarkupSafe-2.0.1.tar.gz'] errored with the following return code 1, and output: 
  Looking in indexes: https://pypi.org/simple, 
  Processing /home/local/.cache/pypoetry/artifacts/09/a9/a6/7e309619221c03e9a693897779a3fa9b06b4c71edb005b6919654585d3/MarkupSafe-2.0.1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /tmp/foo/.venv/bin/python /home/local/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkvyouqoq
         cwd: /tmp/pip-req-build-80e95xhl
    Complete output (1 lines):
    /tmp/foo/.venv/bin/python: can't find '__main__' module in '/home/local/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py'
    ----------------------------------------
  WARNING: Discarding file:///home/local/.cache/pypoetry/artifacts/09/a9/a6/7e309619221c03e9a693897779a3fa9b06b4c71edb005b6919654585d3/MarkupSafe-2.0.1.tar.gz. Command errored out with exit status 1: /tmp/foo/.venv/bin/python /home/local/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkvyouqoq Check the logs for full command output.
  ERROR: Command errored out with exit status 1: /tmp/foo/.venv/bin/python /home/local/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkvyouqoq Check the logs for full command output.
  

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1473 in _run
      1469│                 output = subprocess.check_output(
      1470│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1471│                 )
      1472│         except CalledProcessError as e:
    → 1473│             raise EnvCommandError(e, input=input_)
      1474│ 
      1475│         return decode(output)
      1476│ 
      1477│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:250 in _execute_operation
      248│ 
      249│             try:
    → 250│                 result = self._do_execute_operation(operation)
      251│             except EnvCommandError as e:
      252│                 if e.e.returncode == -2:

  4  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:323 in _do_execute_operation
      321│             return 0
      322│ 
    → 323│         result: int = getattr(self, f"_execute_{method}")(operation)
      324│ 
      325│         if result != 0:

  3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:443 in _execute_install
      441│ 
      442│     def _execute_install(self, operation: Install | Update) -> int:
    → 443│         status_code = self._install(operation)
      444│ 
      445│         self._save_url_reference(operation)

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:485 in _install
      483│         )
      484│         self._write(operation, message)
    → 485│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      486│ 
      487│     def _update(self, operation: Install | Update) -> int:

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:120 in pip_install
      118│     ) -> int:
      119│         try:
    → 120│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      121│         except EnvCommandError as e:
      122│             output = decode(e.e.output)

  PoetryException

  Failed to install /home/local/.cache/pypoetry/artifacts/09/a9/a6/7e309619221c03e9a693897779a3fa9b06b4c71edb005b6919654585d3/MarkupSafe-2.0.1.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

[tomwatson1024]

Ah, you beat me to it, I was just about to raise this issue too! I've got a Docker-based repro:

# syntax=docker/dockerfile:1.3-labs
FROM ubuntu:18.04
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
    apt-get install -y python3.6 python3.8 python3-distutils python3.6-venv curl
RUN  curl -sSL https://install.python-poetry.org | python3.8 -

RUN mkdir /src
COPY <<"EOF" /src/pyproject.toml
[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.6"
untokenize = "^0.1.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
EOF

RUN export PATH="/root/.local/bin:$PATH" && \
    cd /src && \
    python3.6 -m venv .venv && \
    . .venv/bin/activate && \
    poetry install

producing output:

#14 6.408 Package operations: 1 install, 0 updates, 0 removals
#14 6.408
#14 6.409   • Installing untokenize (0.1.1)
#14 10.83
#14 10.83   CalledProcessError
#14 10.83
#14 10.83   Command '['/src/.venv/bin/python', '/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/src/.venv', '--no-deps', '/root/.cache/pypoetry/artifacts/d0/bb/16/9d113f610914e00bfed3d34597785b6f845d3760f26de934e729821f50/untokenize-0.1.1.tar.gz']' returned non-zero exit status 1.
#14 10.83
#14 10.83   at /usr/lib/python3.8/subprocess.py:512 in run
#14 10.91        508│             # We don't call process.wait() as .__exit__ does that for us.
#14 10.91        509│             raise
#14 10.91        510│         retcode = process.poll()
#14 10.91        511│         if check and retcode:
#14 10.91     →  512│             raise CalledProcessError(retcode, process.args,
#14 10.91        513│                                      output=stdout, stderr=stderr)
#14 10.91        514│     return CompletedProcess(process.args, retcode, stdout, stderr)
#14 10.91        515│
#14 10.91        516│
#14 10.91
#14 10.91 The following error occurred when trying to handle this error:
#14 10.91
#14 10.91
#14 10.91   EnvCommandError
#14 10.91
#14 10.91   Command ['/src/.venv/bin/python', '/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/src/.venv', '--no-deps', '/root/.cache/pypoetry/artifacts/d0/bb/16/9d113f610914e00bfed3d34597785b6f845d3760f26de934e729821f50/untokenize-0.1.1.tar.gz'] errored with the following return code 1, and output:
#14 10.91   Processing /root/.cache/pypoetry/artifacts/d0/bb/16/9d113f610914e00bfed3d34597785b6f845d3760f26de934e729821f50/untokenize-0.1.1.tar.gz
#14 10.91     Installing build dependencies: started
#14 10.91     Installing build dependencies: finished with status 'done'
#14 10.91     Getting requirements to build wheel: started
#14 10.91     Getting requirements to build wheel: finished with status 'error'
#14 10.91     ERROR: Command errored out with exit status 1:
#14 10.91      command: /src/.venv/bin/python /root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp6y23q36c
#14 10.91          cwd: /tmp/pip-req-build-c_prxdlr
#14 10.91     Complete output (1 lines):
#14 10.91     /src/.venv/bin/python: can't find '__main__' module in '/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py'
#14 10.91     ----------------------------------------
#14 10.91   WARNING: Discarding file:///root/.cache/pypoetry/artifacts/d0/bb/16/9d113f610914e00bfed3d34597785b6f845d3760f26de934e729821f50/untokenize-0.1.1.tar.gz. Command errored out with exit status 1: /src/.venv/bin/python /root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp6y23q36c Check the logs for full command output.
#14 10.91   ERROR: Command errored out with exit status 1: /src/.venv/bin/python /root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp6y23q36c Check the logs for full command output.
#14 10.91
#14 10.91
#14 10.91   at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1473 in _run
#14 11.01       1469│                 output = subprocess.check_output(
#14 11.01       1470│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
#14 11.01       1471│                 )
#14 11.01       1472│         except CalledProcessError as e:
#14 11.01     → 1473│             raise EnvCommandError(e, input=input_)
#14 11.01       1474│
#14 11.01       1475│         return decode(output)
#14 11.01       1476│
#14 11.02       1477│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
#14 11.02
#14 11.02 The following error occurred when trying to handle this error:
#14 11.02
#14 11.02
#14 11.02   PoetryException
#14 11.02
#14 11.02   Failed to install /root/.cache/pypoetry/artifacts/d0/bb/16/9d113f610914e00bfed3d34597785b6f845d3760f26de934e729821f50/untokenize-0.1.1.tar.gz
#14 11.02
#14 11.02   at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
#14 11.02        47│
#14 11.02        48│     try:
#14 11.02        49│         return environment.run_pip(*args)
#14 11.02        50│     except EnvCommandError as e:
#14 11.02     →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#14 11.02        52│
#14 11.02
------
executor failed running [/bin/sh -c export PATH="/root/.local/bin:$PATH" &&     cd /src &&     python3.6 -m venv .venv &&     . .venv/bin/activate &&     poetry install]: exit code: 1

[tomwatson1024]

Looks like the same error message as #6288, but in this case poetry is installed as recommended (i.e. not in the same venv)

[bearprevent]

It is not entirely the same error message.
In #6288 it is a [Errno 2] No such file or directory, while we get `can't find 'main' module in ....

[dimbleby]

poetry is simply shelling out to pip here, it looks as though your pip is broken.

Similar issue in the pip repository pins the blame on debian - pypa/pip#7946

[tomwatson1024]

I can believe that, though this wasn't a problem in poetry <1.2 - I'm guessing there's been a change in the way poetry invokes pip/which pip it uses? Any suggestions to work around this issue?

[neersighted]

Poetry 1.2 makes use of PEP 517 for a variety of reasons -- there's not a way to disable it for our purposes. You can work around this by installing upstream pip and dropping the patched Debian version in your image -- pip install --upgrade --force-reinstall pip or similar, inside the Poetry environment. poetry self add pip@latest might also work without having to figure out where you installed Poetry.

[jorgeavaldez]

Poetry 1.2 makes use of PEP 517 for a variety of reasons -- there's not a way to disable it for our purposes. You can work around this by installing upstream pip and dropping the patched Debian version in your image -- pip install --upgrade --force-reinstall pip or similar, inside the Poetry environment. poetry self add pip@latest might also work without having to figure out where you installed Poetry.

Just tried out both the pip upgrade using pip install and with poetry self add as well. We're using a docker container for our CI with virtualenv creation set to false and it still failed. The error message I get is

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-22.2.2-py3-none-any.whl'

[neersighted]

Are you installing Poetry into your system site-packages as well as your project? That looks like what I'd expect -- Poetry will be uninstalling its own dependencies out from under itself most likely. Your issue is not related to this thread -- instead it is #6398.

[tomwatson1024]

No luck with either /root/.local/share/pypoetry/venv/bin/pip install --upgrade --force-reinstall pip or poetry self add pip@latest - hitting the same error as above.

[neersighted]

No luck with either /root/.local/share/pypoetry/venv/bin/pip install --upgrade --force-reinstall pip or poetry self add pip@latest - hitting the same error as above.

The No such file or directory (from the system site-packages), or the original error?

[tomwatson1024]

No luck with either /root/.local/share/pypoetry/venv/bin/pip install --upgrade --force-reinstall pip or poetry self add pip@latest - hitting the same error as above.

The No such file or directory (from the system site-packages), or the original error?

Sorry, the original error from my post that includes the Dockerfile.

[neersighted]

I've done some digging and I'm not 100% sure why we're getting these issues, even with Python 3.6 built from source on Debian/Ubuntu. However, the unreleased 1.2 branch does have a behavior change that happens to mitigate this (by using the pip inside the target environment). However, it is very possible to not have a pip in the target environment, so I can't consider it to be a 100% complete fix.

[neersighted]

More debugging has been done -- this is a consequence of Poetry attempting to use its own embedded pip (or rather the pip embedded in the virtualenv module) with an interpreter that that version of pip does not support. We are discussing strategies to resolve this both short-term in 1.2 and longer term now.

[bearprevent]

Thank you for looking into this! Very appreciated.

I also tried built python from source to narrow down the problem.

Poetry tries to run the following and fails:

/tmp/foo/.venv/bin/python /home/john/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip install --disable-pip-version-check --prefix /tmp/foo/.venv --no-deps MarkupSafe-2.0.1.tar.gz

Whereas works fine:

/tmp/foo/.venv/bin/python -m pip install --disable-pip-version-check --prefix /tmp/foo/.venv --no-deps MarkupSafe-2.0.1.tar.gz

The second attempt also uses pip 21.3.1.

So the interpreter seems to support the used version.

[neersighted]

Can you please post the output of pip --version from both invocations?

[bearprevent]

Here is a bit more detailed output and the pip versions.
Both commands should have the same parameters and are the once also used by poetry.

> .venv/bin/python -m pip install --use-pep517 --disable-pip-version-check --prefix /tmp/foo/.venv --no-deps /tmp/MarkupSafe-2.0.1.tar.gz
Processing /tmp/MarkupSafe-2.0.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: MarkupSafe
  Building wheel for MarkupSafe (pyproject.toml) ... done
  Created wheel for MarkupSafe: filename=MarkupSafe-2.0.1-cp36-cp36dm-linux_x86_64.whl size=27428 sha256=0e3312bb73719a8024481c9f3866ba1382605ef58e65e3e4426679ea37082c47
  Stored in directory: /home/john/.cache/pip/wheels/58/e4/dd/2b7b68c9716e8f9f4ca7745a91f3bc71c5e53705fe4c4fd2c5
Successfully built MarkupSafe
Installing collected packages: MarkupSafe
Successfully installed MarkupSafe-2.0.1
> .venv/bin/python -m pip --version
pip 21.3.1 from /tmp/foo/.venv/lib/python3.6/site-packages/pip (python 3.6)

> .venv/bin/python /home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip install --use-pep517 --disable-pip-version-check --prefix /tmp/foo/.venv --no-deps /tmp/MarkupSafe-2.0.1.tar.gz
ERROR: Command errored out with exit status 1:
   command: /tmp/foo/.venv/bin/python /home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpyck2ffd8
       cwd: /tmp/pip-req-build-5wzunie3
  Complete output (1 lines):
  /tmp/foo/.venv/bin/python: can't find '__main__' module in '/home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip/_vendor/pep517/in_process/_in_process.py'
> .venv/bin/python /home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip --version
pip 21.3.1 from /home/john/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip (python 3.6)

[dimbleby]

poetry 1.2.1 doesn't default to the embedded pip. So for those who were hitting this it's likely worth trying again

[sify21]

I had a similiar issue with pyrsistent 0.16.1, poetry 1.2.1 solved it

poetry 1.2.1 doesn't default to the embedded pip. So for those who were hitting this it's likely worth trying again

[Apxdono]

Can confirm as well, poetry 1.2.1 fixed this behavior.

[neersighted]

Glad to hear it -- anyone else who wants to chime in, please use a 👍 on a matching comment to avoid sending unnecessary emails 😄

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.