Wrong Python version in a docker container running pex

[mdelmans]

Describe the bug

$ pants run src/py/docker_test:main

/usr/bin/env: ‘python3.9’: No such file or directory

Pants version
2.16

OS
Mac OS 13.4 Intel

Additional info

# src/py/docker_test/main.py

if __name__ == "__main__":
    print('Hello world!')

# src/py/docker_test/BUILD

python_sources()

pex_binary(
    name="main",
    entry_point="main.py",
    platforms=[
        "linux-x86_64-cp-310-cp310m",
        "macosx_10.12_x86_64-cp-310-cp310"
    ]
)

docker_image(
    name="docker"
)

# src/py/docker_test/Dockerfile

FROM python:3.10-slim
ENTRYPOINT ["/bin/main"]
COPY src.py.docker_test/main.pex /bin/main

# pants.toml

[python]
interpreter_constraints = ["CPython==3.10.*"]

If I remove platforms from the pex_binary everything works fine.

[jward-premise]

I had this same problem too starting with 2.16.0. Here is a work around that worked for me.

In pants.toml, tell pants to use your PYENV installs to run pex. It defaults to ""

[pex]
executable_search_paths = ["<PYENV>"]

Then, where you build your PEX targets, override the default shebang line:

pex_binary(
    name="bin",
    entry_point="foo.bar.main",
    shebang="#!/usr/bin/env python3",
)

Hopefully that helps!

[mdelmans]

Thanks @jward-premise ! Not sure I understand what's happening but as long as it works, well, it woks.

[cburroughs]

Just to confirm, this worked in 2.15 and only started in 2.16?

Some background at pex-tool/pex#1540

[cburroughs]

Okay I have pants able to pass --sh-boot plumbed through

  --sh-boot, --no-sh-boot
                        Create a modified ZIPAPP that uses `/bin/sh` to boot.
                        If you know the machines that the PEX will be
                        distributed to have POSIX compliant `/bin/sh` (almost
                        all do, see: https://pubs.opengroup.org/onlinepubs/969
                        9919799/utilities/sh.html); then this is probably the
                        way you want your PEX to boot. Instead of launching
                        via a Python shebang, the PEX will launch via a
                        `#!/bin/sh` shebang that executes a small script
                        embedded in the head of the PEX ZIPAPP that performs
                        initial interpreter selection and re-execution of the
                        underlying PEX in a way that is often more robust than
                        a Python shebang and always faster on 2nd and
                        subsequent runs since the sh script has a constant
                        overhead of O(1ms) whereas the Python overhead to
                        perform the same interpreter selection and re-
                        execution is O(100ms). (default: False)

But! I'm not exactly sure what to do with that. The current state as I understand it:

• pex defaults to using a shebang from the python that invoked it, aka the "hermetically sealed" PBS that comes with pants.
• This will "accidentally" work if your target has python3.9, but otherwise breaks as described in this ticket. Before scie-pants this wasn't guaranteed to work either but "the version of python that happened to invoke pants with" probably had a higher chance of matching the deployment target, or at least existing on the local workstation!
• Fun tangent: I didn't notice this for ~a year because the official docker Python 3.10 image also has Python 3.9.
• When Pants upgrades to Python > 3.9, a different set of users will subtly break.

Some Options:

• (A) Do basically nothing! Maybe document shebang more prominently or plumb through --sh-boot with a default of False. Users will continue to trip over this, and each PBS version bump is in effect a "breaking change".
• (B) Plumb --sh-boot through with a default of True. About a breaking change as the PBS bump above ^^. I've toyed around with this locally, but it's not a small change I'm unsure of the "blast radius".
• (C) Solve this better on the pex side as described in the linked ticket.
• (D) Use Pant's knowledge of interpreter constraints to make a slightly better but still possibly wrong ad-hoc guess at a python based shebang.

(B) feels like the least bad option -- or the default pants would start with today -- but I'm unsure of the deprecation policy, testing, and messaging to get there.

[jsirois]

Plumbing the --sh-boot option makes sense to me if even just because its faster than without, but I wanted to note that when Pants incorporates Pex 2.1.154 with pex-tool/pex#2296 the narrow case of the OP with 2 CPython 3.10 platforms will work without resorting to --sh-boot.

[jsirois]

Pex 2.1.154 is now released with the feature from pex-tool/pex#2296 that correctly auto-selects an appropriate shebang for the OP case: https://github.com/pantsbuild/pex/releases/tag/v2.1.154