Shebangs using different Python binary with latest Docker image in 3.10

[stephenmelrose]

Hi there,

We use Python 3.10 for our application Docker images. Our build process utilizes the python:3.10 Docker image to do a pip3 install, and then we copy all the dependencies over into our main Docker image in a multi-stage build. The main Docker image in question is Ubuntu Jammy based and also has Python 3.10.

We install various Python packages in our Docker image. One for example is pylint. Before today, the shebang for /usr/local/bin/pylint was #!/usr/local/bin/python. Today, this is now #!/usr/local/bin/python3.10. Said file does not exist.

I believe this is related to the latest Docker image release: https://hub.docker.com/layers/library/python/3.10/images/sha256-88cf7f1097c7a1afcf157272fd5b82709a97a68bbc4ad97a5a3119af7f2fe58b?context=explore

I'm not sure what has caused this change, likely a pip update or something? But I wanted to report it in the event it's a bug.

Thanks

[stephenmelrose]

The shebang is set/determined by the runtime pip3 is run within. For the linked above 3.10.15 image the shebang for /usr/local/bin/pip3 is #!/usr/local/bin/python3.10, and for tag 3.10.14 the shebang is #!/usr/local/bin/python. So this is changed in the actual Docker image by something.

Interestingly both Docker images are running pip version 23.0.1. So I'm not sure what has set this.

[LaurentGoderre]

@edmorley I wonder if this change in behavior could be coming from the switch to ensurepip

[edmorley]

This will presumably be a side-effect of #955.

The prior get-pip.py script was run using python get-pip.py (so I would imagine inherited the versionless python command from that), whereas I imagine the ensurepip run during the build will be run via the exact Python binary that was just built (so eg python3.10):
https://github.com/python/cpython/blob/b1d6f8a2ee04215c64aa8752cc515b7e98a08d28/Makefile.pre.in#L2198

Both shebang lines are technically valid, though I personally feel the new more precise shebang line is generally safer and more appropriate, since it will avoid problems in environments where there are multiple Python versions installed.

The main Docker image in question is Ubuntu Jammy based and also has Python 3.10.

So I just installed python3 on Ubuntu 22.04, and it's not installed into /usr/local/bin/ - so this must be a symlink you are setting up manually?

$ docker run --rm -it ubuntu:22.04 bash
...
root@27ab5e2a497e:/# apt-get update -qq
root@27ab5e2a497e:/# apt-get install -qq python3
...
root@27ab5e2a497e:/# which python3
/usr/bin/python3
root@27ab5e2a497e:/# ls -al /usr/local/bin/
total 12
drwxr-xr-x 2 root root 4096 Aug  8 15:17 .
drwxr-xr-x 1 root root 4096 Aug  8 15:17 ..
root@27ab5e2a497e:/#

If so, I would add additional symlinks for the other python command name variants too (eg python3.10, python3, python etc).

(Though in general, I suspect that trying to copy files from a Debian image to an Ubuntu image is always going to be a bit fragile, and require lots of fixups.)

[stephenmelrose]

Both shebang lines are technically valid, though I personally feel the new more precise shebang line is generally safer and more appropriate, since it will avoid problems in environments where there are multiple Python versions installed.

Oh for sure, I don't disagree there. Just felt like an unintended change that, for us at least, has had some knock on consequences. One was python3.10 missing, but that's our fault due to our strange setup.

The other was our Helm chart. We had a couple of probes that were doing pidof python and now don't work as the processes are running under python3.10.

Either way, if you feel this is fine/valid that's ok, I just wanted to surface it.

[edmorley]

Ah that makes sense - thank you for raising it. It wasn't a side-effect I'd spotted before (and even if I had, I probably wouldn't have imagined those impacts), so it's good to know about.

If we wanted to switch the shebangs back in the image, we could disable ensurepip during the build again, but instead of switching back to get-pip.py, we could run python -m ensurepip after the Python compile - which I'm presuming would result in python shebangs.

Though as mentioned above, I tend to think the more precise shebangs are probably best longer term.

For anyone affected, a quick way to switch back to the old shebangs as a short-term workaround, is by invoking pip via python -m pip, eg:

$ docker run --rm python:3.10.15-slim bash -c 'python -m pip install pylint &> /dev/null && head -n1 /usr/local/bin/pylint'
#!/usr/local/bin/python