Hatch doesn’t find Python versions other than its own and …/python{,3} on Linux

[flying-sheep]

If I configure a project with requires_python '>=3.12', on my system I see this error:

hatch/src/hatch/env/virtual.py

Line 258 in 8648544

else 'no compatible Python distribution available'

I don’t understand what the code is supposed to do. It requires either an env variable or one of two functions to not return None:

1. _get_available_distribution seems to call hatch.python.resolve.get_compatible_distributions, and then it removes the “installed distribution”

   hatch/src/hatch/env/virtual.py

   Lines 374 to 376 in 8648544

   	compatible_distributions = get_compatible_distributions()
   	for installed_distribution in self.python_manager.get_installed():
   	compatible_distributions.pop(installed_distribution, None)

   get_compatible_distributions() returns everything hatch can download:

   $ ~/.local/pipx/venvs/hatch/bin/python -c '
   ❯ from hatch.python.resolve import get_compatible_distributions
   ❯ print(get_compatible_distributions().keys())'
   dict_keys(['3.7', '3.8', '3.9', '3.10', '3.11', '3.12', 'pypy2.7', 'pypy3.9', 'pypy3.10'])

   In my case self.python_manager.get_installed().keys() == {'3.12'}, so _get_available_distribution returns None

2. _find_existing_interpreter calls virtualenv.discovery.builtin.propose_interpreters, which yields this on my system:

   PythonInfo(spec=CPython3.11.8.final.0-64, system=/usr/bin/python3.11, exe=/home/phil/.local/pipx/venvs/hatch/bin/python, platform=linux, version='3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]', encoding_fs_io=utf-8-utf-8)
   PathPythonInfo(spec=CPython3.11.8.final.0-64, exe=/usr/bin/python, platform=linux, version='3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]', encoding_fs_io=utf-8-utf-8)

   so that function also returns None.

I can work around this using

pipx install --python=python3.12 hatch

but this should still be fixed. I’d be happy to help, but I don’t know what the intended semantics of _get_available_distribution and _find_existing_interpreter are: Which one of these is supposed to return /usr/bin/python3.12?

[flying-sheep]

When I run rm -rf /home/phil/.local/share/hatch/env/virtual/.pythons/, on the next hatch run, Hatch re-downloads Python 3.12, and it works until the next hatch prune.

I wonder why Hatch downloads Python 3.12 when /usr/bin/python3.12 exists and is in the PATH?

[ofek]

Is this issue present on master?

[flying-sheep]

yes, Hatch 1.9.2.dev93 (8648544) throws the same error (I didn’t check if the APIs return the same values as in 1.9.2)

[ofek]

Please show me:

from virtualenv.discovery.builtin import propose_interpreters
from virtualenv.discovery.py_spec import PythonSpec

def main():
    for data in propose_interpreters(PythonSpec.from_string_spec('3.12'), (), None):
        interpreter = data[0]
        version = '.'.join(map(str, interpreter.version_info[:3]))
        print(f'{version} -> {interpreter.executable}')

if __name__ == '__main__':
    main()

[flying-sheep]

3.11.8 -> /home/phil/.local/pipx/venvs/hatch/bin/python
3.12.2 -> /usr/bin/python3.12
3.11.8 -> /usr/bin/python3
3.11.8 -> /usr/bin/python

[ofek]

That conflicts with what happened in the opening description. Is this a new terminal session? Has PATH been updated since the first one which is why originally there was no 3.12?

[flying-sheep]

OK, I have now slept and executed it with the correct interpreter lol. Edited above with the correct output.

For context on why your script output differs from the hatch invocation: When running hatch run ..., _find_existing_interpreter is only called once, with python_version == ''.

So propose_interpreters is called with PythonSpec.from_string_spec('') instead of ...('3.12').

[ofek]

Can you please then show the output of the empty string change?

[flying-sheep]

When replacing '3.12' with '', it prints:

3.11.8 -> /home/phil/.local/pipx/venvs/hatch/bin/python
3.11.8 -> /usr/bin/python

I think the problem is that it doesn’t find all binaries on the $PATH:

$ ls /usr/bin/python* | where name =~ '\d$'
╭───┬─────────────────────┬─────────┬─────────┬───────────────╮
│ # │        name         │  type   │  size   │   modified    │
├───┼─────────────────────┼─────────┼─────────┼───────────────┤
│ 0 │ /usr/bin/python3    │ symlink │    10 B │ 2 months ago  │
│ 1 │ /usr/bin/python3.10 │ file    │ 14.3 KB │ 3 weeks ago   │
│ 2 │ /usr/bin/python3.11 │ file    │ 14.4 KB │ 2 months ago  │
│ 3 │ /usr/bin/python3.12 │ file    │ 14.2 KB │ 2 months ago  │
│ 4 │ /usr/bin/python3.7  │ file    │ 14.2 KB │ 10 months ago │
│ 5 │ /usr/bin/python3.8  │ file    │ 14.3 KB │ 3 weeks ago   │
│ 6 │ /usr/bin/python3.9  │ file    │ 14.3 KB │ 3 weeks ago   │
╰───┴─────────────────────┴─────────┴─────────┴───────────────╯

[ofek]

That's very interesting, I interpreted empty string as the way to get everything, which I suppose does work on Windows because there is never a version suffix! Oh my

[ofek]

what happens when you try just '3'?

[flying-sheep]

Still only 3.11 found

[flying-sheep]

@gaborbernat is started to extract this from virtualenv recently (https://github.com/tox-dev/py-discovery), maybe he’s immersed deeply enough into the code to help here.

[ofek]

Yes @gaborbernat what is the magic first argument spec I must pass to virtualenv.discovery.builtin.propose_interpreters so that it returns everything? So far it's not an empty string nor '3'. Here is the script:

from virtualenv.discovery.builtin import propose_interpreters
from virtualenv.discovery.py_spec import PythonSpec

def main():
    for data in propose_interpreters(PythonSpec.from_string_spec('3'), (), None):
        interpreter = data[0]
        version = '.'.join(map(str, interpreter.version_info[:3]))
        print(f'{version} -> {interpreter.executable}')

if __name__ == '__main__':
    main()

[flying-sheep]

I don’t think there is any.

• possible_specs(PythonSpec.from_string_spec('')) yields ('', False), ('python', True)
• possible_specs(PythonSpec.from_string_spec('3')) yields ('3', False), ('python3', True), ('python', True)

So these are searched in the PATH. Seems like virtualenv.discvery.builtin doesn’t support finding all Python interpreters on Linux.

[gaborbernat]

I don't know from the top of my head 😔

[flying-sheep]

Well, I checked the code, there doesn‘t seem to be any functionality to find $PATH[$i]/python3.12 when specifying PythonSpec.from_string_spec('') or PythonSpec.from_string_spec('3').

@gaborbernat is there interest to support that? Then I think I can come up with an algorithm that works. A more robust version of:

import os
import re
from collections.abc import Generator, MutableMapping
from pathlib import Path

from virtualenv.discovery.builtin import get_paths, PathPythonInfo


def find_candidates(
    spec: str,
    env: MutableMapping[str, str] | None = None,
) -> Generator[Path, None, None]:
    env = os.environ if env is None else env
    maj, min = spec.split(":") if ":" in spec else (spec, r"\d+")
    maj = maj or r"\d+"
    regex = f"python|(python)?{maj}(\.{min}+)?"
    regex = re.compile(f"({regex}).exe" if sys.platform == "win32" else regex)
    
    tested_exes = set()
    for found in (found for path in get_paths(env) for found in Path(path).iterdir()):
        if not regex.fullmatch(found.name) or not found.is_file():
            continue
        exe = found.resolve()
        if exe in tested_exes:
            continue
        tested_exes.add(exe)
        got = PathPythonInfo.from_exe(exe, raise_on_error=False, env=env)
        if got is not None:
            yield got, True  # TODO: what is the bool here?


if __name__ == "__main__":
    import sys
    for path, match in find_candidates(sys.argv[1] if len(sys.argv) > 1 else ''):
        print(path, match)

[musicinmybrain]

In Fedora Linux, we can’t upgrade our python-virtualenv package past 20.21.1 until we retire support for Python 2.7 and 3.6. We can backport individual fixes if they are straightforward enough.

Now that we finally managed to package uv, I’m able to consider updating our python-hatchling package past 1.21.1 and hatch past 1.9.7. I’m trying to figure out if I need to attempt a backport of pypa/virtualenv#2708 in order to do this, or if the issue is a regression introduced sometime after virtualenv 20.21.1 and we are not affected. Do you have any insight into that?

Since I’m not intimately familiar with Hatch’s expected behavior in different environments and situations, I haven’t been able to figure out a simple, straightforward test I can run in a Fedora Rawhide chroot to try to reproduce this bug. That would help me figure out the answer for myself, so any suggestions in that direction are appreciated as well.

[ofek]

I don't have time to check but please note that you actually would require both of these because the first introduced another regression:

• Allow builtin interpreter discovery to find specific Python versions given a general spec virtualenv#2709
• Fix PATH-based Python discovery on Windows virtualenv#2712

[musicinmybrain]

Hmm, I had ignored the second PR because it fixed a problem “on Windows,” but reading the description, it sounds like “Addresses another issue found during debugging which is that the PR that introduced the regression took out a check to make sure that executables exist before trying to gather data about the interpreter” might still apply to us.