pip install hatch fails on free-threaded build

[ngoldbaum]

This ends up coming down to the dependency on zstandard:

± pip install hatch
Collecting hatch
  Downloading hatch-1.13.0-py3-none-any.whl.metadata (5.6 kB)
Collecting click>=8.0.6 (from hatch)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting hatchling>=1.24.2 (from hatch)
  Downloading hatchling-1.26.1-py3-none-any.whl.metadata (3.8 kB)
Collecting httpx>=0.22.0 (from hatch)
  Using cached httpx-0.27.2-py3-none-any.whl.metadata (7.1 kB)
Collecting hyperlink>=21.0.0 (from hatch)
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting keyring>=23.5.0 (from hatch)
  Downloading keyring-25.5.0-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: packaging>=23.2 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (24.1)
Collecting pexpect~=4.8 (from hatch)
  Using cached pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB)
Requirement already satisfied: platformdirs>=2.5.0 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (4.3.6)
Collecting rich>=11.2.0 (from hatch)
  Downloading rich-13.9.4-py3-none-any.whl.metadata (18 kB)
Collecting shellingham>=1.4.0 (from hatch)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting tomli-w>=1.0 (from hatch)
  Downloading tomli_w-1.1.0-py3-none-any.whl.metadata (5.7 kB)
Collecting tomlkit>=0.11.1 (from hatch)
  Downloading tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Collecting userpath~=1.7 (from hatch)
  Downloading userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uv>=0.1.35 (from hatch)
  Downloading uv-0.5.1-py3-none-macosx_11_0_arm64.whl.metadata (11 kB)
Requirement already satisfied: virtualenv>=20.26.1 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (20.27.1)
Collecting zstandard<1 (from hatch)
  Downloading zstandard-0.23.0.tar.gz (681 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 681.7/681.7 kB 28.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting packaging>=23.2 (from hatch)
  Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pathspec>=0.10.1 (from hatchling>=1.24.2->hatch)
  Downloading pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Collecting pluggy>=1.0.0 (from hatchling>=1.24.2->hatch)
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting trove-classifiers (from hatchling>=1.24.2->hatch)
  Downloading trove_classifiers-2024.10.21.16-py3-none-any.whl.metadata (2.2 kB)
Collecting anyio (from httpx>=0.22.0->hatch)
  Using cached anyio-4.6.2.post1-py3-none-any.whl.metadata (4.7 kB)
Collecting certifi (from httpx>=0.22.0->hatch)
  Using cached certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx>=0.22.0->hatch)
  Using cached httpcore-1.0.6-py3-none-any.whl.metadata (21 kB)
Collecting idna (from httpx>=0.22.0->hatch)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting sniffio (from httpx>=0.22.0->hatch)
  Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.22.0->hatch)
  Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting jaraco.classes (from keyring>=23.5.0->hatch)
  Downloading jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring>=23.5.0->hatch)
  Downloading jaraco.functools-4.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring>=23.5.0->hatch)
  Downloading jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Collecting ptyprocess>=0.5 (from pexpect~=4.8->hatch)
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting markdown-it-py>=2.2.0 (from rich>=11.2.0->hatch)
  Downloading markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich>=11.2.0->hatch)
  Using cached pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
Requirement already satisfied: distlib<1,>=0.3.7 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from virtualenv>=20.26.1->hatch) (0.3.9)
Requirement already satisfied: filelock<4,>=3.12.2 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from virtualenv>=20.26.1->hatch) (3.16.1)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=11.2.0->hatch)
  Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Collecting more-itertools (from jaraco.classes->keyring>=23.5.0->hatch)
  Downloading more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Downloading hatch-1.13.0-py3-none-any.whl (125 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
Downloading hatchling-1.26.1-py3-none-any.whl (75 kB)
Using cached httpx-0.27.2-py3-none-any.whl (76 kB)
Using cached httpcore-1.0.6-py3-none-any.whl (78 kB)
Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Downloading keyring-25.5.0-py3-none-any.whl (39 kB)
Using cached packaging-24.2-py3-none-any.whl (65 kB)
Using cached pexpect-4.9.0-py2.py3-none-any.whl (63 kB)
Downloading rich-13.9.4-py3-none-any.whl (242 kB)
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading tomli_w-1.1.0-py3-none-any.whl (6.4 kB)
Downloading tomlkit-0.13.2-py3-none-any.whl (37 kB)
Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
Downloading uv-0.5.1-py3-none-macosx_11_0_arm64.whl (12.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.5/12.5 MB 29.5 MB/s eta 0:00:00
Using cached idna-3.10-py3-none-any.whl (70 kB)
Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
Downloading pathspec-0.12.1-py3-none-any.whl (31 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Using cached pygments-2.18.0-py3-none-any.whl (1.2 MB)
Using cached anyio-4.6.2.post1-py3-none-any.whl (90 kB)
Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
Downloading jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Downloading trove_classifiers-2024.10.21.16-py3-none-any.whl (13 kB)
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
Building wheels for collected packages: zstandard
  Building wheel for zstandard (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for zstandard (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module '_cffi_backend', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
      not modified: 'build/zstandard/_cffi.c'
      generating build/zstandard/_cffi.c
      (already up-to-date)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.macosx-14.7-arm64-cpython-313
      creating build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/__init__.py -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/backend_cffi.py -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/__init__.pyi -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/py.typed -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      running build_ext
      building 'zstandard.backend_c' extension
      creating build/temp.macosx-14.7-arm64-cpython-313
      creating build/temp.macosx-14.7-arm64-cpython-313/c-ext
      clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -I/opt/homebrew/opt/zlib -I/opt/homebrew/opt/zlib -Ic-ext -Izstd -I/Users/goldbaum/.pyenv/versions/3.13.0t/include/python3.13t -c c-ext/backend_c.c -o build/temp.macosx-14.7-arm64-cpython-313/c-ext/backend_c.o -DZSTD_SINGLE_FILE -DZSTDLIB_VISIBLE= -DZDICTLIB_VISIBLE= -DZSTDERRORLIB_VISIBLE= -fvisibility=hidden
      c-ext/backend_c.c:316:17: error: no member named 'ob_refcnt' in 'struct _object'
        316 |     if ((*obj)->ob_refcnt == 1) {
            |         ~~~~~~  ^
      1 error generated.
      error: command '/opt/homebrew/opt/ccache/libexec/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for zstandard
Failed to build zstandard

see indygreg/python-zstandard#231 for the zstandard issue.

It looks like zstandard is only used if you encounter a .tar.zst file. I'm not sure if there's an easy way to do this, but if it were an optional dependency then most users who don't need zstandard support could get past this issue.

[ofek]

This is a tough one, I'm going to see if the maintainer is willing to accept a contribution and release it.

[ngoldbaum]

I pinged again on the python-zstandard issue tracker.

The lack of a way to specify the free-threaded build in a pyproject.toml makes it impossible to do the most obvious fix and remove python-zstandard from hatch's dependencies on the free-threaded build. The only thing that's possible right now is to e.g. make it optional on 3.13 or newer or just remove it from the required dependencies completely and perhaps add it to an extras section.

That could work, but it's certainly annoying that users would see a runtime error about the python-zstandard library not being installed instead of just having it work on the GIL-enabled build of 3.13 as it does right now.

I would only do this as a last resort, but would you be open to adding a dependency on e.g. python-zstandard-ft-support which would track the upstream project (which doesn't change much these days) and apply the PR that adds free-threading support. Obviously not a great option but it might be the best possibility?