building python 3.13 wheels is failing on linux

[jswhit]

see https://github.com/Unidata/netcdf4-python/actions/runs/10985046122

[jswhit]

@ocefpaf

[ocefpaf]

I'll take a look at this next week. I think I know what the problem is.

[veenstrajelmer]

Might be related, when installing netCDF4 on linux with pip, I get ValueError: did not find HDF5 headers

[EDM115]

fyi it also fails on Windows :

Collecting netCDF4==1.7.1.post2 (from -r ./pip.txt (line 123))
  Downloading netcdf4-1.7.1.post2.tar.gz (827 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 827.8/827.8 kB 1.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      reading from setup.cfg...
          HDF5_DIR environment variable not set, checking some standard locations ..
      checking C:\Users\EDM115\include...
      hdf5 headers not found in C:\Users\EDM115\include
      checking /usr/local\include...
      hdf5 headers not found in /usr/local\include
      checking /sw\include...
      hdf5 headers not found in /sw\include
      checking /opt\include...
      hdf5 headers not found in /opt\include
      checking /opt/local\include...
      hdf5 headers not found in /opt/local\include
      checking /opt/homebrew\include...
      hdf5 headers not found in /opt/homebrew\include
      checking /usr\include...
      hdf5 headers not found in /usr\include
      Traceback (most recent call last):
        File "D:\EDM115\Programmes\Python313\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "D:\EDM115\Programmes\Python313\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\EDM115\Programmes\Python313\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "C:\Users\EDM115\AppData\Local\Temp\pip-build-env-fpgem0g5\overlay\Lib\site-packages\setuptools\build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\EDM115\AppData\Local\Temp\pip-build-env-fpgem0g5\overlay\Lib\site-packages\setuptools\build_meta.py", line 302, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "C:\Users\EDM115\AppData\Local\Temp\pip-build-env-fpgem0g5\overlay\Lib\site-packages\setuptools\build_meta.py", line 318, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 277, in <module>
        File "<string>", line 226, in _populate_hdf5_info
      ValueError: did not find HDF5 headers
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

[veenstrajelmer]

And also on macOS, The Python 3.13 testcases on windows, ubuntu and macOS all fail because of the same reason: https://github.com/Deltares/HYDROLIB-core/actions/runs/11250994627/job/31281068148
@jswhit could you rerun your pipeline to see if it results in the same error as @EDM115 and I reported? I noticed it was ran before python 3.13 was officially available so it might behave slightly different today.

[ocefpaf]

@veenstrajelmer are you using our docker image? You should not see a similar failure if you use the built wheel. BTW, the issue should be resolved. Also, it is odd that you have failures on Windows and macOS too. I don't think your failure is related to this issue, but I may be wrong.

[veenstrajelmer]

I'm not using the docker images. My pip install netcdf4 commands are failing everywhere, on different platforms in Github actions and also on my local windows 10 machine with conda environments. Are you saying you would expect that it is possible to install netcdf4 via pip in a python 3.13 environment? I just assumed it was not supported yet since I cannot get it installed anywhere

[ocefpaf]

I'm not using the docker images.

Then your failures are unrelated to this issue. This issue reports how the wheel building mechanism here was failing on Linux. That happenedb/c the docker image had a RC release of Python 3.13. This was fixed in the docker image in past few days, and everything is working now.

My pip install netcdf4 commands are failing everywhere, on different platforms in Github actions and also on my local windows 10 machine with conda environments.

I'm guessing for 3.13, right? That is probably expected with pip as there are no wheels for Python 3.13, Python 3.13 itself was just released. However, if you are using conda environments, and you don't really need pip, you can use the conda package for it:

conda env create --name PY313 "python=3.13" netcdf4

It should give you netcdf4 on Python 3.13.

Are you saying you would expect that it is possible to install netcdf4 via pip in a python 3.13 environment? I just assumed it was not supported yet since I cannot get it installed anywhere

What I'm saying is that there are no built wheels. You can install it by compiling it yourself. You will need to provide the HDF5 and netcdf-c dependencies, which are missing from your builds there according to the error message. With that said, I would not build my own, I'd use conda's package or wait on #1370 to use pip.

[veenstrajelmer]

Thanks for the elaborate response, sorry for polluting this issue with off-topic comments in that case. I was not aware this issue was docker specific, just noticed the wheels failing to build for python 3.13. So it was the most relevant issue I could find earlier. Thanks for creating #1370, I will link to that one instead!