Investigate numpy 2.0.0 support

[veenstrajelmer]

Is your feature request related to a problem? Please describe.
Numpy 2.0.0 was released on 16 June 2024. HYDROLIB-core enforces numpy<2.0.0 via the pyproject.toml, so it will not be installed. However it is good to be future proof and test if everything still works when also allowing numpy 2.0.0 and above.

Describe the solution you'd like
A quick investigation at least, preferably resulting in support for numpy>=2

Additional information
Poetry is always blocking these types of tests. In fixing #599 we saw that poetry thinks that all python versions must be able to install the same numpy version for some reason. It seems very likely that supporting numpy 2 is only neatly possible after dropping python 3.8 support.

when running the tests with numpy>=2.0, there are several test failing:

ERROR tests/dflowfm/ini/test_ini.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_mdu.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_net.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_obs.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_obscrosssection.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_onedfield.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_research.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_storagenode.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_structure.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_xyn.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dflowfm/test_xyz.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/dimr/test_dimr.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/rr/test_fnm.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/test_basemodel.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/test_docker.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/test_model.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...
ERROR tests/test_utils.py - ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from P...`

Related issue: Unidata/netcdf4-python#1354

[evetion]

Python 3.8 is end of life (EOL) in 3 months and 3 weeks (31 Oct 2024, 5 years old at that point), so dropping that seems a good thing.

[visr]

I did some quick static analysis based on https://docs.astral.sh/ruff/rules/numpy2-deprecation/, with:

[lint]
select = ["NPY"]

Which gives me:

❯ pixi run ruff check
tests\dflowfm\test_bc.py:362:21: NPY002 Replace legacy `np.random.uniform` call with `np.random.Generator`
    |
361 |     def test_forcing_model_with_datablock_that_has_nan_values_should_raise_error(self):
362 |         datablock = np.random.uniform(low=-40, high=130.3, size=(4, 2)) * np.nan
    |                     ^^^^^^^^^^^^^^^^^ NPY002
363 |         datablock_list = datablock.tolist()
    |

So the code here should already work on both pre and post NumPy 2. Looking at the dependencies, they should also all have NumPy 2 compatible releases out now. I just had to double check MeshKernel, but doing the same check there didn't highlight any issues either.

So perhaps hydrolib can just stop enforcing numpy<2 and do a new release?

[veenstrajelmer]

Thanks @visr for this analysis

As stated, this requires either of these:

• dropping py3.8 support
• moving from poetry to e.g. pixi (or define numpy version requirements per python version)

Both are not a big effort, but at the moment there is no time to do or discuss these options. However, it will be looked at during the next sprint I believe. Is it ok to wait a bit or are you running into issues now?

[visr]

Ok I'm not a poetry user, I thought that issue just affected your ability to test the numpy compat, but it affects declaring the numpy compat.

I ran into it now in a project, but since the D-Hydro related feature is not a direct requirement we'll just remove hydrolib support, so it isn't an issue.

[veenstrajelmer]

The ValueError: numpy.dtype size changed occured because of mismatching numpy and netcdf4 libraries. Poetry failed to resolve these apparently. Furthermore, it tries to install pandas 2.0.3 for all python versions, since python 3.8 support was dropped from pandas 2.1 onwards. Pandas 2.0.3 does not support python 3.12 however, but poetry is not super flexible in installing different package versions for different python versions. The simplest way forward seemed to drop python 3.8 support, official support was also dropped three days ago (7 Oct 2024), so I guess it is now ok to do so.

However, when doing that, poetry locks numpy on 2.0.2 since that is the latest version available for python 3.9. When installing this numpy version on python 3.12 together with netcdf4==1.7.1.post2 (the latest), this results in RuntimeError: only endian='native' allowed for NETCDF3 files, got 'big' (variable 'network', group '/'). This also happens when forcing numpy>=2.1.2 for python 3.12. What is odd is that this error does not occur in a very similar environment (same python version, dependencies also installed via poetry lockfile). Reported in Unidata/netcdf4-python#1373. Hopefully it will be resolved with a next netcdf4 release.

edit: the netcdf issue seems to be resolved in Unidata/netcdf4-python#1355, so just await a new release should do the trick. Might be wise to define that as a minimal version for 3.13 and maybe also for 3.12.