vaex does not work with numpy==2.0.0

[iisakkirotko]

Hi! Noticed an issue caused by vaex in our daily ci run on solara. This issue should break any new install of vaex where the version of numpy is not restricted.

Description:
vaex is not compatible with the newly released numpy==2.0.0.

Software information

• Vaex version: any
• Vaex was installed via: pip
• OS: any

To Reproduce:

install vaex, which pulls in the newest numpy. Run import vaex. This results in

Traceback (most recent call last):  File "<stdin>", line 1, in <module>
  File "/Users/.../lib/python3.10/site-packages/vaex/__init__.py", line 45, in <module>
    import vaex.dataframe
  File "/Users/.../lib/python3.10/site-packages/vaex/dataframe.py", line 30, in <module>
    import vaex.grids
  File "/Users/.../lib/python3.10/site-packages/vaex/grids.py", line 3, in <module>
    import vaex.vaexfast
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/.../lib/python3.10/site-packages/vaex/__init__.py", line 45, in <module>
    import vaex.dataframe
  File "/Users/.../lib/python3.10/site-packages/vaex/dataframe.py", line 30, in <module>
    import vaex.grids
  File "/Users/.../lib/python3.10/site-packages/vaex/grids.py", line 3, in <module>
    import vaex.vaexfast
ImportError: numpy.core.multiarray failed to import

with the following hint:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

[RubTalha]

https://stackoverflow.com/questions/78641150/a-module-that-was-compiled-using-numpy-1-x-cannot-be-run-in-numpy-2-0-0

[ddelange]

for now, vaex-core will be pinning numpy~=1.17 in #2331 (it's currently numpy>=1.16 so it incorrectly allows numpy v2)

[EwoutH]

Would it work to pin numpy<2? That way you can use some more modern releases (up to NumPy 1.26.4)

[maartenbreddels]

I think that numpy~=1.17 means >=1.17, <2 right? https://github.com/vaexio/vaex/blob/e3e1842d3c3d70a4d33dd4f80cb71ba12943f721/packages/vaex-core/setup.py

Good to check before we release :)

[EwoutH]

I think that numpy~=1.17 only allows upgrading within the 1.17.x range, meaning versions like 1.17.1, 1.17.2, etc., but not 1.18.0 or higher.

Edit: Nevermind, I think it does allow <2.0.

https://peps.python.org/pep-0440/#compatible-release

[maartenbreddels]

numpy~=1.17.0 does mean what you mentioned, it's different from the JS ecosystem (npm) where they use ^ and ~.

[ddelange]

yes, and poetry adds caret (numpy^1.17.1). it's nice to exclude everything that came before that specific patch release, but still open up towards the next major release. this is not possible with the PEP440 compatible release clause, which is sometimes a bit annoying.

[ddelange]

fyi to support Python 3.13, vaex will need to drop python 3.8, and move build-system to numpy 2.1+ ref https://github.com/scipy/oldest-supported-numpy/pull/86/files

runtime can still support both v2 and recent versions of v1, see pandas for instance

[maartenbreddels]

Maybe do 1 release with < 2, and the next release go to 2.1+?

[ddelange]

apart from #2434 the current state (python 3.8 - 3.12 support with numpy v1 support) is ready for release as for me.

this ticket should stay open, bumping numpy to v2 will need some work I think

[maartenbreddels]

Indeed.
I think next week, we should try a release first with what is currently in master (module a few ci fixes).
Would be nice to get numpy 2 support in for the next version.