Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange · 2023-01-20T14:54:17Z

Hoi 👋

linux-aarch64 makes up for almost 10% of all platforms ref giampaolo/psutil#2103

aarch64 has already surpassed windows in terms of downloads for this package. Oracle, Amazon, Google, and Microsoft are all offering aarch64 cloud instances at an undeniable price point compared to amd/intel, so the demand will undoubtedly only grow

this PR is adapted from Add arm64 mac and linux wheels MagicStack/asyncpg#954
uses QEMU emulation for linux arm64 wheels: manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs 😅
manylinux2014 wheels are built with GCC 10, which I think does not guarantee proper functioning of pybind11 (docs).
- so with this PR, linux wheels are built with GCC 12 (manylinux_2_28).
- pip will only install these wheels on linux operating systems with glibc >= 2.28 (mostly all 2020+ linux distributions like debian 10 buster, ubuntu 20.04 focal, almalinux/rhel 8, ...).

the wheels from this PR can be installed with:

# comma separated list for --find-links
export PIP_FIND_LINKS=https://github.com/ddelange/vaex/releases/expanded_assets/core-v4.17.1.post4
pip install --force-reinstall vaex

fixes #2366, fixes #2368, fixes #2397, closes #2427, fixes #2384

maartenbreddels · 2023-01-21T08:32:29Z

Hoi 👋

exciting, will take a look early next week!

manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs

that worries me a bit.. :)

groeten,

Maarten

ddelange · 2023-01-21T09:40:18Z

here are all timings: https://github.com/ddelange/vaex/actions/runs/3965720337/usage

depending on how often a month you release vaex, this could eat into the 2k free minutes of GH...

as the parallelization is maximised and they're pushed to PyPI as soon as they're built, most of the wheels will be available soon upon release regardless

here are all the wheels: distributions.zip

ddelange · 2023-01-21T10:32:44Z

interestingly, that was 8260 minutes ^

apparently that's OK? then I don't understand their explanation 🤔 https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes

ddelange · 2023-01-21T10:40:57Z

ah there is a fair amount of duplication in that usage table for whatever reason 🤯

ddelange · 2023-01-22T10:54:32Z

a diff of current PyPI vs the zip above:

 vaex_core-4.16.1-cp310-cp310-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp310-cp310-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-win_amd64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_10_9_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_11_0_arm64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-win_amd64.whl
 vaex_core-4.16.1-cp36-cp36m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp36-cp36m-win_amd64.whl
 vaex_core-4.16.1-cp37-cp37m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp37-cp37m-win_amd64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp38-cp38-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-win_amd64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp39-cp39-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-win_amd64.whl

ddelange · 2023-01-23T07:30:45Z

packages/vaex-core/src/hash_object.cpp

-namespace std {
-    template<>
-    struct hash<PyObject*> {
-        size_t operator()(const PyObject *const &o) const {
-            return PyObject_Hash((PyObject*)o);
-        }
-    };
-}


This was my best guess...

pybind11 ref https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859

failure log https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110

@maartenbreddels any thoughts on this (incl me updating the pybind11 submodule)?

ddelange · 2023-01-23T07:31:30Z

packages/vaex-core/setup.py

@@ -183,12 +183,14 @@ def __str__(self):
      include_package_data=True,
      ext_modules=([extension_vaexfast] if on_rtd else [extension_vaexfast, extension_strings, extension_superutils, extension_superagg]) if not use_skbuild else [],
      zip_safe=False,
+      python_requires=">=3.6",


cibuildwheel parses this to determine which wheels to build

cc @franz101

see also the diff above

ddelange · 2023-02-27T07:19:30Z

I'm guessing this is blocked by #2339

maartenbreddels · 2023-03-06T13:11:53Z

Just letting you know i'm very busy and had a vacation.
Yes, I'll try to get #2339 green first!

ddelange · 2023-05-25T23:20:01Z

fwiw there are now third party free minutes on native arm64 machines, to get rid of the slow qemu builds

maartenbreddels · 2023-07-10T20:25:06Z

Could you try rebasing this?

ddelange · 2023-07-10T20:48:15Z

@maartenbreddels already merged in master 👍

ddelange · 2023-07-11T06:55:46Z

    ERROR: Could not find a version that satisfies the requirement vaex-core<4.17,>=4.17.0 (from vaex)
    ERROR: No matching distribution found for vaex-core<4.17,>=4.17.0

maartenbreddels · 2023-07-11T15:50:10Z

Yeah, a bug/artifact or our release script. Should be good now.

ddelange · 2023-08-03T20:50:04Z

hoi @maartenbreddels 👋

I pulled master and fixed merge conflicts, but it looks like CI is still not very happy. Seeing errors like hdf file missing on disk, and TypeError: train() got an unexpected keyword argument 'early_stopping_rounds'.

Do you think it might be related to this PR?

franz101 · 2023-08-17T20:51:39Z

Just wondering here on the Python packaging. Python 3.6 and 3.7 are now deprecated on the other hand we can bump to 3.10 and 3.11?

to-bee · 2023-08-28T07:48:57Z

Do we have any updates on this MR?

ddelange · 2023-09-01T08:07:30Z

HI @maartenbreddels 👋

Was your s3 account deleted by any chance?

vaex.open('s3://vaex/taxi/yellow_taxi_2009_2015_f32.hdf5?anon=true')

raises

FileNotFoundError: [Errno 2] Path does not exist 'vaex/taxi/yellow_taxi_2009_2015_f32.hdf5'. Detail: [errno 2] No such file or directory

maartenbreddels · 2024-08-30T10:26:56Z

Love the fix! I was afraid this didn't work, because you cannot release with references to files or URLs. but the top level setup.py does not see pypi, so it should be good.

I'm installing vaex on a windows machine to see if I can debug that segfault.

maartenbreddels · 2024-08-30T10:27:44Z

I think 91263bc makes it impossible to release on pypi, they will refuse an upload.

ddelange · 2024-08-30T10:30:39Z

ahh of course!

This reverts commit 91263bc.

maartenbreddels · 2024-08-30T13:35:27Z

I ran Windows + Python 3.8 100s of times, and I cannot reproduce the failure. I'm leaning to either:

Drop Python 3.8 support on windows only (is that possible?)
Release it anyway, assuming it is a 'fluke'

Ideas?

ddelange · 2024-08-30T16:50:36Z

looks like not allowed. have a feeling it might be windows 3.8 threading implementation memory leak

ddelange · 2024-08-30T16:57:27Z

you could simply skip publish of that vaex-core wheel, and don't upload tar.gz for vaex-core. it's the tensorflow approach

done in the commits below

.github/workflows/wheel.yml

ddelange · 2024-08-30T19:08:00Z

.github/workflows/wheel-universal.yml

+    # https://github.com/pypa/gh-action-pypi-publish#trusted-publishing
+    - name: Publish package distributions to PyPI
+      uses: pypa/gh-action-pypi-publish@v1.9.0
+      if: startsWith(github.ref, 'refs/tags')
+      with:
+        skip-existing: true


all you need to do for this to work is add vaexio/vaex as trusted publisher here: https://pypi.org/manage/project/vaex/settings/publishing (and to the other vaex pypi projects)

ddelange · 2024-08-31T15:02:31Z

that did it. we're green 🏁

fwiw, pandas deprecated python 3.8 in August 2023 (v2.1.0)

maartenbreddels · 2024-09-05T14:53:33Z

Whow, amazing. You did most of the work @ddelange thanks a lot for the work and support !

I'll set up the trusted publisher (probably next week) and see if we can make a release 🥳

EwoutH · 2024-09-11T11:03:25Z

Wow it got merged! Congratulations!!

erwanp · 2024-09-13T09:17:36Z

Well done @ddelange @maartenbreddels @EwoutH ; that's a great news for our codes!

HajimeKawahara · 2024-09-13T09:30:45Z

Great news! Thanks a lot. @ddelange @maartenbreddels @EwoutH

franz101 · 2024-09-18T01:36:35Z

@Ben-Epstein wake up it's christmas

ddelange · 2024-09-18T06:05:13Z

fyi followup PR #2434 should land soon, so release is imminent:)

ddelange mentioned this pull request Jan 21, 2023

Maximize wheel build parallellization ddelange/vaex#1

Closed

ddelange commented Jan 23, 2023

View reviewed changes

ddelange force-pushed the build-matrix branch from beff33a to 7bf26cc Compare February 27, 2023 07:15

This was referenced Jun 30, 2023

[FEATURE-REQUEST] Support Python 3.11.2 #2368

Closed

[BUG-REPORT] Failed to build vaex-core (pybind11 issue?) #2366

Closed

ddelange changed the title ~~Build aarch64 wheels~~ Build aarch64 wheels and support python 3.11 Jul 10, 2023

maartenbreddels force-pushed the master branch from 8d44942 to d4bf017 Compare July 11, 2023 13:55

CarrieFilion mentioned this pull request Aug 2, 2023

Install with Python 3.11 fails athob/py-ananke#1

Closed

ddelange referenced this pull request in rapidfuzz/RapidFuzz Aug 10, 2023

upgrade cython

dfea552

ddelange mentioned this pull request Aug 30, 2023

Discussion: support bundling libmagic ahupp/python-magic#233

Closed

ddelange force-pushed the build-matrix branch 3 times, most recently from 5680eb9 to 2136629 Compare September 4, 2023 08:28

Simplify

91263bc

Revert "Simplify"

3995302

This reverts commit 91263bc.

ddelange commented Aug 30, 2024

View reviewed changes

.github/workflows/wheel.yml Outdated Show resolved Hide resolved

ddelange added 2 commits August 30, 2024 20:33

Add back tags event trigger for core wheel upload

bac3efb

Switch to trusted publishing

93ce2c0

ddelange commented Aug 30, 2024

View reviewed changes

ddelange added 5 commits August 30, 2024 21:10

Add trusted publishing for vaex-core

f6c8b79

Exclude windows 3.8 from pythonpackage.yml

7988bb4

Don't upload vaex-core source distribution

b23f5a7

Disable cp38-win vaex-core wheel build

b594d77

Add permission

faaf23e

This was referenced Sep 9, 2024

[BUG-REPORT] PydanticImportError: BaseSettings has been moved #2384

Closed

vaex does not work with numpy==2.0.0 #2425

Open

maartenbreddels merged commit e3e1842 into vaexio:master Sep 11, 2024
51 checks passed

ddelange mentioned this pull request Sep 11, 2024

Fix pythonpackage.yml failing on macos-13 #2432

Merged

erwanp mentioned this pull request Sep 13, 2024

support for python 3.11 HajimeKawahara/exojax#502

Closed

minouHub mentioned this pull request Sep 18, 2024

Vaex compatible up to python<=3.12 radis/radis#698

Draft

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support CPython 3.11, 3.12, and aarch64 processors #2331

Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange commented Jan 20, 2023 •

edited

Loading

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 •

edited

Loading

ddelange Feb 27, 2023

ddelange Jan 23, 2023

ddelange Aug 17, 2023

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 •

edited

Loading

maartenbreddels commented Aug 30, 2024

maartenbreddels commented Aug 30, 2024

ddelange commented Aug 30, 2024

maartenbreddels commented Aug 30, 2024 •

edited

Loading

ddelange commented Aug 30, 2024

ddelange commented Aug 30, 2024 •

edited

Loading

ddelange Aug 30, 2024 •

edited

Loading

ddelange commented Aug 31, 2024

maartenbreddels commented Sep 5, 2024

EwoutH commented Sep 11, 2024

erwanp commented Sep 13, 2024

HajimeKawahara commented Sep 13, 2024

franz101 commented Sep 18, 2024

ddelange commented Sep 18, 2024

Support CPython 3.11, 3.12, and aarch64 processors #2331

Support CPython 3.11, 3.12, and aarch64 processors #2331

Conversation

ddelange commented Jan 20, 2023 • edited Loading

the wheels from this PR can be installed with:

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 • edited Loading

Choose a reason for hiding this comment

ddelange Feb 27, 2023

Choose a reason for hiding this comment

ddelange Jan 23, 2023

Choose a reason for hiding this comment

ddelange Aug 17, 2023

Choose a reason for hiding this comment

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 • edited Loading

maartenbreddels commented Aug 30, 2024

maartenbreddels commented Aug 30, 2024

ddelange commented Aug 30, 2024

maartenbreddels commented Aug 30, 2024 • edited Loading

ddelange commented Aug 30, 2024

ddelange commented Aug 30, 2024 • edited Loading

ddelange Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

ddelange commented Aug 31, 2024

maartenbreddels commented Sep 5, 2024

EwoutH commented Sep 11, 2024

erwanp commented Sep 13, 2024

HajimeKawahara commented Sep 13, 2024

franz101 commented Sep 18, 2024

ddelange commented Sep 18, 2024

ddelange commented Jan 20, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange Jan 23, 2023 •

edited

Loading

ddelange commented Sep 1, 2023 •

edited

Loading

maartenbreddels commented Aug 30, 2024 •

edited

Loading

ddelange commented Aug 30, 2024 •

edited

Loading

ddelange Aug 30, 2024 •

edited

Loading