Support CPython 3.11, 3.12, and aarch64 processors (#2331)

* fix[ml]: adjust tests to reflect latest apis of 3rd party libraries (xgboost, lightgbm) * Build wheels on pull_request * Maximize wheel build parallellization * git submodule update --remote --merge packages/vaex-core/vendor/pybind11 * Fix gcc error * Fix workflow syntax * Remove redefinition https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859 https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110 * Disable win32 https://github.com/ddelange/vaex/actions/runs/3965689146/jobs/6795667118#step:6:538 * Remove testing leftovers * Add upper cap on lightgbm microsoft/LightGBM#5196 (comment) * Migrate to mamba-org/setup-micromamba@v1 * Upload release assets on Github (pre)release * Add missing permission for release assets * Build cp312 wheels * Remove setuptools and wheel from pyproject.toml * Replace imp with importlib * chore: trigger ci * ci: upgrade xcode for brew install libomp * update requirements-ml to comply with the latest veex-ml expectations. * try to install lightgbm via pip * fix: only the mini taxi file is on s3 for cost savings * ci: pin dask<2024.2.0 to get the same hash keys This version gives different results, although not a problem in production (it will make your cache invalid though), for CI we test that we have stable keys (fingerprints) * fix: only the mini taxi file is on s3 for cost savings (2) * ci: skip notebooks that depend on the dataframe server * ci: skip ci steps in various old python versions * Bump micromamba and other actions * ci: specific test hangs on ci+osx * Bump cibuildwheel, use native arm64 mac runners * test: log is renamed to log loss * test: skip lightgbm tests on 36 and 37 * test: skip sklearn tests on 36 and 37 * Update packages/vaex-core/setup.py Co-authored-by: ddelange <14880945+ddelange@users.noreply.github.com> * chore: drop python 3.6 and 3.7 support Co-authored-by: Ewout ter Hoeven <E.M.terHoeven@student.tudelft.nl> * test: skip a failing test for windows * ci: macOS 11 is retired as of June 28 * ci: always build wheels, but do not publish instead * ci: try with older micromamba, 1.5.6 seems to sometimes hang * Install setuptools in wheel.yml * Add sudo * Fix windows * Use sudo only on macos * Fix empty string evaluating false * Add .readthedocs.yaml * Pull submodules * Try editable rtd install * Try move editable install to requirements_rtd.txt * Allow newer sphinx * Autocancel previous runs in PRs * Autocancel all jobs * Sphinx sidebar fix https://github.com/dmwyatt/sphinx-book-theme/blob/2416ef6cebc3d83019dbc83f25c61b72719cad55/docs/customize/sidebar-primary.md#default-sidebar-elements * Remove autocancel, doesn't work from forks * Add cancel-in-progress keys * Amend to only cancel on PRs * Add CIBW_TEST_COMMAND * Disallow numpy v2 * Skip musllinux due to misding blake3 wheels * Install carco for blake3 source install * Fix CIBW_ENVIRONMENT * Add CIBW_TEST_SKIP * Build wheels with numpy v1 * run tests on 3.10-3.12 * upgrade micromamba * unpin pytest-asyncio * try different pin as <2.0 because mamba crashes on that * Fix dephell and CI hell (ddelange#3) * allow for multiple fingerprints related to package versions * explicit dtype=object * fix pandas issues * revert graphene upgrade * force uv to ignore Werkzeug (we do not use it, its a subdependency which has conflicts) * fix: work with modern scipy/sparse arrays * skip test for python 3.8 * skip test on windows * add missing import * skip some tests conditionally * tuple comparison fix * does osx run without sourcing .bash profile? * skip a few more tests * only run test on linux * Try building arm64 wheels on macos-13 * Switch to older Xcode * Fix readthedocs by manually unpinning werkzeug in the graphene-tornado v2.6.1 release # for now we leave graphene-tornado / graphene on v2 # ref #2356 (comment) # in top level vaex[ci] we added the following (additional) direct reference requirement which adds one commit (ddelange/graphene-tornado@d75f01f) on top of the 2.6.1 release to unpin werkzeug # "graphene-tornado @ https://github.com/ddelange/graphene-tornado/archive/refs/heads/2.6.1.unpin-werkzeug.zip", * Simplify * Revert "Simplify" This reverts commit 91263bc. * Add back tags event trigger for core wheel upload * Switch to trusted publishing * Add trusted publishing for vaex-core * Exclude windows 3.8 from pythonpackage.yml * Don't upload vaex-core source distribution * Disable cp38-win vaex-core wheel build * Add permission --------- Co-authored-by: Jovan Veljanoski <jovan.veljanoski@tiqets.com> Co-authored-by: Maarten A. Breddels <maartenbreddels@gmail.com> Co-authored-by: Ewout ter Hoeven <E.M.terHoeven@student.tudelft.nl>
vaexio · Sep 11, 2024 · e3e1842 · e3e1842
1 parent 15245cf
commit e3e1842
Show file tree

Hide file tree

Showing 90 changed files with 604 additions and 1,563 deletions.
diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml
@@ -5,130 +5,74 @@ on:
     branches:
       - master
   pull_request:
-    branches:
-    - master
-    - docs
   schedule:
   - cron: "0 2 * * 1-5"
 
-# to make bash use ~/.bashrc, see https://github.com/marketplace/actions/setup-miniconda#example-5-custom-installer
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 defaults:
   run:
-    shell: bash -l {0}
-
-concurrency:
-  group: ${{ github.ref }}
-  cancel-in-progress: true
+    shell: bash
 
 jobs:
   main:
-    env:
-      CACHE_NUMBER_COMPILE: 6
-      MAMBA_ROOT_PREFIX: /dev/shm/mambaroot
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
-      max-parallel: 12
       matrix:
-        os: [ubuntu-latest, windows-latest, macOS-11]
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        os: [ubuntu-latest, windows-latest, macos-13]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        exclude:
+          - {os: windows-latest, python-version: "3.8"}
 
     steps:
-    - name: disk space
-      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
-      run: |
-        df -h
-    - uses: maxim-lobanov/setup-xcode@v1
-      # alternative would be to upgrade tapi to 1100.0.11, we can possibly remove this in the future
-      if: matrix.os == 'macOS-11'
-      with:
-        xcode-version: "11.7.0"
     - uses: actions/checkout@v2
       with:
         submodules: true
-    - uses: actions/checkout@v2
-      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
-      with:
-        repository: vaexio/vaex-enterprise
-        token: ${{ secrets.PAT_PULL_ENTERPRISE }}
-        path: vaex-enterprise
-    - name: install micromamba
-      uses: mamba-org/provision-with-micromamba@main
+
+    - uses: actions/setup-python@v5
       with:
-        environment-name: vaex-dev
-        environment-file: ci/conda-env.yml
-        extra-specs: |
-            python=${{ matrix.python-version }}
-    - name: Extra non-windows installs
+        python-version: ${{ matrix.python-version }}
+
+    - run: python -m pip install --upgrade pip setuptools wheel uv
+
+    # - uses: actions/checkout@v2
+    #   if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
+    #   with:
+    #     repository: vaexio/vaex-enterprise
+    #     token: ${{ secrets.PAT_PULL_ENTERPRISE }}
+    #     path: vaex-enterprise
+
+    - name: Setup Graphviz
+      uses: ts-graphviz/setup-graphviz@v2
+
+    - name: Setup PCRE (Nix-only)
       if: matrix.os != 'windows-latest'
-      run: |
-        echo "not needed for now"
-    # - name: Extra windows installs
-    #   if: matrix.os == 'windows-latest'
-    #   run: |
-    #     # not sure why, but the conda env is strange (old dask) if we put this constraint in the conda solver
-    #     # also might solve scipy issues
-    #     micromamba update scipy ipython dask
-    #     micromamba install scipy=1.7.1
-    # - name: free disk space
-    #   if: matrix.os == 'ubuntu-latest'
-    #   run: |
-    #     sudo swapoff -a
-    #     sudo rm -f /swapfile
-    #     sudo apt clean
-    #     docker rmi $(docker image ls -aq)
-    #     df -h
+      run: sudo -E bash bin/install_pcre.sh
+
     - name: Install OpenMP runtime (Mac-only)
-      if: matrix.os == 'macOS-11'
-      run: |
-        brew install libomp
-    - name: Cache compiled binaries
-      # this fails for this combination, leading to binaries filled with 0's
-      if: matrix.python-version != '3.6' || matrix.os != 'macOS-11'
-      id: cache-compiled-binaries
-      uses: actions/cache@v2
-      with:
-        path: |
-          packages/vaex-core/build/lib*/vaex/vaexfast*
-          packages/vaex-core/build/lib*/vaex/super*
-        key: ${{ runner.OS }}-${{ matrix.python-version }}-${{ hashFiles('packages/vaex-core/src/*') }}-${{ env.CACHE_NUMBER_COMPILE }}
-    - name: Fix cache timestamp
-      run: |
-        touch packages/vaex-core/build/lib*/vaex/vaexfast* || echo "ok to fail"
-        touch packages/vaex-core/build/lib*/vaex/super* || echo "ok to fail"
-        ls -alh packages/vaex-core/build/lib*/vaex/ || echo "ok to fail"
+      if: startswith(matrix.os, 'macos')
+      run: brew install libomp
+
+    - name: Copy dll (Windows-only)
+      if: (matrix.os == 'windows-latest')
+      uses: ./ci/actions/windll
+
     - name: Install vaex
-      run: |
-        pip install myst_parser
-        pip install -e . -v
-        # ./ci/03-install-vaex.sh micromamba
+      run: uv pip install -v --system .[ci]
+
+    - run: uv pip freeze
+
 #    - name: Install vaex-enterprise
 #      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
 #      run: |
 #        cd vaex-enterprise
 #        micromamba install -c conda-forge distributed
 #        pip install ray
 #        pip install -e .
-    - name: Copy dll
-      if: (matrix.os == 'windows-latest')
-      uses: ./ci/actions/windll
-    - name: Build Python packages
-      run: |
-        mkdir dist
-        (cd packages/vaex-core && python setup.py bdist_wheel); cp packages/vaex-core/dist/* dist
-        (cd packages/vaex-graphql && python setup.py bdist_wheel); cp packages/vaex-graphql/dist/* dist
-        (cd packages/vaex-jupyter && python setup.py bdist_wheel); cp packages/vaex-jupyter/dist/* dist
-        (cd packages/vaex-ml && python setup.py bdist_wheel); cp packages/vaex-ml/dist/* dist
-        (cd packages/vaex-viz && python setup.py bdist_wheel); cp packages/vaex-viz/dist/* dist
-        (cd packages/vaex-astro && python setup.py bdist_wheel); cp packages/vaex-astro/dist/* dist
-        (cd packages/vaex-hdf5 && python setup.py bdist_wheel); cp packages/vaex-hdf5/dist/* dist
-        (cd packages/vaex-server && python setup.py bdist_wheel); cp packages/vaex-server/dist/* dist
-        (cd packages/vaex-meta && python setup.py bdist_wheel); cp packages/vaex-server/dist/* dist
-    - name: Upload builds
-      uses: actions/upload-artifact@v2
-      with:
-        name: dist-${{ github.run_number }}-${{ matrix.python-version }}-${{ matrix.os }}
-        path: ./dist
+
     # - name: Lint with flake8
     #   run: |
     #     echo "soon"
@@ -137,123 +81,56 @@ jobs:
     #     # flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
     #     # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
     #     # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Install fixes for Python 3.6
-      if: matrix.python-version == '3.6'
-      run: |
-        # by default on 3.6 we get an old version, so manually upgrade
-        pip install gcsfs==0.8.0
-    - name: Install pytest-asyncio correct version
-      if: matrix.python-version != '3.10'
-      run: |
-        # 3.10 doesn't seem to work
-        pip install "pytest-asyncio<0.14"
+
     - name: Test with pytest
       run: |
         ./ci/04-run-test-suite.sh
+
     - name: Test with pytest (with cache on)
-      if: matrix.python-version == '3.9' && matrix.os == 'ubuntu-latest'
+      if: matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest'
       run: |
         VAEX_CACHE_RESULTS=1 ./ci/04-run-test-suite.sh
+
     - name: Check ml spec
-      # no catboost for py39 and py37
-      if: matrix.python-version != '3.9' && matrix.os != 'windows-latest' && matrix.python-version != '3.7'
       run: |
         python -m vaex.ml.spec packages/vaex-ml/vaex/ml/spec_new.json
         diff packages/vaex-ml/vaex/ml/spec_new.json packages/vaex-ml/vaex/ml/spec.json
+
 #    - name: Test with pytest (vaex-enterprise)
 #      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
 #      run: |
 #        py.test vaex-enterprise/tests --timeout=1000
+
     - name: Test notebooks
-      if: matrix.os != 'windows-latest' && matrix.python-version != '3.6'
-      run: |
-        ./ci/05-run-notebooks.sh
+      if: matrix.os != 'windows-latest'
+      run: ./ci/05-run-notebooks.sh
+
     - name: Authenticate Google Cloud Platform
-      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest') || (matrix.os == 'macOS-11' && matrix.python-version == '3.6'))  }}
+      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest'))  }}
       uses: google-github-actions/setup-gcloud@v0
       with:
         project_id: ${{ secrets.GCP_PROJECT_ID_VAEX }}
         service_account_key: ${{ secrets.GCP_SA_KEY_VAEX }}
         export_default_credentials: true
+
     - name: Test vaex-contrib
-      # do not run in a PR from someone else, skip windows, and osx+py36
-      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest') || (matrix.os == 'macOS-11' && matrix.python-version == '3.6'))  }}
+      # do not run in a PR from someone else, skip windows
+      if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os != 'windows-latest')  }}
       env:
         PROJECT_ID: ${{ secrets.GCP_PROJECT_ID_VAEX }}
-      run: |
-        pip install -e 'packages/vaex-contrib[gcp]'
-        ./ci/06-run-contrib-tests.sh
-
-  install:
-    runs-on: ${{ matrix.os }}
-    needs: [main]
-
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, windows-latest, macOS-11]
-        python-version: [3.7, 3.8, 3.9]
-        # ssl/certifi issues with this combination
-        exclude:
-          - os: windows-latest
-            python-version: 3.7
-
-    steps:
-    - uses: actions/checkout@v2
-    - uses: actions/download-artifact@v2
-      with:
-        name: dist-${{ github.run_number }}-${{ matrix.python-version }}-${{ matrix.os }}
-        path: ./dist
-
-    - name: Install micromamba
-      uses: mamba-org/provision-with-micromamba@main
-      with:
-        environment-name: vaex-test
-        environment-file: ci/conda-base-minimal.yml
-        extra-specs: |
-            python=${{ matrix.python-version }}
-            pcre
-            rich
-    # it shouldn't be needed to use pcre, maybe we should make the wheels similar to how we make distributions
-    - name: Cache pip
-      uses: actions/cache@v2
-      env:
-        # Increase this value to reset cache
-        CACHE_NUMBER: 0
-      with:
-        path: |
-          ~/.cache/pip
-        key:
-          pip-${{ runner.os }}-${{ matrix.python-version }}-${{ env.CACHE_NUMBER }}
-
-    - name: Install core packages
-      run: |
-        pip install -vv dist/vaex*core* dist/vaex*hdf5*
-
-    - name: Test core import
-      run: |
-        python -c "import vaex; df = vaex.example()"
-
-    - name: Install all packages
-      run: |
-        pip install -vv dist/*
-
-    - name: Test import
-      run: |
-        python -c "import vaex; df = vaex.example()"
+      run: ./ci/06-run-contrib-tests.sh
 
     - name: Test comand line
       run: |
         vaex convert ~/.vaex/data/helmi-dezeeuw-2000-FeH-v2-10percent.hdf5 test.parquet
-        pip install rich
         VAEX_PROGRESS_TYPE=rich vaex convert ~/.vaex/data/helmi-dezeeuw-2000-FeH-v2-10percent.hdf5 test.parquet
         vaex settings yaml
         vaex settings md
 
     - name: Test server
       if: matrix.os != 'windows-latest'
       run: |
-        vaex server --add-example --port 9999&
+        vaex server --add-example --port 9999 &
 
     - name: Wait for Vaex server
       if: matrix.os != 'windows-latest'