Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support arm64 macOS runner for GitHub Actions #1746

Closed
TonyXiang8787 opened this issue Jan 31, 2024 · 14 comments
Closed

Support arm64 macOS runner for GitHub Actions #1746

TonyXiang8787 opened this issue Jan 31, 2024 · 14 comments

Comments

@TonyXiang8787
Copy link

TonyXiang8787 commented Jan 31, 2024

Description

GitHub has announced that m1 macOS runners are available for open-source project. It would be a good idea to enable cibuildwheel to use m1 macOS runner to build and test x64, arm64 and universal2 wheel files.

https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

Or maybe does it just work already out of the box? In the official readme it says not supported.

CI config

Config to use runs-on: macos-14 in the GitHub Actions job, the cibuildwheel should be able to build and test x64, arm64 and universal2 wheel files.

@perklet
Copy link

perklet commented Jan 31, 2024

I just tested for my project and it worked! The only problem I encountered is that cp38 is cumbersome due to some implementation details, use cp39 and above.

@njzjz
Copy link
Contributor

njzjz commented Jan 31, 2024

I have no problem with arm64. However, I don't know how to test universal2 for both architectures.

@James-E-A
Copy link

James-E-A commented Jan 31, 2024

I believe I'm currently building x86_64, arm64, and universal2 for Mac with no issue.

It was literally 1 line of code, setting env var CIBW_ARCH=all. (Admittedly, this is probably overbuilding; I don't know who actually needs the "fat binaries" when they could just get the AMD64 or ARM build as appropriate, but it's still looking nice...)

*The run I linked is overall "failed" due to issues with delvewheel and Windows cross-compilation, but ignore that; as you can see, the specific job for Mac was an unqualified success!

@nightlark
Copy link

nightlark commented Jan 31, 2024

It appears to work out of the box, at least on the cibuildwheel side. The only change I made is changing the OS for the macOS builds to use macos-14 in my GHA workflow, and then it built the x86_64, arm64, and universal2 wheels.

Even better, it ran the test-command defined in pyproject.toml for all the architectures in all the wheels. It used arch -x86_64 to test the x86_64 wheel, and the x86_64 portion of the universal2 wheel, and ran the tests for the universal2 wheel a second time to test the arm64 portion.

Here's the changes I had to make -- most of the issues were due to a lack of pipx in the new macos arm64 images, and trouble installing ninja as part of pyproject.toml/cibuildwheel: https://github.com/nightlark/swig-pypi/pull/117/files

@henryiii
Copy link
Contributor

Was pip install ninja not working? We make universal wheels and I've checked them once in the past. (I also maintain that)

Scikit-build-core automatically adds ninja as a dependency only if it's not present, by the way.

@henryiii
Copy link
Contributor

However, I don't know how to test universal2 for both architectures

Just don't skip the tests. cibuildwheel will test everything it can.

(We've had AS building on CirrusCI for a bit, so not surprised if it's working fairly well out of the box on GHA)

@henryiii
Copy link
Contributor

Though if you pin an old version, it might not have AS wheels:

https://github.com/nightlark/swig-pypi/blob/d7bd8e36f5b50f5e468ed37ba852facd4b001405/pyproject.toml#L10

Again, scikit-build-core does all this for you, also the py2.py3 tags are just a setting, etc.

@njzjz
Copy link
Contributor

njzjz commented Jan 31, 2024

I got an architecture error from NumPy when testing the arm64 part of universal2: (GitHub Actions log)

    File "/private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/cibw-run-sejyakil/cp39-macosx_universal2/venv-test/lib/python3.9/site-packages/numpy/core/overrides.py", line 8, in <module>
      from numpy.core._multiarray_umath import (
  ImportError: dlopen(/private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/cibw-run-sejyakil/cp39-macosx_universal2/venv-test/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): tried: '/private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/cibw-run-sejyakil/cp39-macosx_universal2/venv-test/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/cibw-run-sejyakil/cp39-macosx_universal2/venv-test/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (no such file), '/private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/cibw-run-sejyakil/cp39-macosx_universal2/venv-test/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

It seems that pip used the same virtual environment for x86_64 and arm64 when testing universal2. Pip first installed x86_64 NumPy. Then, when pip installed arm64 NumPy, it said Requirement already satisfied and left the x86_64 NumPy in the environment...

@nightlark
Copy link

Was pip install ninja not working? We make universal wheels and I've checked them once in the past. (I also maintain that)

Scikit-build-core automatically adds ninja as a dependency only if it's not present, by the way.

The pip(x) install ninja was working, but the location it installed to wasn’t in the macOS PATH so no ninja command was found. I tried pipx ensurepath in a few places, but the PATH changes didn’t seem to get propagated to the steps that actually needed ninja.

I think the issue you opened about pipx in the GHA runner images repo would be the “best” place to fix it.

@nightlark
Copy link

Though if you pin an old version, it might not have AS wheels:

https://github.com/nightlark/swig-pypi/blob/d7bd8e36f5b50f5e468ed37ba852facd4b001405/pyproject.toml#L10

Again, scikit-build-core does all this for you, also the py2.py3 tags are just a setting, etc.

ninja 1.10.2.2 has a wheel with arm64/universal2 tags. I have this issue in the backlog to try switching to scikit-build-core. It would be great to get rid of the setup.py file eventually.

@henryiii
Copy link
Contributor

henryiii commented Feb 1, 2024

Okay, happy as long as it's not an issue with the ninja package that I need to fix. :) Yeah, not having pipx is a huge downgrade.

It seems that pip used the same virtual environment for x86_64 and arm64 when testing universal2

We should probably investigate making two environments instead of one.

@webknjaz
Copy link
Member

(observation)
FTR, since several hours ago I started getting macOS 14 ARM64 in jobs where I have macos-latest set. I checked that cibuildwheel already uses this image in the CI: https://github.com/pypa/cibuildwheel/actions/runs/8785063316/job/24104740143#step:1:8.

@henryiii
Copy link
Contributor

Yes, they seem to be doing a staggered rollout of macos-latest pointing at macos-14, which means Intel jobs are being switched to ARM (and Python 3.10 is the oldest supported Python on those images via setup-python). Will surprise a lot of cibuildwheel users.

@webknjaz
Copy link
Member

@henryiii yep, it broke my pure-python pipelines even. So the effect is wider than just people building C-extensions...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants