Add wheels for arm64 macos #1534
Conversation
glandium
force-pushed
the
arm64-macos
branch
7 times, most recently
from
d6d95e0
to
ea27194
Compare
|
The failure on Android tests don't seem related. |
Yeah, ignore those. Will need to look into them separately. |
All versions of python >= 3.2 are compatible with abi3. cp36 indicates compat with >= 3.6. The wheels module used to create wheels with py3_none on Windows, but that was, as far as I know, a bug, and wheels with cp3*_abi3 work just fine.
This avoids special-casing in the CI configuration, allowing to build locally with less extra manual steps.
The rust host triple is its default target. Instead of relying on the python platform, we now rely on that host triple to make decisions, which is more likely to fit what we really want. And now that we know the default target for the rust compiler, we can also always explicitly pass it, such that the shared library path is consistently under target/$target, which simplifies things. At the same time, we stop assuming "linux" and "darwin" means x86_64.
This allows a more general way to use a custom target, and use that for Windows wheels on CI.
The python wheel says 11.0 for arm64 and 10.7 otherwise, but the crate has dependencies that build C code that could pull symbols that aren't available in older version of macOS, because of building against a newer SDK. Although practically speaking, it was probably not happening. Technically, though, rust code could be doing the same and that wouldn't be detected.
|
I just realized there was temporary garbage in the history. Removed them. |
There was a problem hiding this comment.
I like it! The part for arm64 macos looks good.
Moving around where and how we build the wheel looks fine as well.
As long as from-source-and-build-it continues to work, we can land this.
Defering to @mdboom for a final look, because he's the Python expert here.
There was a problem hiding this comment.
Thanks for this. Generally a nice cleanup of the setup.py, especially the standard way to cross compile anything.
Before merging, we should confirm that:
- The new Windows wheels produced by this work with the different pattern matching values
- Source installation through a non-wheel-provided platform (e.g. OpenBSD) still works.
glean-core/python/setup.py
Outdated
| return super().get_tag() | ||
| _, __, plat_name = super().get_tag() | ||
|
|
||
| return ("cp36", "abi3", plat_name) |
There was a problem hiding this comment.
Have you confirmed this works with Windows wheels? The pattern matching by pip is slightly different there, hence the weird hard coding of all of the supported options that was here before. (Related, let's update the comment above this function if this new approach does work).
There was a problem hiding this comment.
It worked for me locally, but it also obviously works more generally, since Windows python tests on CI use the wheels.
There was a problem hiding this comment.
Note the comment above get_tag is still relevant as-is.
There was a problem hiding this comment.
I just reminded myself of the details of the issue (which we should probably put in a comment so it isn't lost again).
Windows Python prior to 3.7 does not have a stable ABI, so therefore doesn't support the abi3 tag. Therefore, it won't "match" that wheel and it falls back to installing from source. You can test this locally by putting the wheels in a local directory and forcing pip to look there instead of PyPI with the -f directory option. It won't work with the wheel named as this PR does it.
Of course, we don't actually use the Python3 ABI since we use CFFI, so the best way to make a wheel that will work on Windows Python 3.6 and up is to mark it with py3-none rather than cp36-abi3.
There was a problem hiding this comment.
I'm not sure this is the full story. Because while installing a fresh Windows Python 3.6 and running pip install glean_sdk-35.0.0-cp36-abi3-win_amd64.whl doesn't work, pip tells you it's version 18.1 and that you should upgrade to 21.0.1, and after doing that, it works. Trying a few versions, it started working with pip 20.0.1.
There was a problem hiding this comment.
Installing directly from a file path always works. What doesn't is the pattern matching engine. So you need to do:
pip install -f directory_containing_wheels glean_sdk
to match what would happen in normal situations when installing from pypi.
This doesn't work for me on Python 3.6, even when upgrading pip.
There was a problem hiding this comment.
Installing directly from a file path always works
That's not my experience:
# ./AppData/Local/Programs/Python/Python36/python -m pip install glean_sdk-35.0.0-cp36-abi3-win_amd64.whl
glean_sdk-35.0.0-cp36-abi3-win_amd64.whl is not a supported wheel on this platform.
You are using pip version 18.1, however version 21.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
This doesn't work for me on Python 3.6, even when upgrading pip.
That doesn't look to be true (after upgrading to pip 21.0.1):
# ./AppData/Local/Programs/Python/Python36/python -m pip install -f . glean_sdk
Looking in links: .
Processing c:\users\glandium\glean_sdk-35.0.0-cp36-abi3-win_amd64.whl
(...)
There was a problem hiding this comment.
Thanks for that. I think I wasn't upgrading pip correctly.
So then I think the only thing to confirm is that we aren't held back to a particular version of pip by how things are vendored in m-c. Assuming that checks out, 👍 from me.
There was a problem hiding this comment.
Thank you for making me look. Things are grim, unfortunately, as the common way to install and upgrade moz-phab on Windows uses pip 19.3 :(
cp36-abi3 only works with pip >= 20, but we still have use-cases on pip 19.3.
|
Any idea yet when the next release is going to be? |
This week. I'll probably get one done today or tomorrow. |
|
Upload failed. Tracking here: https://bugzilla.mozilla.org/show_bug.cgi?id=1698860 We can manually upload that wheel when we figure out what platform tag we need. |
This also refactors wheels creation to allow cross-compilation in a smoother manner.