-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Universal2 wheels on macOS #484
Conversation
Hmm, due to actions/runner-images#2247 this is gonna be slow going... it might be better to start work on this next week... |
We will have to download the new images for Python 3.9 specifically I think the builds are a bit slower, but still faster than Travis! |
Do the 10.6 32/64 fat binaries not work on macOS 11? Is that perhaps the issue with 3.5? 2.7 does seem to be working, but not 3.5, if I read the log correctly? Possibly it's the workaround we have in place for SSL? Just a thought, how about we drop 3.5 on Windows and macOS, but keep it on Linux for a little while longer (April 2021, EOL for Ubuntu 16.04, or whenever manylinux wheels drop it - ideally, I think we should match exactly what they are doing on Linux) |
https://discourse.jupyter.org/t/jupyter-notebook-error-dyld-library-not-loaded-corefoundation-after-macos-big-sur-upgrade/6838 So it looks like the bigger problem with all not patched python versions.. |
I'm not sure. It's not affecting 2.7, and Big Sur comes with 3.8.2 (CLT), so I think it's worth checking to see if 3.6 builds, anyway - maybe just reverse the order and do 3.9 first, then 3.8, etc. :) It might be that the Universal2 3.9 wheel will need to be in a separate job for now, anyway - but that shouldn't stop us from working out the 11.0 issues. @joerick let me know if / when it's okay to push to this branch. It's better if you do it for now, as I have to prepare to record some CMake training videos, but I'd like to help this along when you aren't working on it. |
Hm, I wonder if you're talking about native ARM support, or the ability to build universal2 wheels? I was under the impression that I'll disable the 3.5 macOS build for now. Feel free to hack on this branch @henryiii, it just takes about half an hour for each macOS build to queue so it's not a lot of fun :) |
PyPy is not macOS 11 ready either? At least all the normal wheels built. Though are they univesal2? Don't see a difference in naming. |
Yeah, maybe you're right that we need the new Python macos11 builds |
In the current release, of the wheel depends on So without changing the installer platform tag will be x86. As I do not understand, there is no plan for changing this behavior. |
49dd0b7
to
5b6a2bd
Compare
5b6a2bd
to
609c0f3
Compare
Looks like that did the trick. :) How do we want to switch to using the new download? Always, with a flag, etc? Can you build an Intel (or Apple Silicon) wheel from the universal Python binary? If so, I don't see a reason to not always download it? Also, PyPy and Python 3.5 don't support macOS 11, but I think that's okay - we could either recommend they be skipped for now until 11.0 becomes the default for macOS-latest on one of the CI systems, and by then, hopefully PyPy will have fixed their issues. @mattip is this a known issue with PyPy? Also will it be able to make Universal2 wheels too eventually? I feel like I have seen this in homebrew too for the 11.0 (but could be mixing it with Apple Silicon support - that I know is missing currently for PyPy). GitHub is pretty fast, so maybe macOS-latest will change sooner than I think, but I bet we have some time. |
More wheel related, if you need a really new pip to use a universal2 wheel, I expect most projects will provide both for the near future - in which case, it seems much more sensible to provide a Intel and an Apple Silicon wheel instead of an Intel and a Universal2 wheel. So the question remains, can you set a target? |
Also, can we silence all the silly warnings from pip? The Python 2 warning, the newer version available warning? We pin pip, so these warnings just get in the way when trying to find the real error. |
And, final message tonight - do we have any 10.x builds left? Azure and GHA ran on 11, and I can't see CircleCI, so can't tell if this still works on 10.15 or if using the new Python broke it. |
Is that this issue with Edit: this issue |
I do not know the method to do this. As far as I know, there is a need to check the Xcode version. Maybe @ronaldoussoren will know more. Maybe the best workaround will be updated delocate. |
perhaps you quoted the wrong text there, @Czaki ?
If we can be selective enough about the warnings, yes please :)
Wait, you can't see the CircleCI logs? I thought it was open-source. I do have to login with Github, maybe you need to do that? To answer your question, Circle's mac builds are still on xcode:9.4.1, macOS 10.13. Which makes this test failure all the more interesting:
To me, that looks like it produced a universal2 wheel, even though it was built on xcode9 which almost certainly doesn't support Apple Silicon! Wat.
Did a little searching though the CPython source and found If that works, we might as well select the universal2 Python distribution where it's available. |
@joerick I think I quote too long text. I comment mainly:
|
Okay, this is ready for review! The only think that will probably change is the removal of the wheel rename hack - judging by pypa/pip#9533 it looks like pip will soon support universal2 on arm64, and then we can remove that too. Thank you @henryiii for pushing that forward! |
Let's bump our pins and dump the hack!!! |
I can't actually run |
Ah, it does not want to make a PR on a PR. Hey ho. I'll do it on master instead and merge that in. |
You could let it make it on master then redirect it. But not really any different. |
Actually, you can just cherry pick the change here and keep going. :) |
@@ -180,6 +238,41 @@ def setup_python(python_configuration: PythonConfiguration, | |||
# https://github.com/python/cpython/blob/a5ed2fe0eedefa1649aa93ee74a0bafc8e628a10/Lib/_osx_support.py#L260 | |||
env.setdefault('ARCHFLAGS', '-arch x86_64') | |||
|
|||
if python_configuration.version == '3.9': |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once we have access to packaging from #536 , then we should do a >=
3.9 here so we don't have to change it when we add 3.10. But this is fine until then.
|
||
# Different versions of Xcode contain different SDK versions... | ||
# we're happy with anything newer than macOS 11.0 | ||
arm64_compatible_sdks = [s for s in sdks if not s.startswith('macosx10.')] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we sort and get the newest one? Not a change for this PR, but selecting the latest might be better than selecting randomly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From what I can see on https://xcodereleases.com/ , each version of Xcode only ships with one macOS SDK version. I couldn't find a case where there are multiple. But I wasn't 100% sure, hence the use of lists here.
Also not sure what the selection criteria would be... would we want the newest , or the oldest? I tend to find that binaries are more often forward-compatible than backward-compatible. So then we'd want the oldest? ¯\_(ツ)_/¯
Wow. I haven't really been following (partially because I didn't feel like I wanted more macOS issues in my life) this intensively, but great job, everyone! :-) |
Aaaand it's in! Thanks everyone for the input, especially @henryiii , @Czaki , @ronaldoussoren and @mayeut ! |
Thanks very much to everybody involved in making this happen. |
Add support for building
arm64
anduniversal2
wheels on macOS 11+.CIBW_ARCHS
Closes #473.