-
-
Notifications
You must be signed in to change notification settings - Fork 549
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use wider-compatible LLVM-OpenMP, bundle libgfortran
dylibs for macOS wheels
#4092
Use wider-compatible LLVM-OpenMP, bundle libgfortran
dylibs for macOS wheels
#4092
Conversation
Looks like matthew-brett/delocate#211 and matthew-brett/delocate#212 are some related threads. https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9096310836/job/25001433738 has some progress because the SuiteSparse and SUNDIALS dylibs are now exported correctly, but the build is still failing because we use gfortran and OpenMP headers from Homebrew, and Homebrew binaries are not necessarily good to publish Python wheels with. It is not recommended to downgrade the macOS runner version, since they shall be phased out soon... I'll look further into how we can replace these dependencies with other alternatives. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #4092 +/- ##
===========================================
- Coverage 99.56% 99.54% -0.02%
===========================================
Files 287 287
Lines 21744 21744
===========================================
- Hits 21649 21646 -3
- Misses 95 98 +3 ☔ View full report in Codecov by Sentry. |
Another solution is to set them to 12.0 and 14.0 as requested – however, this would break compatibility for a big proportion of users |
Yet another option: set up C and C++ compilers with the LLVM-OpenMP variant from https://scikit-learn.org/dev/developers/advanced_installation.html#macos-compilers-from-conda-forge rather than using Homebrew Point of breakage is pypa/cibuildwheel#1766, but this was just a documentation update, so our builds might have been incompatible silently: matthew-brett/delocate#198 is where all dylibs are being checked for compatibility, and matthew-brett/delocate#214 reveals that this was previously undetected across projects, so in any case we will need to migrate to using |
See also: Homebrew/legacy-homebrew#48437 |
edc970b
to
0c8a929
Compare
MACOSX_DEPLOYMENT_TARGET
when building SuiteSparse and SUNDIALS710189a
to
3470a82
Compare
3470a82
to
77e33e2
Compare
I was able to handle OpenMP, the other remaining task is to handle |
1c29827
to
2affb9d
Compare
Almost all the way, here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9116260227/job/25064471991 CasADi wheels are not repaired properly and it has a higher macOS minimum version requirement than expected – I'll simply rename the wheels to get the builds passing, since we cannot do much about it at this time. We're going to lose out on compatibility on macOS versions prior to 11.1 silently – let's hope we don't receive too many bug reports about it. |
6f0e995
to
3f82289
Compare
lilbgfortran
dylibs for macOS wheels
4aec23a
to
18dcbbd
Compare
lilbgfortran
dylibs for macOS wheelslibgfortran
dylibs for macOS wheels
Wheel builds are finally passing (🎉): https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9117439816 I can open this up for review now, cc: @kratman and @martinjrobins. The Codacy failures are irrelevant for the most part since we need to call a few subprocesses, I don't seem to remember how one can ignore those. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@agriyakhetarpal, is there anything still left in this one? Looks pretty good to me.
Let know if you still plan to do something in this PR, happy to approve and merge if not.
I don't think there should be anything else. I checked the @arjxn-py, you can go ahead with your approval and merge this. If we receive any complaints from macOS arm64 users about the IDAKLU solver, we'll just ask them to upgrade from macOS X 1A.B to 11.1 or later, that should fix their issues (because the CasADi issue above will probably remain open for a while). cc: @BradyPlanden – this fixes the issue that was being discussed at the PyBOP 16/05/2024 developer meeting :) |
Also, cc: @cringeyburger @prady0t this PR, while this should be okay to merge for now, is one of the reasons why we should be working towards a mechanism for testing the built wheels this year ;) currently we just check if the IDAKLU solver can be imported correctly, but it can hide issues related to the solver at runtime—see the older issue #3783 for example—running the full test suite against the wheels (or a fraction of it) would be great |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @agriyakhetarpal 🎉
… for macOS wheels (pybamm-team#4092)" This reverts commit becb1a4.
…OS wheels (pybamm-team#4092) * Set `MACOSX_DEPLOYMENT_TARGET` for prerequisites * Remove previously set environment variables * Remove Homebrew libomp installations * Ignore `pybamm-requires` OpenMP downloads * Rework docs to add OpenMP instructions * Download LLVM-OpenMP on macOS * Remove unused compiler argument * Fix rpath for libomp dylib at wheel repair * Fix a failing link * Revert "Fix rpath for libomp dylib at wheel repair" This reverts commit 6fb8ef0. * Fix RPATHs for SuiteSparse and libomp * 11.1 for arm64 macOS and 11.0 for amd64 macOS * Set up Fortran compiler via GHA * Downgrade to gcc 12 * Try fix for macOS amd64 wheels first * Build on macos-13 image instead * Debug further, try building against 11.0 * And now build for macOS arm64 * Try building arm64 against 11.0 * Rename wheel from 11_1 to 11_0 * Cleanup and fix up commands * Rename wheel in dest dir instead * Cleanup plus references to SciPy license
… for macOS wheels (pybamm-team#4092)" (pybamm-team#4109) This reverts commit becb1a4.
Description
Closes #4091, binaries on macOS need to be built with OpenMP which is more compatible with older macOS versions, rather than Homebrew OpenMP, which is not built for use cases where one needs to set MACOSX_DEPLOYMENT_TARGET to a lower value than the version specifier of the host macOS machine, e.g., at the time of building wheels with
cibuildwheel
.This change is required due to stricter requirements across macOS version compatibility when repairing the wheel with tools like
delocate
.Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.
Key checklist:
$ pre-commit run
(or$ nox -s pre-commit
) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)$ python run-tests.py --all
(or$ nox -s tests
)$ python run-tests.py --doctest
(or$ nox -s doctests
)You can run integration tests, unit tests, and doctests together at once, using
$ python run-tests.py --quick
(or$ nox -s quick
).Further checks: