Use wider-compatible LLVM-OpenMP, bundle libgfortran dylibs for macOS wheels

[agriyakhetarpal]

Description

Closes #4091, binaries on macOS need to be built with OpenMP which is more compatible with older macOS versions, rather than Homebrew OpenMP, which is not built for use cases where one needs to set MACOSX_DEPLOYMENT_TARGET to a lower value than the version specifier of the host macOS machine, e.g., at the time of building wheels with cibuildwheel.

This change is required due to stricter requirements across macOS version compatibility when repairing the wheel with tools like delocate.

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python run-tests.py --all (or $ nox -s tests)
• The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[agriyakhetarpal]

Looks like matthew-brett/delocate#211 and matthew-brett/delocate#212 are some related threads.

https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9096310836/job/25001433738 has some progress because the SuiteSparse and SUNDIALS dylibs are now exported correctly, but the build is still failing because we use gfortran and OpenMP headers from Homebrew, and Homebrew binaries are not necessarily good to publish Python wheels with. It is not recommended to downgrade the macOS runner version, since they shall be phased out soon... I'll look further into how we can replace these dependencies with other alternatives.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.54%. Comparing base (33d9ea7) to head (11015cb).

❗ Current head 11015cb differs from pull request most recent head 30a76b9

Please upload reports for the commit 30a76b9 to get more accurate results.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4092      +/-   ##
===========================================
- Coverage    99.56%   99.54%   -0.02%     
===========================================
  Files          287      287              
  Lines        21744    21744              
===========================================
- Hits         21649    21646       -3     
- Misses          95       98       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[agriyakhetarpal]

Another solution is to set them to 12.0 and 14.0 as requested – however, this would break compatibility for a big proportion of users

[agriyakhetarpal]

Yet another option: set up C and C++ compilers with the LLVM-OpenMP variant from https://scikit-learn.org/dev/developers/advanced_installation.html#macos-compilers-from-conda-forge rather than using Homebrew

Point of breakage is pypa/cibuildwheel#1766, but this was just a documentation update, so our builds might have been incompatible silently: matthew-brett/delocate#198 is where all dylibs are being checked for compatibility, and matthew-brett/delocate#214 reveals that this was previously undetected across projects, so in any case we will need to migrate to using gcc which provides libgomp headers or use a different installation of libomp (LLVM variant) that has greater compatibility across macOS versions.

[agriyakhetarpal]

See also: Homebrew/legacy-homebrew#48437

[agriyakhetarpal]

I was able to handle OpenMP, the other remaining task is to handle gfortran dynamic libraries. We are still getting them from Homebrew, so they have to either be replaced with a different installation. Using gfortran and gcc as compilers doesn't bring issues, but we have a tricky dependency from suitesparseconfig.dylib on libgfortran.dylib which is why the latter gets bundled into our wheels, and therefore breaks compatibility... A way out could be to load at runtime instead while excluding it from the wheels, but this should be used as a last resort. The macos-11 images have been deprecated and will be removed in Q2 2024 (this quarter), so we can't use those, and we will need to migrate from macos-12 to macos-13 sometime soon.

[agriyakhetarpal]

Almost all the way, here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9116260227/job/25064471991

CasADi wheels are not repaired properly and it has a higher macOS minimum version requirement than expected – libcasadi.3.7.dylib requires 11.1, so I have opened an issue on their repository: casadi/casadi#3698

I'll simply rename the wheels to get the builds passing, since we cannot do much about it at this time. We're going to lose out on compatibility on macOS versions prior to 11.1 silently – let's hope we don't receive too many bug reports about it.

[agriyakhetarpal]

Wheel builds are finally passing (🎉): https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9117439816

I can open this up for review now, cc: @kratman and @martinjrobins. The Codacy failures are irrelevant for the most part since we need to call a few subprocesses, I don't seem to remember how one can ignore those.

[arjxn-py]

@agriyakhetarpal, is there anything still left in this one? Looks pretty good to me.
Let know if you still plan to do something in this PR, happy to approve and merge if not.

[agriyakhetarpal]

I don't think there should be anything else. I checked the --force and --install-dir posargs combinations for pybamm-requires as well locally – everything seems to install in the right place and work as desired. Until XCode's Clang adds support for the OpenMP specification someday in a later macOS version, this piece of code can stay for now. I would note that the same bundling process for libgfortran is present for the NumPy wheels as well, besides SciPy.

@arjxn-py, you can go ahead with your approval and merge this. If we receive any complaints from macOS arm64 users about the IDAKLU solver, we'll just ask them to upgrade from macOS X 1A.B to 11.1 or later, that should fix their issues (because the CasADi issue above will probably remain open for a while).

cc: @BradyPlanden – this fixes the issue that was being discussed at the PyBOP 16/05/2024 developer meeting :)

[agriyakhetarpal]

Also, cc: @cringeyburger @prady0t this PR, while this should be okay to merge for now, is one of the reasons why we should be working towards a mechanism for testing the built wheels this year ;) currently we just check if the IDAKLU solver can be imported correctly, but it can hide issues related to the solver at runtime—see the older issue #3783 for example—running the full test suite against the wheels (or a fraction of it) would be great

[arjxn-py]

Thanks @agriyakhetarpal 🎉