Suggestion: provide universal binaries for macOS

[dtcooper]

Hi there,

Wondering if would be a good idea to package this up as a universal binary for both x86_64 and aarch64 to support Intel Macs and Apple M1/M2 chips together in one fell swoop. Cheers and thanks!

David

[indygreg]

This is a feature that's long been on my mind as well.

Random considerations...

Today, we target macOS 10.9 on x86-64 and 11.0 on aarch64 since we build each arch independently. This gives us maximum binary portability. If you pass -arch to the compiler multiple times to produce a universal binary natively, I believe you have to use the same Apple SDK and targeting settings for each architecture. Since you need the macOS 11 SDK to target aarch64, I believe this means the x86-64 binaries can only target 11.0+. That's somewhat undesirable.

Ideally we would build the x86-64 and aarch64 binaries separately and then join them into a universal binary. This should be theoretically possible. However, there are cases where you'd need to merge the Python stdlib. e.g. the sysconfig module would needs its compiler settings normalized to target multi-arch by default. This is doable, but added complexity.

Path of least resistance is passing -arch multiple times. It works. But I don't like sacrificing the macOS <11 compatibility, as people in the wild rely on that.

[kara-todd]

@indygreg I was actually just coming to request this as well. We are running into this over on touilleMan/godot-python#322

Path of least resistance is passing -arch multiple times. It works. But I don't like sacrificing the macOS <11 compatibility, as people in the wild rely on that.

I think in our case, we'd be willing to sacrifice the <11 compatibility as the use case (game engine) will be targeting relatively modern environments. I'm not quite sure how to use this approach though. Would you be able to provide more clarity here? Or relevant documentation on how to achieve this?

[kara-todd]

@indygreg Would you be able to offer any advice on how to build a universal binary? The M1 PR on the godot-python library is sort of stalled out until I can get a fix of some sorts... I'm not really clear on how all the dependencies are being packed together. 😬

[dtcooper]

This is a feature that's long been on my mind as well.

Random considerations...

Today, we target macOS 10.9 on x86-64 and 11.0 on aarch64 since we build each arch independently. This gives us maximum binary portability. If you pass -arch to the compiler multiple times to produce a universal binary natively, I believe you have to use the same Apple SDK and targeting settings for each architecture. Since you need the macOS 11 SDK to target aarch64, I believe this means the x86-64 binaries can only target 11.0+. That's somewhat undesirable.

Ideally we would build the x86-64 and aarch64 binaries separately and then join them into a universal binary. This should be theoretically possible. However, there are cases where you'd need to merge the Python stdlib. e.g. the sysconfig module would needs its compiler settings normalized to target multi-arch by default. This is doable, but added complexity.

Path of least resistance is passing -arch multiple times. It works. But I don't like sacrificing the macOS <11 compatibility, as people in the wild rely on that.

Any thought to providing a new, universal binary build that targets macOS 11 in addition to the aarch64 (macOS 11) and x86-64 (macOS <11)?

I did it the "poor man's" way by using the lipo tool, ie for any architecture specific files I could find (the python binary, dylibs, and .a files):

1. Download the same builds for aarch64 and x86-64
2. Find all architecture specific files
3. lipo <aarch64 file> <x86-64 file> -create -output <file>
4. Copy files above to one of the builds.

A diff of the two builds after doing this still shows some differences, for example some .pyc files and C headers. Not sure if this is the "right" way to do it, but it appears to work.

[dtcooper]

Path of least resistance is passing -arch multiple times. It works. But I don't like sacrificing the macOS <11 compatibility, as people in the wild rely on that.

Revisiting this one. Is providing an additional macOS build that's a "universal" binary, ie with -arch provided multiple times with macOS>=11 compatibility something worth doing?