packaging does not know that macOS 10.16 and 11.0 are the same

[mkoeppe]

In python 3.9.9, as provided by the python.org macOS installer,

$ /Library/Frameworks/Python.framework/Versions/3.9/bin/python3
Python 3.9.9 (v3.9.9:ccb0e6a345, Nov 15 2021, 13:29:20) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.mac_ver()
('10.16', ('', '', ''), 'x86_64')

As a result, packaging 21.3 (and packaging main HEAD) do not include ...11_0... in packaging.tags.sys_tags().
However, wheel happily builds wheels with this tag, which pip (e.g. latest, 21.3.1) then refuses to install.

As platform.mac_ver() cannot be retroactively changed for released versions of Python, this should be addressed in packaging to make the support for macOS 11.x from #319 more robust.

Reference: https://trac.sagemath.org/ticket/33155

[pradyunsg]

Duplicate of #379?

[mkoeppe]

No

[mayeut]

packaging uses platform.mac_ver() as the only source to determine the current OS version.

packaging/packaging/tags.py

Line 356 in b10b90a

version_str, _, cpu_arch = platform.mac_ver()

There's definitely an issue here. Using the latest installers available on python.org for macOS:

Matt@MacBook-Pro ~ % python3.6 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c "import platform; print(platform.mac_ver())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c "import platform; print(platform.mac_ver())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.6 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c "import platform; print(platform._mac_ver_xml())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c "import platform; print(platform._mac_ver_xml())"
('12.0.1', ('', '', ''), 'x86_64')

The code for platform._mac_ver_xml() did not change much itself across versions.
It seems there are some tricks involved when reading /System/Library/CoreServices/SystemVersion.plist

10.16 and 11.0 are not the same and, at first glance, the only thing that seems to do the trick is to read this file from a subprocess:

Matt@MacBook-Pro ~ % python3.6 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')

[mkoeppe]

10.16 and 11.0 are not the same

https://news.ycombinator.com/item?id=23910165

[mayeut]

When I said not the same, I meant 10.16 implies >= 11.0 rather than ==11.0
As can be seen above, I'm on macOS 12 but 10.16 is still reported so, in the context of this project, 10.16 shall not be considered "the same as" 11.0, it would just fix a subset of the issue. If I build a wheel for macOS 12, I want to be able to install it on macOS 12+ and that would not be the case if packaging replaces 10.16 with 11.0

[brettcannon]

Is there something more official from Apple's docs than a YC link to a blog post to a screenshot of Twitter for this policy? I'm not questioning that you're right, but it would be good to have something more official to link to if we add a hack to support this. I'm also interested in official docs to know if 10.16/11 is a one-off or if this will be a perpetual issue (i.e. there won't be a 10.17 for macOS 12).

And is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

[mkoeppe]

When I said not the same, I meant 10.16 implies >= 11.0 rather than ==11.0

OK, agreed.

[mkoeppe]

is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

wheel inspects the deployment targets of the built dynamic libraries to determine the tag - https://github.com/pypa/wheel/blob/main/src/wheel/macosx_libfile.py

[brettcannon]

And can someone include the complete list of tags they are seeing on Big Sur so we have a real-world example to work against?

[mayeut]

Is there something more official from Apple's docs than a YC link to a blog post to a screenshot of Twitter for this policy? I'm not questioning that you're right, but it would be good to have something more official to link to if we add a hack to support this. I'm also interested in official docs to know if 10.16/11 is a one-off or if this will be a perpetual issue (i.e. there won't be a 10.17 for macOS 12).

Agreed, I did not stumble upon official docs when (quickly) looking at this. Maybe @lawrence-danna-apple who wrote the PR
for #318 has some more official pointers to share ?

[mkoeppe]

is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

wheel inspects the deployment targets of the built dynamic libraries to determine the tag - https://github.com/pypa/wheel/blob/main/src/wheel/macosx_libfile.py

It boils down to this behavior:

>>> from wheel.macosx_libfile import calculate_macosx_platform_tag
>>> calculate_macosx_platform_tag('/usr/local/opt/gmp/lib', 'macosx-10.14-x86_64')  # from homebrew on macOS Big Sur 11.6 
[WARNING] This wheel needs a higher macOS version than the version your Python interpreter is compiled against.  To silence this warning, set MACOSX_DEPLOYMENT_TARGET to at least 11_0 or recreate these files with lower MACOSX_DEPLOYMENT_TARGET:  
/usr/local/opt/gmp/lib/libgmpxx.4.dylib
/usr/local/opt/gmp/lib/libgmp.dylib
/usr/local/opt/gmp/lib/libgmp.10.dylib
/usr/local/opt/gmp/lib/libgmpxx.dylib'macosx_11_0_x86_64'
>>> calculate_macosx_platform_tag('/Library/Frameworks/Python.framework/Versions/3.9/lib', 'macosx-10.14-x86_64')
'macosx_10_14_x86_64'
>>> calculate_macosx_platform_tag('/Library/Frameworks/Python.framework/Versions/3.9/lib', 'macosx-10.9-x86_64')
'macosx_10_9_x86_64'

[mayeut]

And can someone include the complete list of tags they are seeing on Big Sur so we have a real-world example to work against?

On macOS 12 (Monterey):
tags.python3.6.txt
tags.python3.10.txt

Dump done with the following command (packaging 5984e3b):

python -c $'from packaging.tags import mac_platforms\nfor tag in mac_platforms(): print(tag)' > tags.python3.10.txt

--- tags.python3.6	2022-01-13 22:08:21.000000000 +0100
+++ tags.python3.10	2022-01-13 22:08:37.000000000 +0100
@@ -1,3 +1,15 @@
+macosx_12_0_x86_64
+macosx_12_0_intel
+macosx_12_0_fat64
+macosx_12_0_fat32
+macosx_12_0_universal2
+macosx_12_0_universal
+macosx_11_0_x86_64
+macosx_11_0_intel
+macosx_11_0_fat64
+macosx_11_0_fat32
+macosx_11_0_universal2
+macosx_11_0_universal
 macosx_10_16_x86_64
 macosx_10_16_intel
 macosx_10_16_fat64

[mkoeppe]

tags.python3.9.bigsur.txt

[brettcannon]

@mayeut Thanks! So macOS 12 is fine and this issue is restricted to macOS 11 Big Sur.

@mkoeppe Thanks! Here's your file inlined.

macOS 11/10.16 tags

macosx_10_16_x86_64
macosx_10_16_intel
macosx_10_16_fat64
macosx_10_16_fat32
macosx_10_16_universal2
macosx_10_16_universal
macosx_10_15_x86_64
macosx_10_15_intel
macosx_10_15_fat64
macosx_10_15_fat32
macosx_10_15_universal2
macosx_10_15_universal
macosx_10_14_x86_64
macosx_10_14_intel
macosx_10_14_fat64
macosx_10_14_fat32
macosx_10_14_universal2
macosx_10_14_universal
macosx_10_13_x86_64
macosx_10_13_intel
macosx_10_13_fat64
macosx_10_13_fat32
macosx_10_13_universal2
macosx_10_13_universal
macosx_10_12_x86_64
macosx_10_12_intel
macosx_10_12_fat64
macosx_10_12_fat32
macosx_10_12_universal2
macosx_10_12_universal
macosx_10_11_x86_64
macosx_10_11_intel
macosx_10_11_fat64
macosx_10_11_fat32
macosx_10_11_universal2
macosx_10_11_universal
macosx_10_10_x86_64
macosx_10_10_intel
macosx_10_10_fat64
macosx_10_10_fat32
macosx_10_10_universal2
macosx_10_10_universal
macosx_10_9_x86_64
macosx_10_9_intel
macosx_10_9_fat64
macosx_10_9_fat32
macosx_10_9_universal2
macosx_10_9_universal
macosx_10_8_x86_64
macosx_10_8_intel
macosx_10_8_fat64
macosx_10_8_fat32
macosx_10_8_universal2
macosx_10_8_universal
macosx_10_7_x86_64
macosx_10_7_intel
macosx_10_7_fat64
macosx_10_7_fat32
macosx_10_7_universal2
macosx_10_7_universal
macosx_10_6_x86_64
macosx_10_6_intel
macosx_10_6_fat64
macosx_10_6_fat32
macosx_10_6_universal2
macosx_10_6_universal
macosx_10_5_x86_64
macosx_10_5_intel
macosx_10_5_fat64
macosx_10_5_fat32
macosx_10_5_universal2
macosx_10_5_universal
macosx_10_4_x86_64
macosx_10_4_intel
macosx_10_4_fat64
macosx_10_4_fat32
macosx_10_4_universal2
macosx_10_4_universal

What we can probably do is add some logic such that if the version is 10.16 we force it to 11.0 and let the preexisting logic do its thing.

[brettcannon]

And if anyone wants to submit a patch for this then that would obviously be appreciated!

[brettcannon]

But I will say I don't think there's a way to go past 11.0 for 10.16 as we simply don't know if the bump would be accurate all the way up to e.g. 12.0 while real-world data suggests 10.16 == 11.0.

[mkoeppe]

What we can probably do is add some logic such that if the version is 10.16 we force it to 11.0 and let the preexisting logic do its thing.

I don't think this is enough, as tags.python3.6 on Monterey (12) shows.

[mayeut]

So macOS 12 is fine and this issue is restricted to macOS 11 Big Sur.

No, macOS 12 is not fine.
It's fine on python 3.10 official python.org installer (and in my case python 3.9) but it's not fine with python 3.6 -> 3.8 official python.org installers.

[mayeut]

In the end, it's not related to the python version itself but with which macOS SDK python was built against (i.e. wether the odd "compatibility" 10.16 behavior kicks in, with old macOS SDKs, or not, with newer macOS SDKs)

[lawrence-danna-apple]

I think what's going on here is that there's some bincompat magic applied to SystemVersion.plist. If you read it from a a program linked with an older SDK version it will return 10.16 for compatibility reasons. If you use a program linked with a newer SDK it sees the real version. I'll post some official docs here if i can find them.

[mayeut]

And if anyone wants to submit a patch for this then that would obviously be appreciated!

I could probably propose something before the end of the month (but I'd rather have official docs to rely on for this)

[lawrence-danna-apple]

well, I'm not sure if there's documentation of this behavior, but the implementation is here:

https://github.com/apple-oss-distributions/xnu/blob/xnu-7195.141.2/libsyscall/wrappers/system-version-compat-support.c

[mkoeppe]

I could probably propose something before the end of the month

Sounds good. For now on the side of SageMath we will just work around it by setting SYSTEM_VERSION_COMPAT=0 around pip install, https://trac.sagemath.org/ticket/33155#comment:4

[brettcannon]

So macOS 12 is fine and this issue is restricted to macOS 11 Big Sur.

No, macOS 12 is not fine. It's fine on python 3.10 official python.org installer (and in my case python 3.9) but it's not fine with python 3.6 -> 3.8 official python.org installers.

I'll wait to see what someone proposes to fix this then before commenting further.

And do note that Python 3.6 is EOL and so we no longer support it, so this is only being discussed in the context of Python 3.7 and Python 3.8.

[mkoeppe]

FTR, also the just released 3.9.10 installer is affected.

[mayeut]

@mkoeppe,
Only the deprecated macOS 64-bit Intel-only installer is affected.
The non-deprecated macOS 64-bit universal2 installer is not affected.

I think that's the diff in behavior we saw with 3.9 official installers.

[mkoeppe]

Thanks, that's good to know!