Black 23.10.0 and 23.10.1 wheels don't work on macOS Ventura on Apple Silicon

[ambv]

On macOS Ventura 13.6 on Apple Silicon, pip selects the universal pure Python wheel when installing Black 23.10.1:

❯ pip install black==23.10.1 --only-binary :all: --no-cache
Collecting black==23.10.1
  Downloading black-23.10.1-py3-none-any.whl.metadata (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.0/66.0 kB 1.2 MB/s eta 0:00:00
Requirement already satisfied: click>=8.0.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.10.1) (8.1.6)
Requirement already satisfied: mypy-extensions>=0.4.3 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.10.1) (1.0.0)
Requirement already satisfied: packaging>=22.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.10.1) (23.1)
Requirement already satisfied: pathspec>=0.9.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.10.1) (0.11.2)
Requirement already satisfied: platformdirs>=2 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.10.1) (3.10.0)
Downloading black-23.10.1-py3-none-any.whl (184 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.6/184.6 kB 3.4 MB/s eta 0:00:00
Installing collected packages: black
Successfully installed black-23.10.1

If you want to install the wheel file directly, you get an error:

❯ pip install 'https://files.pythonhosted.org/packages/bf/eb/8b3e03153f0cb204244ee9c9e6004bc71afc9740cef3e46788326049fb23/black-23.10.1-cp311-cp311-macosx_10_16_arm64.whl#egg=black'
ERROR: black-23.10.1-cp311-cp311-macosx_10_16_arm64.whl is not a supported wheel on this platform.

Black 23.10.0 has the same problem.

However, Black 23.9.1 and older installs just fine:

❯ pip install black==23.9.1 --only-binary :all: --no-cache
Collecting black==23.9.1
  Downloading black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl.metadata (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.1/65.1 kB 1.2 MB/s eta 0:00:00
Requirement already satisfied: click>=8.0.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.9.1) (8.1.6)
Requirement already satisfied: mypy-extensions>=0.4.3 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.9.1) (1.0.0)
Requirement already satisfied: packaging>=22.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.9.1) (23.1)
Requirement already satisfied: pathspec>=0.9.0 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.9.1) (0.11.2)
Requirement already satisfied: platformdirs>=2 in /Users/ambv/.virtualenvs/aiotone11/lib/python3.11/site-packages (from black==23.9.1) (3.10.0)
Downloading black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl (2.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 13.4 MB/s eta 0:00:00
Installing collected packages: black
  Attempting uninstall: black
    Found existing installation: black 23.10.0
    Uninstalling black-23.10.0:
      Successfully uninstalled black-23.10.0
Successfully installed black-23.9.1
❯ black --version
black, 23.9.1 (compiled: yes)
Python (CPython) 3.11.6

[ambv]

Actually, Glyph's duplicate provides the reason for this not working.

10.16 wheels should be universal2 and not arm64.

[hauntsaninja]

Hmm, this is curious.

I dropped support for universal2 builds in #3881. We did this in mypy a long time back without any problems. This is because we build single arch wheels and pip will prefer single arch builds over universal2, so basically no one used the universal2 wheels. Instead they just slow down builds, waste space on PyPI, etc. The mypy change was done by henryiii who helps maintain cibuildwheel / knows a lot about the space. I think numpy also no longer ships universal2 wheels.

Anyway regardless of whether we ship universal2, the single arch arm64 wheels we ship should work. It's unclear to me where 10_16 is coming from. For all the arm64 wheels, the macOS deployment target is usually 11_0. Maybe something something hatch? I'm seeing some suspicious code in hatch related to macos-max-compat (I'll go ask in the hatch discord)

[hugovk]

Setting MACOSX_DEPLOYMENT_TARGET="11.0" for arm64 might help?

We do this for Pillow:

  if [[ "$PLAT" == "arm64" ]]; then
    export MACOSX_DEPLOYMENT_TARGET="11.0"
  else
    export MACOSX_DEPLOYMENT_TARGET="10.10"
  fi

https://github.com/python-pillow/pillow-wheels/blob/92612210ec8225018c8a8309d3ddbb19a67dd6ec/.github/workflows/build.sh#L12-L16

Although we use multibuild, so see https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions for cibuildwheel options.

---

Here's some investigation showing which wheels are compatible and selected.

With Python 3.11 on macOS Ventura Apple Silicon, adding --force-reinstall -vv for verbose logging:

pip install black==23.9.1 --only-binary :all: --no-cache --force-reinstall -vv > 23.9.1.txt
pip install black==23.10.1 --only-binary :all: --no-cache --force-reinstall -vv > 23.10.1.txt

Then grepping the logs, 23.9.1 is selecting the universal2:

❯ grep 23.9.1 23.9.1.txt | grep macos | grep cp311
  Skipping link: none of the wheel's tags (cp311-cp311-macosx_10_16_arm64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/64/fa/1107aafc073fd6340c09a95b2367c8f5127083bf6cb9738bdd11c671350f/black-23.9.1-cp311-cp311-macosx_10_16_arm64.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8)
  Found link https://files.pythonhosted.org/packages/72/1a/fc7a669677250d73ea190342d360b3bd150043e61e85d1dbb7ae8bd8525a/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8), version: 23.9.1
  Skipping link: none of the wheel's tags (cp311-cp311-macosx_10_16_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/db/9e/1efe457688f60c8a3b364724828a75dd9939d77c9deaa821d7fb09af3c7f/black-23.9.1-cp311-cp311-macosx_10_16_x86_64.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8)
  Obtaining dependency information for black==23.9.1 from https://files.pythonhosted.org/packages/72/1a/fc7a669677250d73ea190342d360b3bd150043e61e85d1dbb7ae8bd8525a/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl.metadata
  https://files.pythonhosted.org:443 "GET /packages/72/1a/fc7a669677250d73ea190342d360b3bd150043e61e85d1dbb7ae8bd8525a/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl.metadata HTTP/1.1" 200 65132
  Downloading black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl.metadata (65 kB)
https://files.pythonhosted.org:443 "GET /packages/72/1a/fc7a669677250d73ea190342d360b3bd150043e61e85d1dbb7ae8bd8525a/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl HTTP/1.1" 200 2597509
Downloading black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl (2.6 MB)
Downloading link https://files.pythonhosted.org/packages/72/1a/fc7a669677250d73ea190342d360b3bd150043e61e85d1dbb7ae8bd8525a/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8) to /private/var/folders/c8/hl_96fcs0hjgr29njx33x20c0000gn/T/pip-unpack-3qfqo5oh/black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl

But 23.10.1 is not selecting any cp311 wheels:

❯ grep 23.10.1 23.10.1.txt | grep macos | grep cp311
  Skipping link: none of the wheel's tags (cp311-cp311-macosx_10_16_arm64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/bf/eb/8b3e03153f0cb204244ee9c9e6004bc71afc9740cef3e46788326049fb23/black-23.10.1-cp311-cp311-macosx_10_16_arm64.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8)
  Skipping link: none of the wheel's tags (cp311-cp311-macosx_10_16_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/8b/3c/c9a03a4d5dd8c18c4af211e694bcc73dd305a2b85788eb311d3dbb14cfe9/black-23.10.1-cp311-cp311-macosx_10_16_x86_64.whl (from https://pypi.org/simple/black/) (requires-python:>=3.8)

As suggested by the logs, running pip debug --verbose:

❯ pip debug --verbose | grep cp311
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
  cp311-cp311-macosx_13_0_arm64
  cp311-cp311-macosx_13_0_universal2
  cp311-cp311-macosx_12_0_arm64
  cp311-cp311-macosx_12_0_universal2
  cp311-cp311-macosx_11_0_arm64
  cp311-cp311-macosx_11_0_universal2
  cp311-cp311-macosx_10_16_universal2
  cp311-cp311-macosx_10_15_universal2
  cp311-cp311-macosx_10_14_universal2
  cp311-cp311-macosx_10_13_universal2
  cp311-cp311-macosx_10_12_universal2
  cp311-cp311-macosx_10_11_universal2
  cp311-cp311-macosx_10_10_universal2
  cp311-cp311-macosx_10_9_universal2
  cp311-cp311-macosx_10_8_universal2
  cp311-cp311-macosx_10_7_universal2
  cp311-cp311-macosx_10_6_universal2
  cp311-cp311-macosx_10_5_universal2
  cp311-cp311-macosx_10_4_universal2
  cp311-abi3-macosx_13_0_arm64
  cp311-abi3-macosx_13_0_universal2
  cp311-abi3-macosx_12_0_arm64
  cp311-abi3-macosx_12_0_universal2
  cp311-abi3-macosx_11_0_arm64
  cp311-abi3-macosx_11_0_universal2
  cp311-abi3-macosx_10_16_universal2
  cp311-abi3-macosx_10_15_universal2
  cp311-abi3-macosx_10_14_universal2
  cp311-abi3-macosx_10_13_universal2
  cp311-abi3-macosx_10_12_universal2
  cp311-abi3-macosx_10_11_universal2
  cp311-abi3-macosx_10_10_universal2
  cp311-abi3-macosx_10_9_universal2
  cp311-abi3-macosx_10_8_universal2
  cp311-abi3-macosx_10_7_universal2
  cp311-abi3-macosx_10_6_universal2
  cp311-abi3-macosx_10_5_universal2
  cp311-abi3-macosx_10_4_universal2
  cp311-none-macosx_13_0_arm64
  cp311-none-macosx_13_0_universal2
  cp311-none-macosx_12_0_arm64
  cp311-none-macosx_12_0_universal2
  cp311-none-macosx_11_0_arm64
  cp311-none-macosx_11_0_universal2
  cp311-none-macosx_10_16_universal2
  cp311-none-macosx_10_15_universal2
  cp311-none-macosx_10_14_universal2
  cp311-none-macosx_10_13_universal2
  cp311-none-macosx_10_12_universal2
  cp311-none-macosx_10_11_universal2
  cp311-none-macosx_10_10_universal2
  cp311-none-macosx_10_9_universal2
  cp311-none-macosx_10_8_universal2
  cp311-none-macosx_10_7_universal2
  cp311-none-macosx_10_6_universal2
  cp311-none-macosx_10_5_universal2
  cp311-none-macosx_10_4_universal2
  cp311-none-any

So the only compatible 10_16 wheels are universal2, which obviously don't match the only two cp311/macos wheels:

• black-23.10.1-cp311-cp311-macosx_10_16_x86_64.whl
• black-23.10.1-cp311-cp311-macosx_10_16_arm64.whl

For an arm64, 11_0 or higher is needed.

[ofek]

This is the logic: https://github.com/pypa/hatch/blob/hatchling-v1.18.0/backend/src/hatchling/builders/wheel.py#L620-L641
The macos-max-compat option is documented here: https://hatch.pypa.io/latest/plugins/builder/wheel/#options

History:

• Acknowledge the ARCHFLAGS environment variable on macOS pypa/hatch#672
• Add macos-max-compat option to the wheel target pypa/hatch#699

[ofek]

Related, non-Hatch: microsoft/onnxruntime#17166