Pre-installed Python conflicts with Homebrew Python on macOS

[bebound]

Description

In macOS 12 and 13, it has pre-installed Python /Library/Frameworks/Python.framework/Versions/3.xx/, which creates soft link such as /usr/local/bin/2to3-3.xx. Homebrew's python@3.xx also wants to create this link.

This causes issue when there is new Python is released, and brew install xx tries to install the latest Python: see Homebrew/homebrew-core#165793 (comment)

Runner image has a fix for Python 2.7:

runner-images/images/macos/scripts/build/install-python.sh

Lines 17 to 25 in 9d5d1be

	choice_changes_xml=$(mktemp /tmp/python2_choice_changes.xml.XXXXXX)
	sudo installer -showChoiceChangesXML -pkg $python2_pkg -target / | tee $choice_changes_xml > /dev/null
	# To avoid symlink conflicts, remove tools installation in /usr/local/bin using installer choices
	xmllint --shell $choice_changes_xml <<EOF
	cd //array/dict[string[text()='org.python.Python.PythonUnixTools-2.7']]/integer
	set 0
	save
	EOF

It used to let brew overwrite the python@3.11 soft link to resolve the conflict, but is removed in https://github.com/actions/runner-images/pull/8452/files#diff-5490845eec9cd46fa52e10c62bd6ec6a76c27f17a338963a508414efd8b2252dR33-R34

Now I have to run brew unlink python@3.11 && brew link --overwrite python@3.11 before using brew install xxx to eliminate the potential conflict. It's tedious.

There is a PR attempting to address this issue: #7710, but it was rejected. Regardless of whether the fix should be implemented in actions/setup-python, we are still facing this issue one year later. I believe the runner image should consider providing a temporary fix for this.

Related issue:
Homebrew/homebrew-core#173191
Azure/azure-cli#29054

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

This could be reproduced by reinstalling python@3.11

brew uninstall azure-cli
brew install python@3.11

https://github.com/bebound/starter-workflows/actions/runs/9298969591/job/25591905178

Is it regression?

Yes?

Expected behavior

Successfully install python@3.11 with brew.

Actual behavior

Pouring python@3.11--3.11.9.ventura.bottle.tar.gz
Error: The `brew link` step did not complete successfully
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3-3.[11](https://github.com/bebound/starter-workflows/actions/runs/9298969591/job/25591905178#step:7:12)
Target /usr/local/bin/2to3-3.11
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3-3.11'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.11

To list all files that would be deleted:
  brew link --overwrite python@3.11 --dry-run

Possible conflicting files are:
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config

Repro steps

brew uninstall azure-cli
brew install python@3.11

[Alexey-Ayupov]

Hello @bebound, this issue should be fixed after the next image release.

[bebound]

@Alexey-Ayupov May I know which PR fixes this?

[erik-bershel]

Hello @bebound !

A little (not exactly true 😄) clarification. @Alexey-Ayupov meant that the problem would resolve itself with updating the macOS images until Homebrew released dependency updates.
Unfortunately this problem lies between setup-python and Homebrew - their packages conflict due to the paths used as @cho-m mentioned.
Hypothetically, we could change the path, which would remove the problem for Homebrew users, but would break users of the setup-python action for some scenarios. Unfortunately, the most complete solution to this problem “forever” would be to remove the Python packages from the image in order to install it during runtime from specific source by customers, which is a bit irrational. Relinking is always easier than reinstalling.

We will do some more research on what can be done in a way not to break existed workflows and welcome any ideas from you as a user.

[bebound]

I prefer let homebrew overwrite setup-python for two reasons:

1. Homebrew is more likely to trigger the conflict as it always checks for new version, and it's hard to debug when brew install fails suddenly because Python release a new version.
2. It's more explicit/efficient to resolve the conflict when users use setup-python than use Brew to install packages depending on python@xxx.

Also, if this possible to pre-install brew's python@xx to replace setup-python on macOS?

Out of curiosity, which line creates the soft link from /usr/local/bin/python3.11 to /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11?

[QuLogic]

It seems like this is slowly rolling out to more runners.

[QuLogic]

Oh, and it's back again now that Homebrew released Python 3.12.5, but the images are on an older version. It's also affecting both Azure and GitHub Actions now.

The annoying thing is that we aren't even using the Homebrew Python, but the one from setup-python/UsePythonVersion. But one of our system dependencies (installed via Homebrew) does, and so we have this conflict arise for something we don't really use.