[workspace] Create a virtual environment for MacOS Python deps.

[MarcusSorealheis]

This Pull Request creates a virtual environment for Python dependencies on MacOS so that we can remove the --break-system-packages argument to the pip install invocation. It fixes: #8392

---

This change is 

[MarcusSorealheis]

@jwnimmer-tri and @svenevs I think this is a simple approach, and I added a comment so that there is a bit of a paper trail. It may not be necessary. I think there are other approaches particularly with respect to control flow, but pip3.12 should be included in the python distribution from that version going forward.

[jwnimmer-tri]

Thanks, I will take a look in 2-3 weeks, but I'm short on time these days.

One big-picture problem: we must allow for the possibility of multiple different binary installs of Drake (at different versions) that are all independent (don't share a venv), and likewise we must allow for multiple different source checkouts of drake (at different versions) that are all independent (don't share a venv). One way or another, the venv need to be somehow related to the particular copy of (i.e., path to) Drake that it's associated with. That might mean we need different techniques for binary vs source prereqs, I don't know.

I'll also note this comment from #8392:

It's probable that Drake on macOS should only use the Python interpreter from Homebrew, and for other Python ecosystem dependencies we use pip (with a lockfile).

It possible / probable that source install_prereqs should be doing nothing with pip (and no venv). For source builds, we might want to use bazel rules_python to fetch pip packages into the workspace automatically, with no venv at all.

The main work of #8392 is not the writing of a few lines of code, it's studying the several possible approaches, understanding their pros and cons, and choosing which mechanism to use.

[MarcusSorealheis]

The main work of #8392 is not the writing of a few lines of code, it's studying the several possible approaches, understanding their pros and cons, and choosing which mechanism to use.

On this, I totally agree. The initial work here was meant to be a forcing function for me to outline the pros and cons with data more than anything else at this stage. It will take me ~2-3 weeks to outline what I perceive about the different challenges and approaches between homebrew, venv, and pipx. I do have some of the data noted but I still need to spend time working through it to make it presentable.

I consulted some Python experts on this one as I am more of a systems programming and databases person day-to-day. With open source, everyone needs to pitch in across skillsets and organizations, though. I have a few clean Mac machines to work through as well.

Your comments have given me some clarity on the goals to inform my proposed approach. Thank you.

[rpoyner-tri]

Assigning build_system component owner @jwnimmer-tri for eventual review or delegation.

[rpoyner-tri]

Reviewed 2 of 2 files at r1, all commit messages.
Reviewable status: LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, missing label for release notes (waiting on @MarcusSorealheis)

[MarcusSorealheis]

I'm still learning a lot and have a lot more studying to do, but preliminarily your intuition is correct @jwnimmer-tri with respect to the source distribution. The only risk I see right now is two install paths (source vs binary) that do not resemble each other. I'm not sure if that is a real problem yet.

Using rules_python for source distribution:

Pros	Cons
Users will have fine-grained control over dependencies	There is initially more overhead for people accustomed to a Pythonic ecosystem
The project moves toward more hermiticity across platforms	The overhead may not be worth it unless there is lots of documentation on how to use the tool and there is only a single platform to support
There are not many risks with the number of Python dependencies	This approach grows more difficult as the list of dependencies grows

I'm going to update this PR to include this approach and try it out so that everyone can get a sense for what the implementation might look like. The goal is more reproducibility and usability.

As a part of adding rules_python, I am exploring a few options. Do we need to support Apple cross platform for python3.12? My assumption is yes, but I want to be sure because almost all MacOS users will be on Apple Silicon soon. Something interesting I found for cross-platform inspiration is rules_pycross, though it won't be a huge deal with so few CPU-only dependencies.

[MarcusSorealheis]

Here is a note for posterity. Nothing more:

I just checked the 1.27.0 release notes and read what I was looking for with respect to MacOS support:

Drake's support for running on x86_64 reached end of life in Drake v1.22.0. We have continued to ship x86 packages as a courtesy, but this will be the last release to do so. In the future, we will only ship *.whl packages for macOS x86_64 (not *.tgz packages), and only for stable releases (no more nightly builds).

We are no longer running CI testing on macOS x86_64, except for the minimal recipe that compiles the wheel packages.

Looks like the project will only need to support Apple Silicon going forward.

[jwnimmer-tri]

To be very clear, the macOS x86 wheel builds are still currently supported and will need a python venv during the build. Happily, the wheel builds already handle their own venv setup so are out of scope for this PR.

[jwnimmer-tri]

Reviewed all commit messages.
Reviewable status: LGTM missing from assignee jwnimmer-tri(platform), needs at least two assigned reviewers, missing label for release notes (waiting on @MarcusSorealheis)

[jwnimmer-tri]

Closing as stale.

I have new details at #8392 (comment) which Kitware is planning to carry forward for us.