Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wheel build should use a locked requirements file #22042

Open
jwnimmer-tri opened this issue Oct 15, 2024 · 2 comments
Open

Wheel build should use a locked requirements file #22042

jwnimmer-tri opened this issue Oct 15, 2024 · 2 comments
Assignees
Labels
component: distribution Nightly binaries, monthly releases, docker, installation priority: medium type: feature request

Comments

@jwnimmer-tri
Copy link
Collaborator

jwnimmer-tri commented Oct 15, 2024

Is your feature request related to a problem? Please describe.

In our wheel build, currently we install some unpinned dependencies:

pip install --upgrade \
delocate \
setuptools \
wheel

# TODO(jwnimmer-tri): Should these be version-pinned? What's the process for
# keeping them up to date if they are?
pip install \
matplotlib \
numpy \
pyyaml \
semantic-version \
setuptools \
wheel \
auditwheel
if [[ "$(uname)" == "Linux" ]]; then
pip install patchelf
fi

This is a problem because it means our wheel builds are not reproducible (so checking out a Drake tag and building it might produce a different result, or even not work at all), and that broken uploads to PyPI can break our Nightly CI out from under us.

Describe the solution you'd like

  • Compute requirements lockfile(s) for our wheel builds, so that the versions of everything we need are consistent across repeated builds of the same Drake source tree.
  • Ensure that the lockfile is regularly refreshed (ala Monthly upgrades for venv lockfile #22041).

Describe alternatives you've considered

n/a

Additional context

At the moment, in non-wheel builds, we only have venv lockfiles for macOS. This issue should lock both the Linux wheel build and the macOS wheel build, i.e., the two code snippets above.

This is a sub-task of the #8392 epic.

@jwnimmer-tri
Copy link
Collaborator Author

From f2f, implementation options:

(1) Run the jazzband lockfile generator a half dozen times (or whatever) on the matrix of python versions / platforms, to have N different lockfiles.

(2) Find a lockfile creation tool that can resolve dependencies in an interpreter and platform agnostic way, have just one lockfile.

(3) Just type in hard-coded numbers in https://github.com/RobotLocomotion/drake/blob/master/tools/wheel/image/provision-python.sh, without a lockfile, and without any automated re-locking tool

Doing option (2) would be best if we can find and wrangle such a tool. Otherwise we'll fall back to (3) and live with it indefinitely.

@BetsyMcPhail BetsyMcPhail removed their assignment Dec 23, 2024
@mwoehlke-kitware
Copy link
Contributor

PDM looks plausible for (2) (see also #22349), and if/when that lands, it shouldn't be a big deal to use it for the macOS wheels. However, having a separate implementation for the Linux wheels might be messy. IMHO, if we go in that direction, it would be best to tackle #22040 first, then locking for macOS wheels only, then switch Linux non-wheel builds over to using PDM for Python dependencies, and only lock the Linux wheel-build dependencies once PDM is already in place otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: distribution Nightly binaries, monthly releases, docker, installation priority: medium type: feature request
Development

No branches or pull requests

3 participants