Skip to content

Commit

Permalink
[build] Use MODULE.bazel for all dependencies
Browse files Browse the repository at this point in the history
The WORKSPACE.bzlmod is no longer load-bearing, so as of this commit
it should be possible to consume Drake as module external of other
downstream projects (instead of as a repository external).

Notable changes:

Adjust the pkg_config and java repository rules to handle the new
distinction between canonical vs apparent repository names.

Adjust some of our hard-coded runfiles paths (header_lint test, wheel
build snopt, drake_models parse_test) to align with the new canonical
repository names.

Adjust our lcm native code loader to accommodate the new runfiles
layout. Add test coverage for EventLog (which uses a distinctive
spelling of its import paths in upstream code).

Adjust labels used by our (non-symbolic) macros to only ever refer to
drake labels, not external labels. Textual macros resolve labels in
the workspace context of the code calling them, not Drake. Therefore
they must only ever refer to Drake, since Drake's externals are now
invisible (by default) with bzlmod. We introduce Drake aliases for the
externals so that we can use a safe labels in our macros. (This fix is
only necessary for macros which we expect downstream code to call,
i.e., macros without a "drake_..." prefix in their name. We still have
plenty of other drake-specific macros that refer to non-drake labels,
but that's not a problem.) The longer term fix for this will probably
be switching from textual macros to symbolic macros, but we don't
attempt that here.

Adjust CMake logic to edit MODULE.bazel to opt-out of any overridden
externals, so that the WORKSPACE.bzlmod replacements take effect.

Document our stability promises for our module extension.
  • Loading branch information
jwnimmer-tri committed Dec 21, 2024
1 parent 9be1979 commit 81104a5
Show file tree
Hide file tree
Showing 19 changed files with 489 additions and 162 deletions.
20 changes: 16 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ project(drake
# (e.g., `-DCMAKE_BUILD_TYPE=Release`) and install Drake using those settings.
#
# We'll do that by converting the settings to generated Bazel inputs:
# - a `WORKSPACE.bazel` file that specifies dependencies; and
# - a tweaked copy of our `MODULE.bazel` to opt-out of some dependencies; and
# - a completely new `WORKSPACE.bzlmod` to replace what we opted-out of; and
# - a `.bazelrc` file that specifies configuration choices.
# and then running the `@drake//:install` program from that temporary workspace.

Expand Down Expand Up @@ -77,8 +78,7 @@ else()
endif()
endif()

# The version passed to find_package(Bazel) should match the
# minimum_bazel_version value in the call to versions.check() in WORKSPACE.
# This version number should match bazel_compatibility in MODULE.bazel.
set(MINIMUM_BAZEL_VERSION 7.4)
find_package(Bazel ${MINIMUM_BAZEL_VERSION} MODULE)
if(NOT Bazel_FOUND)
Expand Down Expand Up @@ -350,6 +350,9 @@ endfunction()
set(BAZEL_WORKSPACE_EXTRA)
set(BAZEL_WORKSPACE_EXCLUDES)

# Our cmake/WORKSPACE.bzlmod always provides @python.
list(APPEND BAZEL_WORKSPACE_EXCLUDES "python")

macro(override_repository NAME)
set(repo "${CMAKE_CURRENT_BINARY_DIR}/external/workspace/${NAME}")
string(APPEND BAZEL_WORKSPACE_EXTRA
Expand Down Expand Up @@ -541,10 +544,19 @@ endforeach()
# name `drake_build_cwd` isn't important, it just needs to be unique. Note,
# however, that the macOS wheel builds also need to know this path, so if it
# ever changes, tools/wheel/macos/build-wheel.sh will also need to be updated.
file(READ "${PROJECT_SOURCE_DIR}/MODULE.bazel" BAZEL_MODULE_CONTENTS)
foreach(BAZEL_WORKSPACE_EXCLUDE ${BAZEL_WORKSPACE_EXCLUDES})
string(REGEX REPLACE
"\"${BAZEL_WORKSPACE_EXCLUDE}\","
"# ${BAZEL_WORKSPACE_EXCLUDE} comes from WORKSPACE.bzlmod"
BAZEL_MODULE_CONTENTS "${BAZEL_MODULE_CONTENTS}")
endforeach()
file(WRITE
"${CMAKE_CURRENT_BINARY_DIR}/drake_build_cwd/MODULE.bazel"
"${BAZEL_MODULE_CONTENTS}")
configure_file(cmake/bazel.rc.in drake_build_cwd/.bazelrc @ONLY)
configure_file(cmake/WORKSPACE.bzlmod.in drake_build_cwd/WORKSPACE.bzlmod @ONLY)
file(CREATE_LINK "${PROJECT_SOURCE_DIR}/.bazeliskrc" drake_build_cwd/.bazeliskrc SYMBOLIC)
file(CREATE_LINK "${PROJECT_SOURCE_DIR}/MODULE.bazel" drake_build_cwd/MODULE.bazel SYMBOLIC)
file(CREATE_LINK "${PROJECT_SOURCE_DIR}/WORKSPACE" drake_build_cwd/WORKSPACE SYMBOLIC)

find_package(Git)
Expand Down
201 changes: 194 additions & 7 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@
# This file lists Drake's external dependencies as known to bzlmod. It is used
# in concert with WORKSPACE.bzlmod (which has the workspace-style externals).

module(name = "drake")
module(
name = "drake",
# This version number should match MINIMUM_BAZEL_VERSION in CMakeLists.txt.
bazel_compatibility = [">=7.4.0"],
)

# Add starlark rules.

bazel_dep(name = "apple_support", version = "1.17.1", repo_name = "build_bazel_apple_support") # noqa
bazel_dep(name = "bazel_features", version = "1.22.0")
Expand All @@ -17,18 +23,199 @@ bazel_dep(name = "rules_python", version = "0.40.0")
bazel_dep(name = "rules_rust", version = "0.56.0")
bazel_dep(name = "rules_shell", version = "0.3.0")

# Customize our toolchains.

cc_configure = use_extension(
"@rules_cc//cc:extensions.bzl",
"cc_configure_extension",
)
use_repo(cc_configure, "local_config_cc")

register_toolchains(
"//tools/py_toolchain:toolchain",
"//tools/py_toolchain:exec_tools_toolchain",
"@drake//tools/py_toolchain:toolchain",
"@drake//tools/py_toolchain:exec_tools_toolchain",
)

# Load dependencies which are "public", i.e., made available to downstream
# projects.
#
# Downstream projects may load the same `drake_dep_repositories` module
# extension shown below and call its `use_repo` with whatever list of
# repositories they desire to cite from their project. It's safe to call
# `use_repo` on a subset of this list, or not call it at all downstream.
# Its only effect on a downstream project is to make the repository name
# visible to BUILD rules; Drake's own use of the repository is unaffected.

drake_dep_repositories = use_extension(
"@drake//tools/workspace:default.bzl",
"drake_dep_repositories",
)
use_repo(
drake_dep_repositories,
"blas",
"buildifier",
"drake_models",
"eigen",
"fmt",
"gflags",
"glib",
"glx",
"gtest",
"gurobi",
"lapack",
"lcm",
"libblas",
"liblapack",
"meshcat",
"mosek",
"opencl",
"opengl",
"pybind11",
"pycodestyle",
"python",
"spdlog",
"styleguide",
"x11",
"zlib",
)

# Load dependencies which are "private", i.e., not available for use by
# downstream projects. These are all "internal use only".

internal_repositories = use_extension(
"@drake//tools/workspace:default.bzl",
"internal_repositories",
)
use_repo(
internal_repositories,
"abseil_cpp_internal",
"bazelisk",
"cc",
"ccd_internal",
"clang_cindex_python3_internal",
"clarabel_cpp_internal",
"clp_internal",
"coinutils_internal",
"com_jidesoft_jide_oss",
"common_robotics_utilities_internal",
"commons_io",
"conex_internal",
"csdp_internal",
"curl_internal",
"dm_control_internal",
"doxygen",
"fcl_internal",
"gfortran",
"github3_py_internal",
"gklib_internal",
"googlebenchmark",
"gymnasium_py",
"gz_math_internal",
"gz_utils_internal",
"highway_internal",
"ipopt",
"ipopt_internal_fromsource",
"libjpeg_turbo_internal",
"libpng_internal",
"libtiff_internal",
"metis_internal",
"mpmath_py_internal",
"msgpack_internal",
"mujoco_menagerie_internal",
"mumps_internal",
"mypy_extensions_internal",
"mypy_internal",
"nanoflann_internal",
"nasm",
"net_sf_jchart2d",
"nlohmann_internal",
"nlopt_internal",
"onetbb_internal",
"openusd_internal",
"org_apache_xmlgraphics_commons",
"osqp_internal",
"picosha2_internal",
"poisson_disk_sampling_internal",
"qdldl_internal",
"qhull_internal",
"ros_xacro_internal",
"rules_python_drake_constants",
"scs_internal",
"sdformat_internal",
"snopt",
"spgrid_internal",
"spral_internal",
"stable_baselines3_internal",
"statsjs",
"stduuid_internal",
"suitesparse_internal",
"sympy_py_internal",
"tinygltf_internal",
"tinyobjloader_internal",
"tinyxml2_internal",
"tomli_internal",
"typing_extensions_internal",
"uritemplate_py_internal",
"usockets_internal",
"uwebsockets_internal",
"voxelized_geometry_tools_internal",
"vtk_internal",
"xmlrunner_py",
"yaml_cpp_internal",
)

internal_crate_universe_repositories = use_extension(
"//tools/workspace:default.bzl",
"internal_crate_universe_repositories",
)
use_repo(
internal_crate_universe_repositories,
"crate__amd-0.2.2",
"crate__autocfg-1.4.0",
"crate__blas-0.22.0",
"crate__blas-sys-0.7.1",
"crate__cfg-if-1.0.0",
"crate__clarabel-0.9.0",
"crate__darling-0.14.4",
"crate__darling_core-0.14.4",
"crate__darling_macro-0.14.4",
"crate__derive_builder-0.11.2",
"crate__derive_builder_core-0.11.2",
"crate__derive_builder_macro-0.11.2",
"crate__either-1.13.0",
"crate__enum_dispatch-0.3.13",
"crate__equivalent-1.0.1",
"crate__fnv-1.0.7",
"crate__hashbrown-0.15.2",
"crate__ident_case-1.0.1",
"crate__indexmap-2.7.0",
"crate__itertools-0.11.0",
"crate__itoa-1.0.14",
"crate__lapack-0.19.0",
"crate__lapack-sys-0.14.0",
"crate__lazy_static-1.5.0",
"crate__libc-0.2.168",
"crate__memchr-2.7.4",
"crate__num-complex-0.4.6",
"crate__num-traits-0.2.19",
"crate__once_cell-1.19.0",
"crate__paste-1.0.15",
"crate__proc-macro2-1.0.92",
"crate__quote-1.0.37",
"crate__ryu-1.0.18",
"crate__serde-1.0.216",
"crate__serde_derive-1.0.216",
"crate__serde_json-1.0.133",
"crate__strsim-0.10.0",
"crate__syn-1.0.109",
"crate__syn-2.0.90",
"crate__thiserror-1.0.69",
"crate__thiserror-impl-1.0.69",
"crate__unicode-ident-1.0.14",
)

# TODO(#20731) Move all of our dependencies from WORKSPACE.bzlmod into this
# file, so that downstream projects can consume Drake exclusively via bzlmod
# (and so that we can delete our WORKSPACE files prior to Bazel 9 which drops
# suppose for it).
# TODO(#20731) More improvements are still needed to our MODULE organization:
# - Switch public API dependencies (e.g., eigen) to use modules.
# - Provide better configuation options for choosing dependencies.
# - Adjust the wheel build to build more dependencies as Bazel modules.
# - Deprecate non-bzlmod use of Drake downstream.
16 changes: 3 additions & 13 deletions WORKSPACE.bzlmod
Original file line number Diff line number Diff line change
@@ -1,22 +1,12 @@
# -*- bazel -*-
#
# This file lists Drake's workspace-style external dependencies. It is used in
# concert with MODULE.bazel (which has the module-style externals).
# This file lists Drake's workspace-style external dependencies that are only
# needed by Drake Developers instead of downstream projects. Most dependencies
# are listed in MODULE.bazel, instead.

workspace(name = "drake")

load("//tools/workspace:default.bzl", "add_default_workspace")

add_default_workspace(bzlmod = True)

# Add some special heuristic logic for using CLion with Drake.
load("//tools/clion:repository.bzl", "drake_clion_environment")

drake_clion_environment()

load("@bazel_skylib//lib:versions.bzl", "versions")

# This needs to be in WORKSPACE or a repository rule for native.bazel_version
# to actually be defined. The minimum_bazel_version value should match the
# version passed to the find_package(Bazel) call in the root CMakeLists.txt.
versions.check(minimum_bazel_version = "7.4")
11 changes: 0 additions & 11 deletions cmake/WORKSPACE.bzlmod.in
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
workspace(name = "drake")

load("//:cmake/external/workspace/conversion.bzl", "split_cmake_list")
load("//tools/workspace/python:repository.bzl", "python_repository")
load("//tools/workspace:default.bzl", "add_default_workspace")

# Use Drake's python repository rule to interrogate the interpreter chosen by
# the CMake find_program stanza, in support of compiling our C++ bindings.
Expand All @@ -15,12 +13,3 @@ python_repository(

# Custom repository rules injected by CMake.
@BAZEL_WORKSPACE_EXTRA@

# The list of repositories already provided via BAZEL_WORKSPACE_EXTRA.
_BAZEL_WORKSPACE_EXCLUDES = split_cmake_list("@BAZEL_WORKSPACE_EXCLUDES@")

# For anything not already overridden, use Drake's default externals.
add_default_workspace(
repository_excludes = ["python"] + _BAZEL_WORKSPACE_EXCLUDES,
bzlmod = True,
)
27 changes: 19 additions & 8 deletions doc/_pages/stable.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,20 +104,31 @@ part of the "Stable API":

For Drake's dependencies:

* The `add_default_...` macros defined in `@drake//tools/workspace:default.bzl`
are all part of the Stable API.
* For any Bazel external loaded by these functions (e.g., `"@eigen"`), we
will deprecate it prior to removing our definition of the dependency.
* Excluding any items documented as "internal use only".
* Excluding any items documented with an "experimental" warning.
* When using Bazel to depend on Drake as a Bazel Module (i.e., using bzlmod):
* The extension module
`use_extension("@drake//tools/workspace:default.bzl", "drake_dep_repositories")`
is part of the Stable API, including the names of the repositories it offers
as extensions (e.g., `"eigen"`).
* For any repository provided by the extension, we will deprecate
it prior to removing it.
* When using Bazel to depend on Drake via `WORKSPACE.bazel` (i.e., without
bzlmod):
* The `add_default_...` macros defined in
`@drake//tools/workspace:default.bzl` are all part of the Stable API.
* For any Bazel external loaded by these functions (e.g., `"@eigen"`), we
will deprecate it prior to removing our definition of the dependency.
* Excluding any items documented as "internal use only".
* Excluding any items documented with an "experimental" warning.

We may upgrade any of our dependencies to a newer version without prior notice.
If you require an older version, you will need to rebuild Drake from source and
pin your own WORKSPACE to refer to the older version of the dependency.
customize your own `WORKSPACE.bazel` or `MODULE.bazel` file to refer to the
older version of the dependency.

We may add new dependencies without prior notice. All of our dependencies will
either be installed via the host system via our `install_prereqs` scripts,
and/or downloaded at build-time via our `add_default_...` macros, and/or
and/or downloaded at build-time via our `add_default_...` macros (when not
using `bzlmod`) or our `MODULE.bazel` file (when using bzlmod), and/or
specified via packaging metadata in the case of `apt` or `pip`.

## LCM messages
Expand Down
11 changes: 4 additions & 7 deletions tools/install/libdrake/header_lint.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,10 @@ load("//tools/skylark:sh.bzl", "sh_test")
# without consulting Drake's build system maintainers (see #7451). Keep this
# list in sync with test/header_dependency_test.py.
_ALLOWED_EXTERNALS = [
"eigen",
"fmt",
"lcm",
"spdlog",

# The entries that follow are defects; we should work to remove them.
"zlib",
"+drake_dep_repositories+eigen",
"+drake_dep_repositories+fmt",
"+drake_dep_repositories+lcm",
"+drake_dep_repositories+spdlog",
]

# Drake's allowed list of public preprocessor definitions. The only things
Expand Down
5 changes: 2 additions & 3 deletions tools/skylark/py.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ load(
)

# All of Drake's Python code should depend on our requirements.txt pins, so we
# add it as a data dependency to every python rule. If this particular build
# doesn't use a requirements.txt, then the file will be empty (and thus inert).
# add it as a data dependency to every python rule.

def _add_requirements(data):
return (data or []) + ["@python//:requirements.txt"]
return (data or []) + ["@drake//tools/workspace/python:requirements"]

def py_binary(name, *, data = None, **kwargs):
_py_binary(
Expand Down
Loading

0 comments on commit 81104a5

Please sign in to comment.