-
Notifications
You must be signed in to change notification settings - Fork 424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose crates_vendor deps to bzlmod #2372
Expose crates_vendor deps to bzlmod #2372
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix! I think this looks reasonable (though cc @matts1 for a better look) - would it be possible to expand https://github.com/bazelbuild/rules_rust/tree/main/examples/bzlmod/hello_world and https://github.com/bazelbuild/rules_rust/blob/main/.bcr/presubmit.yml and https://github.com/bazelbuild/rules_rust/blob/main/.bazelci/presubmit.yml to exercise the workflow that was broken for you?
be82581
to
e4e83f3
Compare
#2373 should help with CI |
Thanks! I've added the ERROR: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~override'
ERROR: f2e42d6afe2730a9d87691f9ba152a3d/external/rules_rust~override/crate_universe/private/vendor/BUILD.bazel:3:27: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~override' and referenced by '@@rules_rust~override//crate_universe/private/vendor:buildifier'
ERROR: Analysis of target '//third-party:vendor' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.350s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target |
"""Module extensions for using vendored crates with bzlmod""" | ||
|
||
load("//third-party/crates:defs.bzl", _crate_repositories = "crate_repositories") | ||
|
||
def _crate_repositories_impl(module_ctx): | ||
direct_deps = _crate_repositories() | ||
return module_ctx.extension_metadata( | ||
root_module_direct_deps = [repo.repo for repo in direct_deps], | ||
root_module_direct_dev_deps = [], | ||
) | ||
|
||
crate_repositories = module_extension( | ||
implementation = _crate_repositories_impl, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Separately, we should make crate_universe automatically generate a suitable module extension. Nothing in this file is project-specific.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
run_targets: | ||
- "//third-party:vendor" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In CI this is succeeding on Linux ("bzlmod BCR presubmit on :ubuntu: 20.04 LTS (OpenJDK 11, gcc 9.4.0)") but failing on macOS ("bzlmod BCR presubmit on :darwin: (OpenJDK 11, Xcode)").
/private/var/tmp/_bazel_buildkite/4894c9dc50e84673867cbd80d0914a01/external/rules_rust~override~rust~rust_host_tools/bin/cargo: /private/var/tmp/_bazel_buildkite/4894c9dc50e84673867cbd80d0914a01/external/rules_rust~override~rust~rust_host_tools/bin/cargo: cannot execute binary file
ERROR: /Users/buildkite/builds/bk-imacpro-7/bazel/rules-rust-rustlang/examples/bzlmod/hello_world/third-party/BUILD.bazel:3:14: //third-party:vendor depends on @@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap//:binary in repository @@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap which failed to fetch. no such package '@@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap//': Process exited with code '126'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like it's trying to use a Linux cargo
binary to build cargo-bazel
- I repro locally... See #2021 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like the MODULE.bazel.lock
contains platform-specific information, which gets re-used if nothing else has changed in the repo:
"rust_host_tools": {
"bzlFile": "@@rules_rust~override//rust:repositories.bzl",
"ruleClassName": "rust_toolchain_tools_repository",
"attributes": {
"name": "rules_rust~override~rust~rust_host_tools",
"exec_triple": "x86_64-unknown-linux-gnu",
"target_triple": "x86_64-unknown-linux-gnu",
"allocator_library": "@rules_rust//ffi/cc/allocator_library",
"dev_components": false,
"edition": "2021",
"rustfmt_version": "nightly/2023-12-07",
"sha256s": {},
"urls": [
"https://static.rust-lang.org/dist/{}.tar.gz"
],
"version": "1.74.1"
}
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like we need to do something like this: https://github.com/bazelbuild/rules_python/pull/1433/files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good find. See also bazelbuild/rules_python#1643. Otherwise, according to bazelbuild/rules_python#1433 (comment), you and all downstream users need to regenerate their lockfiles separately from each platform they support. For example I, on Linux, would not be able to generate a lockfile that passes checks on macOS.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Started a thread on Slack: https://bazelbuild.slack.com/archives/C014RARENH0/p1703787329091939 - hopefully there's a recommendation we can follow :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested this in #2374 - it works, but looks like it's going to churn the lockfile every time someone switches platform. Hopefully we can get some useful pointers soon :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, for now let's just delete + .gitignore the MODULE.bazel.lock
file in examples/bzlmod/hello_world and be happy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe there's a way to ensure that you don't use the lockfile for specific module extensions / repository rules. Don't know how though. @Wyverald should know, he implemented the lockfile.
e4e83f3
to
afc7847
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
@matts1 Can you take a look when convenient and follow up with anything you'd like added/changed? :)
Aah sorry we accidentally merged something else while CI was churning - if you can do one more rebase we can get this merged! |
Head branch was pushed to by a user without write access
afc7847
to
6745da8
Compare
Without this, I have been unable to update https://github.com/dtolnay/cxx to bzlmod. When I run
bazel run //third-party:vendor
(this rule) it failed with "unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0":After this change, it works. My MODULE.bazel can be seen in dtolnay/cxx#1294.