MacOS host paths being written into binaries

[roman-kashitsyn]

While experimenting with rules_rust build reproducibility, I found that the process_wrapper binary build is not deterministic.

How I reproduced the problem:
I created 2 identical directories on my macOS machine with the following content:

$ cat panic.rs 
fn main() {
    panic!("boom!")
}

$ cat WORKSPACE.bazel 
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_rust",
    sha256 = "6bfe75125e74155955d8a9854a8811365e6c0f3d33ed700bc17f39e32522c822",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_rust/releases/download/0.9.0/rules_rust-v0.9.0.tar.gz",
        "https://github.com/bazelbuild/rules_rust/releases/download/0.9.0/rules_rust-v0.9.0.tar.gz",
    ],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_register_toolchains(version = "1.60.0")

$ cat BUILD.bazel 
load("@rules_rust//rust:defs.bzl", "rust_binary")

rust_binary(
    name = "panic",
    srcs = ["panic.rs"],
    edition = "2018",
)

$ cat .bazelrc 
build --@rules_rust//:source_path_prefix=/source/
build --execution_log_json_file=bazel-build-log.json

Running bazel build :panic produced slightly different binaries.
An inspection of bazel-build-log.json file showed that process_wrapper had different hashes in the two workspaces:

***************
*** 509,513 ****
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib",
      "digest": {
!       "hash": "8532cb231d123c1a6f64e3129d11461c4062c1ee499ae37654445582d8dd3e15",
        "sizeBytes": "630696",
        "hashFunctionName": "SHA-256"
--- 509,513 ----
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib",
      "digest": {
!       "hash": "2b96b6712a77e791792f9be2d7de9ac89ffda7648724ef3b121071607201eb69",
        "sizeBytes": "630696",
        "hashFunctionName": "SHA-256"
***************
*** 915,919 ****
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper",
      "digest": {
!       "hash": "20cbe0042ca135fad3f79c7dc37176b2d9b274d29e2abf4c7707a5eda31fab88",
        "sizeBytes": "760720",
        "hashFunctionName": "SHA-256"
--- 915,919 ----
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper",
      "digest": {
!       "hash": "e32aa301edcfd8ff12584c886befc2cc2a6409a399679dcd91f46e3fc58dc0a9",
        "sizeBytes": "760720",
        "hashFunctionName": "SHA-256"
***************

The source of non-determinism is likely the libtinyjson.rlib file that has different file paths embedded into it:

The first workspace had path "/tmp/sandbox/darwin-sandbox/1/execroot/main/...", the second workspace had path "/private/var/tmp/_bazel_lifted/dba7...561c/sandbox/darwin-sandbox/1/execroot/main/...".

Comment from Rosica Dejanovska:

Looks like we need to temporarily bring back a tiny part of the old C++ process wrapper that does what --subst pwd=${pwd} does. Long term rust-lang/rust#89434 should save us.

[UebelAndre]

I wonder if #1563 would help with this.

[goffrie]

I wonder if #1563 would help with this.

I don't believe so - the issue is that --remap-path-prefix depends on process_wrapper to work, but when building process_wrapper we don't have access to itself so the remapping just doesn't do anything.

However, I believe including tinyjson directly into the process_wrapper crate would work around this issue.

[bazaglia]

I've run into something that might be related. When I run a build on my Ubuntu, but still do it remotely on RBE, it succeeds. On my pipeline machine, I'd expect to see it reusing actions pre-built just like it happens in all the other programming languages rules for Bazel I use. But what I'm seeing instead is my target trying to build again even when running against the same RBE. It complains I don't have clang in the RBE worker which surprised me to see it was able to build on my Ubuntu providing the same --remote_executor I provide in the pipeline and it succeeds then.

Adding clang to my worker fixed the error but I still don't fully understand why RBE isn't working as expected in rules_rust, causing my target to rebuild when it shouldn't. It is trying to rebuild process_wrapper so when searching related issues I found this issue mentioning it isn't deterministic.

Did someone find a workaround for that?

[227 / 815] [Prepa] Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool]
ERROR: /home/actions/.cache/bazel/_bazel_actions/1bab7e98308a5a2a9bf4653db430f9fe/external/rules_rust/util/process_wrapper/BUILD.bazel:6:36: Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool] failed: (Exit 1): rustc failed: error executing command (from target @rules_rust//util/process_wrapper:process_wrapper) 
  (cd /home/actions/.cache/bazel/_bazel_actions/1bab7e98308a5a2a9bf4653db430f9fe/execroot/main && \
  exec env - \
    CARGO_CFG_TARGET_ARCH=x86_64 \
    CARGO_CFG_TARGET_OS=linux \
    CARGO_CRATE_NAME=process_wrapper \
    CARGO_MANIFEST_DIR='${pwd}/external/rules_rust/util/process_wrapper' \
    CARGO_PKG_AUTHORS='' \
    CARGO_PKG_DESCRIPTION='' \
    CARGO_PKG_HOMEPAGE='' \
    CARGO_PKG_NAME=process_wrapper \
    CARGO_PKG_VERSION=0.0.0 \
    CARGO_PKG_VERSION_MAJOR=0 \
    CARGO_PKG_VERSION_MINOR=0 \
    CARGO_PKG_VERSION_PATCH=0 \
    CARGO_PKG_VERSION_PRE='' \
    PATH=/bin:/usr/bin:/usr/local/bin \
    SYSROOT=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain \
  bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain/bin/rustc external/rules_rust/util/process_wrapper/main.rs '--crate-name=process_wrapper' '--crate-type=bin' '--error-format=human' '--out-dir=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust/util/process_wrapper' '--codegen=opt-level=3' '--codegen=debuginfo=0' '--remap-path-prefix=${pwd}=' '--emit=dep-info,link' '--color=always' '--target=x86_64-unknown-linux-gnu' -L bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib '--edition=2018' '--codegen=linker=/usr/lib/llvm-11/bin/clang' --codegen 'link-args=-fuse-ld=/usr/bin/ld.gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib/llvm-11/bin -Wl,--gc-sections -ldl -lpthread -lstdc++ -lm' '--extern=tinyjson=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib' '-Ldependency=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust_tinyjson')
# Execution platform: @aspect_rules_js//platforms:x86_64_linux_remote
error: linker `/usr/lib/llvm-11/bin/clang` not found
  |
  = note: No such file or directory (os error 2)

[illicitonion]

@bazaglia I successfully have rules_rust working with an Ubuntu remote execution environment without clang pre-installed. My guess is you're missing some toolchain for your exec platform. Some things I'd look at for debugging:

1. The output of running with --toolchain_resolution_debug=.*rust.* - it's hard to read, but should tell you what toolchain you're ending up with on your exec platform and why.
2. Make sure you have --incompatible_enable_cc_toolchain_resolution enabled, have a relevant --extra_execution_platforms registered, and have an appropriate --extra_toolchains registered for your exec platform

[jfirebaugh]

However, I believe including tinyjson directly into the process_wrapper crate would work around this issue.

I attempted this solution. First I tried retaining the @rules_rust_tinyjson repository but using copy_file from skylib to copy individual source files at build time. This fails when building process_wrapper in an external repository:

ERROR: /private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/util/process_wrapper/BUILD.bazel:25:36: in rust_binary_without_process_wrapper rule @rules_rust//util/process_wrapper:process_wrapper: 
Traceback (most recent call last):
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 332, column 42, in _rust_binary_impl
                srcs, crate_root = _transform_sources(ctx, ctx.files.srcs, getattr(ctx.file, "crate_root", None))
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 172, column 68, in _transform_sources
                src_symlink = ctx.actions.declare_file(paths.relativize(src.short_path, package_root))
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/bazel_skylib/lib/paths.bzl", line 186, column 17, in _relativize
                fail("Path '%s' is not beneath '%s'" % (path, start))
Error in fail: Path '../rules_rust/util/process_wrapper/flags.rs' is not beneath 'util/process_wrapper'

An alternative could be to vendor tinyjson sources (i.e. check them into the repository) rather than trying to obtain them via a @rules_rust_tinyjson repository. Is that what you had in mind?

[jfirebaugh]

By the way, does anything prevent process_wrapper itself from being susceptible to the same issue?

[goffrie]

Yes, that's what I was thinking of anyway.

…

On Fri, Dec 23, 2022, 15:20 John Firebaugh ***@***.***> wrote: However, I believe including tinyjson directly into the process_wrapper crate would work around this issue. I attempted this solution. First I tried retaining the @rules_rust_tinyjson repository but using copy_file from skylib to copy individual source files at build time. This fails when building process_wrapper in an external repository: ERROR: /private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/util/process_wrapper/BUILD.bazel:25:36: in rust_binary_without_process_wrapper rule @rules_rust//util/process_wrapper:process_wrapper: Traceback (most recent call last): File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 332, column 42, in _rust_binary_impl srcs, crate_root = _transform_sources(ctx, ctx.files.srcs, getattr(ctx.file, "crate_root", None)) File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 172, column 68, in _transform_sources src_symlink = ctx.actions.declare_file(paths.relativize(src.short_path, package_root)) File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/bazel_skylib/lib/paths.bzl", line 186, column 17, in _relativize fail("Path '%s' is not beneath '%s'" % (path, start)) Error in fail: Path '../rules_rust/util/process_wrapper/flags.rs' is not beneath 'util/process_wrapper' An alternative could be to vendor tinyjson sources (i.e. check them into the repository) rather than trying to obtain them via a @rules_rust_tinyjson repository. Is that what you had in mind? — Reply to this email directly, view it on GitHub <#1530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJO2TXUESS65DQFCD6ZBETWOYCSTANCNFSM57XZ5Y6Q> . You are receiving this because you commented.Message ID: ***@***.***>

[goffrie]

By the way, does anything prevent process_wrapper itself from being susceptible to the same issue?

AFAIK rustc only inserts full file paths when instantiating cross-crate dependencies. It otherwise uses the exact paths provided on the command line, which are workspace-relative paths in our case. This could change in the future though.

[scentini]

I have #1728 which should address the issue. It's hard to write a test for it, but does someone volunteer to do two bazel builds (with a bazel clean inbetween) and diff the process wrapper binary and the tinyjson rlib on windows and osx?

[UebelAndre]

I think I've found this to be a larger issue than than the process wrapper. As of v0.23.0 I'm able to consistently reproduce binaries within the same checkout but I cannot across different checkouts.

When testing I get the following:

~ bazel clean && bazel build //util/process_wrapper && shasum -a 256 bazel-bin/util/process_wrapper/process_wrapper
b15575bbcc2e1ef2946ad164c36c547224a8c909395cf338eda53d694eab5602  bazel-bin/util/process_wrapper/process_wrapper
~ bazel clean && bazel build //util/process_wrapper && shasum -a 256 bazel-bin/util/process_wrapper/process_wrapper
b15575bbcc2e1ef2946ad164c36c547224a8c909395cf338eda53d694eab5602  bazel-bin/util/process_wrapper/process_wrapper

As well as (showing a binary built using the process wrapper)

~/Code/rules_rust bazel clean && bazel build //util/dir_zipper && shasum -a 256 bazel-bin/util/dir_zipper/dir_zipper
8224c6cf12ea72391210062dc0198ec3601fca28c19cc66fe7d7c72c365ca471  bazel-bin/util/dir_zipper/dir_zipper
~Code/rules_rust bazel clean && bazel build //util/dir_zipper && shasum -a 256 bazel-bin/util/dir_zipper/dir_zipper
8224c6cf12ea72391210062dc0198ec3601fca28c19cc66fe7d7c72c365ca471  bazel-bin/util/dir_zipper/dir_zipper

This to me communicates incremental builds are able to produce the same outputs. However, if I were to clone rules_rust to two different directories and compare binaries built there I get different outputs

~/Code shasum -a 256 rules_rust/bazel-bin/util/dir_zipper/dir_zipper
8224c6cf12ea72391210062dc0198ec3601fca28c19cc66fe7d7c72c365ca471  rules_rust/bazel-bin/util/dir_zipper/dir_zipper
~/Code shasum -a 256 rules_rust_2/bazel-bin/util/dir_zipper/dir_zipper
03a66dba1c82976f1c4f9021941d49cfc3edaf9f0c109f873c3b41963c8a928e  rules_rust_2/bazel-bin/util/dir_zipper/dir_zipper

If I do an xxd dump of each file, I find that the Bazel output is embedded in each binary!
(note that I sanitized /private/var/tmp/_bazel_user to all exclamation marks (!))

34379,34381c34379,34381
< 000864a0: 7269 7363 6f2f 3736 3238 3263 3636 6230  !!!!!/76282c66b0
< 000864b0: 6466 6533 6335 6362 3961 3233 3062 6463  dfe3c5cb9a230bdc
< 000864c0: 3931 3361 3532 2f65 7874 6572 6e61 6c2f  913a52/external/
---
> 000864a0: 7269 7363 6f2f 3262 6435 6633 3038 3038  !!!!!/2bd5f30808
> 000864b0: 6234 3531 3263 3830 3963 3561 6435 6130  b4512c809c5ad5a0
> 000864c0: 3539 3032 6532 2f65 7874 6572 6e61 6c2f  5902e2/external/
37589,37591c37589,37591
< 00092d40: 6973 636f 2f37 3632 3832 6336 3662 3064  !!!!/76282c66b0d
< 00092d50: 6665 3363 3563 6239 6132 3330 6264 6339  fe3c5cb9a230bdc9
< 00092d60: 3133 6135 322f 6578 7465 726e 616c 2f72  13a52/external/r
---
> 00092d40: 6973 636f 2f32 6264 3566 3330 3830 3862  !!!!/2bd5f30808b
> 00092d50: 3435 3132 6338 3039 6335 6164 3561 3035  4512c809c5ad5a05
> 00092d60: 3930 3265 322f 6578 7465 726e 616c 2f72  902e2/external/r
37651,37653c37651,37653
< 00093120: 6272 6973 636f 2f37 3632 3832 6336 3662  !!!!!!/76282c66b
< 00093130: 3064 6665 3363 3563 6239 6132 3330 6264  0dfe3c5cb9a230bd
< 00093140: 6339 3133 6135 322f 6578 7465 726e 616c  c913a52/external
---
> 00093120: 6272 6973 636f 2f32 6264 3566 3330 3830  !!!!!!/2bd5f3080
> 00093130: 3862 3435 3132 6338 3039 6335 6164 3561  8b4512c809c5ad5a
> 00093140: 3035 3930 3265 322f 6578 7465 726e 616c  05902e2/external
37699,37701c37699,37701
< 00093420: 6562 7269 7363 6f2f 3736 3238 3263 3636  !!!!!!!/76282c66
< 00093430: 6230 6466 6533 6335 6362 3961 3233 3062  b0dfe3c5cb9a230b
< 00093440: 6463 3931 3361 3532 2f65 7874 6572 6e61  dc913a52/externa
---
> 00093420: 6562 7269 7363 6f2f 3262 6435 6633 3038  !!!!!!!/2bd5f308
> 00093430: 3038 6234 3531 3263 3830 3963 3561 6435  08b4512c809c5ad5
> 00093440: 6130 3539 3032 6532 2f65 7874 6572 6e61  a05902e2/externa
38121,38123c38121,38123
< 00094e80: 6472 6562 7269 7363 6f2f 3736 3238 3263  !!!!!!!!!/76282c
< 00094e90: 3636 6230 6466 6533 6335 6362 3961 3233  66b0dfe3c5cb9a23
< 00094ea0: 3062 6463 3931 3361 3532 2f65 7874 6572  0bdc913a52/exter
---
> 00094e80: 6472 6562 7269 7363 6f2f 3262 6435 6633  !!!!!!!!!/2bd5f3
> 00094e90: 3038 3038 6234 3531 3263 3830 3963 3561  0808b4512c809c5a
> 00094ea0: 6435 6130 3539 3032 6532 2f65 7874 6572  d5a05902e2/exter
38148,38150c38148,38150
< 00095030: 656c 5f61 6e64 7265 6272 6973 636f 2f37  !!!!!!!!!!!!!!/7
< 00095040: 3632 3832 6336 3662 3064 6665 3363 3563  6282c66b0dfe3c5c
< 00095050: 6239 6132 3330 6264 6339 3133 6135 322f  b9a230bdc913a52/
---
> 00095030: 656c 5f61 6e64 7265 6272 6973 636f 2f32  !!!!!!!!!!!!!!/2
> 00095040: 6264 3566 3330 3830 3862 3435 3132 6338  bd5f30808b4512c8
> 00095050: 3039 6335 6164 3561 3035 3930 3265 322f  09c5ad5a05902e2/
38284,38286c38284,38286
< 000958b0: 7265 6272 6973 636f 2f37 3632 3832 6336  !!!!!!!!/76282c6
< 000958c0: 3662 3064 6665 3363 3563 6239 6132 3330  6b0dfe3c5cb9a230
< 000958d0: 6264 6339 3133 6135 322f 6578 7465 726e  bdc913a52/extern
---
> 000958b0: 7265 6272 6973 636f 2f32 6264 3566 3330  !!!!!!!!/2bd5f30
> 000958c0: 3830 3862 3435 3132 6338 3039 6335 6164  808b4512c809c5ad
> 000958d0: 3561 3035 3930 3265 322f 6578 7465 726e  5a05902e2/extern
39091,39093c39091,39093
< 00098b20: 0ae3 64fc e432 f3b0 9ed1 5a6b 42b8 ccf8  ..d..2....ZkB...
< 00098b30: 83cb bcc1 c04e 4e67 3bb3 65ed ba54 0f43  .....NNg;.e..T.C
< 00098b40: 5fd7 f326 c5dc 3897 46b2 3e3f 744e 16f5  _..&..8.F.>?tN..
---
> 00098b20: 0ae3 64fc e432 f3d4 b6a5 17ad c586 10c4  ..d..2..........
> 00098b30: b515 7daa 1e76 bbff e44c b040 ba77 f877  ..}..v...L.@.w.w
> 00098b40: 8051 856f 34b8 0097 46b2 3e3f 744e 16f5  .Q.o4...F.>?tN..
39115,39123c39115,39123
< 00098ca0: 0b02 b312 4c9b 48d0 90a8 8d9d 88c1 f969  ....L.H........i
< 00098cb0: 3b0f 5259 1e1e 0e7d 869b cccc cadf 750a  ;.RY...}......u.
< 00098cc0: aada 93f5 af8f b37b be77 7b91 f87d ba48  .......{.w{..}.H
< 00098cd0: 924d c755 9158 06fa 669e b80a 70f7 f389  .M.U.X..f...p...
< 00098ce0: a680 77e8 362a 20a1 61dc fc50 4f40 71e7  ..w.6* .a..PO@q.
< 00098cf0: 50e3 2ca4 7221 e2ee b700 98a3 1dfa dbf0  P.,.r!..........
< 00098d00: aec0 2032 e1cd 32a3 e3ef 4566 b18d b830  .. 2..2...Ef...0
< 00098d10: f463 36cc 6c9f d6a3 60c9 2207 ac9f 4c67  .c6.l...`."...Lg
< 00098d20: 0d4b e857 a9e7 7119 4e9a c4c8 1ecb 32ef  .K.W..q.N.....2.
---
> 00098ca0: 0b02 b312 4c9b 48e0 403f 4d34 b3c7 7b18  ....L.H.@?M4..{.
> 00098cb0: 3fa4 8933 6468 9d58 1979 0632 e1fa 96a8  ?..3dh.X.y.2....
> 00098cc0: 88b7 1f65 cd72 ddd8 c3d0 9cb1 6648 b4d6  ...e.r......fH..
> 00098cd0: 6f20 67da e3c5 5a6f 3785 fb9a 6c7a b339  o g...Zo7...lz.9
> 00098ce0: dbfc 1694 9707 139e e2ea a74a 4ddd bb90  ...........JM...
> 00098cf0: c878 09f5 c42f 17aa 12a3 94ce 092e db31  .x.../.........1
> 00098d00: e11b b94e 5790 d5ec 0059 445c c246 e3b2  ...NW....YD\.F..
> 00098d10: 5e5b 20c0 febe 14d2 d9fe 4899 1fc3 641e  ^[ .......H...d.
> 00098d20: cb45 3d84 5f0b 5b19 4e9a c4c8 1ecb 32ef  .E=._.[.N.....2.

These match the output bases of each workspace.

~/Code/rules_rust bazel info output_base
/private/var/tmp/_bazel_user/76282c66b0dfe3c5cb9a230bdc913a52
~/Code/rules_rust bazel info output_base
/private/var/tmp/_bazel_user/2bd5f30808b4512c809c5ad5a05902e2

It's worth noting that the sandbox doesn't ever appear in the binary though (e.g. /private/var/tmp/_bazel_user/76282c66b0dfe3c5cb9a230bdc913a52/sandbox/darwin-sandbox/1111/execroot), it's only the output_base

00093090: 5573 6572 732f 7275 6e6e 6572 2f2e 6361  Users/runner/.ca
000930a0: 7267 6f2f 7265 6769 7374 7279 2f73 7263  rgo/registry/src
000930b0: 2f69 6e64 6578 2e63 7261 7465 732e 696f  /index.crates.io
000930c0: 2d36 6631 3764 3232 6262 6131 3530 3031  -6f17d22bba15001
000930d0: 662f 6d65 6d63 6872 2d32 2e35 2e30 2f73  f/memchr-2.5.0/s
000930e0: 7263 2f6c 6962 2e72 732f 402f 006d 656d  rc/lib.rs/@/.mem
000930f0: 6368 722e 3430 3638 3964 3763 2d63 6775  chr.40689d7c-cgu
00093100: 2e30 002f 7072 6976 6174 652f 7661 722f  .0.!!!!!!!!!!!!!
00093110: 746d 702f 5f62 617a 656c 5f61 6e64 7265  !!!!!!!!!!!!!!!!
00093120: 6272 6973 636f 2f37 3632 3832 6336 3662  !!!!!!/76282c66b
00093130: 3064 6665 3363 3563 6239 6132 3330 6264  0dfe3c5cb9a230bd
00093140: 6339 3133 6135 322f 6578 7465 726e 616c  c913a52/external
00093150: 2f72 7573 745f 6461 7277 696e 5f61 6172  /rust_darwin_aar
00093160: 6368 3634 5f5f 6161 7263 6836 342d 6170  ch64__aarch64-ap
00093170: 706c 652d 6461 7277 696e 5f5f 7374 6162  ple-darwin__stab
00093180: 6c65 5f74 6f6f 6c73 2f6c 6962 2f72 7573  le_tools/lib/rus
00093190: 746c 6962 2f61 6172 6368 3634 2d61 7070  tlib/aarch64-app
000931a0: 6c65 2d64 6172 7769 6e2f 6c69 622f 6c69  le-darwin/lib/li
000931b0: 626d 656d 6368 722d 3133 3963 3639 3731  bmemchr-139c6971
000931c0: 3833 6465 3561 3435 2e72 6c69 6228 6d65  83de5a45.rlib(me
000931d0: 6d63 6872 2d31 3339 6336 3937 3138 3364  mchr-139c697183d
000931e0: 6535 6134 352e 6d65 6d63 6872 2e34 3036  e5a45.memchr.406
000931f0: 3839 6437 632d 6367 752e 302e 7263 6775  89d7c-cgu.0.rcgu
00093200: 2e6f 2900 5f5f 5a4e 366d 656d 6368 7236  .o).__ZN6memchr6
00093210: 6d65 6d63 6872 3866 616c 6c62 6163 6b36  memchr8fallback6
00093220: 6d65 6d63 6872 3137 6833 3263 6464 3331  memchr17h32cdd31
00093230: 6339 3964 3930 6331 3245 0073 7263 2f6d  c99d90c12E.src/m
00093240: 656d 6368 722f 6661 6c6c 6261 636b 2e72  emchr/fallback.r
00093250: 7300 2f72 7573 7463 2f39 3063 3534 3138  s./rustc/90c5418
00093260: 3036 6632 3361 3132 3730 3032 6465 3562  06f23a127002de5b
00093270: 3430 3338 6265 3733 3162 6131 3435 3863  4038be731ba1458c
00093280: 612f 6c69 6272 6172 792f 636f 7265 2f73  a/library/core/s
00093290: 7263 2f70 7472 2f63 6f6e 7374 5f70 7472  rc/ptr/const_ptr

The subcommand that produced this binary looks like:

SUBCOMMAND: # //util/dir_zipper:dir_zipper [action 'Compiling Rust bin dir_zipper (1 files)', configuration: 354d65174a36a8d7834dd9afb47309a01d9dcf8cb757a4b94f76317707c733a7, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_user/76282c66b0dfe3c5cb9a230bdc913a52/execroot/rules_rust && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=13.3 \
    CARGO_CFG_TARGET_ARCH=aarch64 \
    CARGO_CFG_TARGET_OS=darwin \
    CARGO_CRATE_NAME=dir_zipper \
    CARGO_MANIFEST_DIR='${pwd}/util/dir_zipper' \
    CARGO_PKG_AUTHORS='' \
    CARGO_PKG_DESCRIPTION='' \
    CARGO_PKG_HOMEPAGE='' \
    CARGO_PKG_NAME=dir_zipper \
    CARGO_PKG_VERSION=0.0.0 \
    CARGO_PKG_VERSION_MAJOR=0 \
    CARGO_PKG_VERSION_MINOR=0 \
    CARGO_PKG_VERSION_PATCH=0 \
    CARGO_PKG_VERSION_PRE='' \
    SYSROOT=bazel-out/darwin_arm64-fastbuild/bin/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain \
    XCODE_VERSION_OVERRIDE=14.3.1.14E300c \
    ZERO_AR_DATE=1 \
  bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- bazel-out/darwin_arm64-fastbuild/bin/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain/bin/rustc util/dir_zipper/dir_zipper.rs '--crate-name=dir_zipper' '--crate-type=bin' '--error-format=human' '--out-dir=bazel-out/darwin_arm64-fastbuild/bin/util/dir_zipper' '--codegen=opt-level=0' '--codegen=debuginfo=0' '--remap-path-prefix=${pwd}=' '--emit=dep-info,link' '--color=always' '--target=aarch64-apple-darwin' -L bazel-out/darwin_arm64-fastbuild/bin/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain/lib/rustlib/aarch64-apple-darwin/lib '--edition=2018' '--codegen=linker=external/local_config_cc/cc_wrapper.sh' --codegen 'link-args=-lc++ -fobjc-link-runtime -headerpad_max_install_names -no-canonical-prefixes -target arm64-apple-macosx13.3 -Xlinker -no_deduplicate -lc++ -target arm64-apple-macosx13.3')
# Configuration: 354d65174a36a8d7834dd9afb47309a01d9dcf8cb757a4b94f76317707c733a7
# Execution platform: @local_config_platform//:host

I'm not sure where this embedded data is coming from or if rust is somehow escaping the sandbox but this to me is a major issue as it's virtually guaranteed that two developers will not be sharing the same output_base which means Rust targets are going to be cache missing and largely unreproducible depending on the CI solution.

Any insight here would be hugely appreicated

cc @illicitonion @scentini @krasimirgg

[bsilver8192]

My experience with caching Rust builds in CI is that it actually works ok @UebelAndre, because process_wrapper itself gets cached with an arbitrary absolute sandbox path embedded, and then that cached result is used everywhere. It's not ideal, but it's not a showstopper as long as all builds are performed with the cache.

It does mean that if the action is ever re-run and produces a result which should be identical, it instead invalidates all the other cached results though. Also if you ever build without the cache, which seems like a possibility for developers, then things won't work.

[UebelAndre]

I don't think the process wrapper is really the core issue anymore. It's an issue of general determinism in the rules. There's for sure band-aids that have helped mask this but I think this is a pretty major violation of one of the core principles of Bazel. For some of the projects I work on where we care immensely about determinism this would be considered a major issue. If anyone knows why those values are being written I'd love to know and want to work toward a fix.

[UebelAndre]

This may be resolved by rust-lang/rust#112586

[UebelAndre]

This may be resolved by rust-lang/rust#112586

This seems to only impact windows.

[UebelAndre]

I dug into this on linux as well and found that linux is actually reproducible. I'm only seeing this issue on MacOS (since I don't have a windows machine to test on).

@illicitonion any suggestions on how to dig into MacOS binaries? I tried to see what section the host paths were being written to but I think I'm just failing to use otool

[illicitonion]

My debugging so far:

1. Build the two binaries and diff them - see what addresses the diffs are in.
2. otool -l /path/to/binary shows the addresses containing diffs are in the __LINKEDIT section
   I don't yet have any useful tips for debugging within that...

However, it looks like the absolute path is one which isn't stripped by our --remap-path-prefix flag?

My build appears to be happening in /private/var/tmp/_bazel_dwagnerhall/eadb11064330d11bc8ecdf193739bcea/execroot/rules_rust, we set --remap-path-prefix=${pwd}=, but the paths I'm seeing getting embedded are things like /private/var/tmp/_bazel_dwagnerhall/eadb11064330d11bc8ecdf193739bcea/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/lib/rustlib/aarch64-apple-darwin/lib/libstd-2ca50e2550a11ed4.rlib - it's possible if we did stripping based on output_base rather than execution_root, things would start working better?

[UebelAndre]

2. otool -l /path/to/binary shows the addresses containing diffs are in the __LINKEDIT section
I don't yet have any useful tips for debugging within that...

Where do you see this output? I think the thing I'm looking for is a path to bazel-out. All I see in otool -l is things like name /usr/lib/libSystem.B.dylib (offset 24) which IMO are acceptable.

I've also tried hacking in an explicit --remap-path-prefix to delete the output_base and it was still being written.

diff --git a/rust/private/rustc.bzl b/rust/private/rustc.bzl
index 9f3b0c26..245d870c 100644
--- a/rust/private/rustc.bzl
+++ b/rust/private/rustc.bzl
@@ -934,6 +934,7 @@ def construct_arguments(
     # For determinism to help with build distribution and such
     if remap_path_prefix != None:
         rustc_flags.add("--remap-path-prefix=${{pwd}}={}".format(remap_path_prefix))
+        rustc_flags.add("--remap-path-prefix=/private/var/tmp/_bazel_user/76282c66b0dfe3c5cb9a230bdc913a52={}".format(remap_path_prefix))

     if emit:
         rustc_flags.add("--emit=" + ",".join(emit_with_paths))

This still results in that path being found in my xxd dumps. (Note that that new abs path is the result of bazel info output_base)

[UebelAndre]

rust-lang/rust#116948 has some very useful info