Make and pkg-config toolchains are not reproducible #1313

nmattia · 2024-11-05T08:56:26Z

When using (a recent) rules_foreign_cc I'm getting determinism issues in the toolchains. In my case the checksum of both the make and pkgconfig binaries change every time they're built:

$ bazel build //my_c_target
...
$ sha256sum bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/bin/make
e00a81041a33af3ed7885449b88129c9f83661bcbec6cc84e51c73f77d93340f  bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/bin/make
$ bazel clean && bazel build //my_c_target
$ sha256sum bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/bin/make
95bf6a1b30b77689370e4e73a6f85a2684a6522db36d54d2744093d30532acb9  bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/bin/make

Note how each build produced a different make executable.

I've noticed that the build logs (Configure.log, BootstrapGNUMake.log, etc) introduce non-determinism too as they include full sandbox paths in their outputs (in commands and in "Can not copy X" lines).

The workaround for me is to use preinstalled toolchains:

register_toolchains(
    "@rules_foreign_cc//toolchains:preinstalled_pkgconfig_toolchain",
    "@rules_foreign_cc//toolchains:preinstalled_make_toolchain",
)

and remove build logs:

diff --git a/foreign_cc/private/framework.bzl b/foreign_cc/private/framework.bzl
index 33129b8..7326107 100644
--- a/foreign_cc/private/framework.bzl
+++ b/foreign_cc/private/framework.bzl
@@ -616,7 +616,7 @@ def wrap_outputs(ctx, lib_name, configure_name, script_text, env_prelude, build_
     cleanup_on_success_function = create_function(
         ctx,
         "cleanup_on_success",
-        "rm -rf $$BUILD_TMPDIR$$ $$EXT_BUILD_DEPS$$",
+        "rm -rf $$BUILD_TMPDIR$$ $$EXT_BUILD_DEPS$$ && echo > $$BUILD_LOG$$",
     )
     cleanup_on_failure_function = create_function(
         ctx,

The text was updated successfully, but these errors were encountered:

This includes a couple of fixes to make the replica build more deterministic. This in theory allows anyone building `//rs/replica` with Bazel in the Docker image to get a bit-for-bit reproducible replica executable, and this should also improve Bazel cache hit rates. This fixes for the following issues: * Non-reproducible `jemalloc` build: the `tikv-jemalloc-sys` crate vendors `jemalloc` and builds it as part of `build.rs`. Unfortunately that build in not deterministic. To make it more deterministic we build `jemalloc` separately, which also speeds up rebuilds as the C code does not need to be rebuilt when rust versions change. We only enable this in Linux; this also includes a patch to support this in `rules_rust`: bazelbuild/rules_rust#2981 * Non-reproducible `rules_rust` build log: this includes a backport of a fix that disables build logs in `rules_rust` by default (backported because our build is not compatible with the latest `rules_rust`) bazelbuild/rules_rust#2974 * Non-reproducible make & pkgconfig toolchains: some toolchains packaged by `rules_foreign_cc` cause build determinism issues so instead we use the ones installed in the container and remove build logs: bazel-contrib/rules_foreign_cc#1313 * Non-reproducible obj file generation in cc-rs: the `cc-rs` crate used in many C builds, including the (ASM) build of `ring`'s crypto bits, generates object files that include the Bazel sandbox full path: rust-lang/cc-rs#1271 * Non-reproducible codegen: `cranelift-isle` and `cranelift-codegen-meta` include references to source files as absolute paths that include the Bazel sandbox path: bytecodealliance/wasmtime#9553

jjmaestro · 2024-11-07T12:56:12Z

I actually tested this further, just to learn more about reproducible builds, and here's what I found!

I followed @nmattia's steps of building-cleaning-building a couple of times in one of the projects I'm working on that uses rules_foreign_cc
Each time, I collected info from the binary:
- readelf --all --string-dump=.rodata bazel-out/(...)/make
Then, I compared the readelf info with diff --brauN

--- logs/make1.readelf2 2024-11-07 12:36:54.673277847 +0000
+++ logs/make2.readelf2 2024-11-07 12:38:59.881880125 +0000
@@ -3691,7 +3691,7 @@
   [  3798]  unknown output-sync type '%s'
   [  37b8]  make
   [  37c0]  true
-  [  37c8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/share/locale
+  [  37c8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/71/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/share/locale
   [  38b0]  getcwd
   [  38b8]  .VARIABLES
   [  38c8]  .RECIPEPREFIX
@@ -3943,7 +3943,7 @@
   [  5788]  $(MAKEFILES)
   [  5798]  GNUmakefile
   [  57a8]  Makefile
-  [  57b8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/include
+  [  57b8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/71/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/include
   [  58a0]  /usr/gnu/include
   [  58b8]  /usr/local/include
   [  58d0]  /usr/include
@@ -3999,7 +3999,7 @@
   [  5fc0]  '%s' is up to date.
   [  5fd8]  /lib
   [  5fe0]  /usr/lib
-  [  5ff0]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/lib
+  [  5ff0]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/71/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/lib
   [  60d0]  (%.o)
   [  60d8]   | 
   [  60e0]   .WAIT
@@ -4123,7 +4123,7 @@
 Displaying notes found in: .note.gnu.build-id
   Owner                Data size       Description
   GNU                  0x00000014      NT_GNU_BUILD_ID (unique build ID bitstring)
-    Build ID: e194acfe778de03c63dca3eb94acd48df4370f97
+    Build ID: 7f851e5b2a8d6c29336feffa291b0ae5ac5e6d4d
 
 Displaying notes found in: .note.gnu.gold-version
   Owner                Data size       Description

The binaries have different build IDs which I think is caused by the differences in the paths stored in them.

Finally, note that the previous diff is a "nice one". If you are repeating these steps, you might also get a longer path caused by a longer sandbox ID which will bump the offsets and generate a much longer and harder to read diff, where the differences in strings, etc, are much harder to find. E.g., in the following case, (...)/sandbox/processwrapper-sandbox/10/(...) VS (...)/sandbox/processwrapper-sandbox/521/(...):

--- logs/make1.readelf2 2024-11-07 12:36:54.673277847 +0000
+++ logs/make2.readelf2 2024-11-07 12:40:13.279110434 +0000
@@ -53,10 +53,10 @@
   [14] .fini             PROGBITS         000000000002fa18  0002fa18
        0000000000000014  0000000000000000  AX       0     0     4
   [15] .rodata           PROGBITS         000000000002fa30  0002fa30
-       0000000000006afa  0000000000000000   A       0     0     16
-  [16] .eh_frame         PROGBITS         0000000000036530  00036530
+       0000000000006b0a  0000000000000000   A       0     0     16
+  [16] .eh_frame         PROGBITS         0000000000036540  00036540
        000000000000527c  0000000000000000   A       0     0     8
-  [17] .eh_frame_hdr     PROGBITS         000000000003b7ac  0003b7ac
+  [17] .eh_frame_hdr     PROGBITS         000000000003b7bc  0003b7bc
        0000000000000bb4  0000000000000000   A       0     0     4
   [18] .data.rel.ro[...] PROGBITS         000000000004f2d0  0003f2d0
        0000000000000420  0000000000000000  WA       0     0     16
@@ -103,14 +103,14 @@
                  0x000000000000001b 0x000000000000001b  R      0x1
       [Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
   LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
-                 0x000000000003c360 0x000000000003c360  R E    0x10000
+                 0x000000000003c370 0x000000000003c370  R E    0x10000
   LOAD           0x000000000003f2d0 0x000000000004f2d0 0x000000000004f2d0
                  0x00000000000023f8 0x0000000000005bf8  RW     0x10000
   DYNAMIC        0x000000000003f700 0x000000000004f700 0x000000000004f700
                  0x0000000000000210 0x0000000000000210  RW     0x8
   NOTE           0x0000000000000254 0x0000000000000254 0x0000000000000254
                  0x0000000000000044 0x0000000000000044  R      0x4
-  GNU_EH_FRAME   0x000000000003b7ac 0x000000000003b7ac 0x000000000003b7ac
+  GNU_EH_FRAME   0x000000000003b7bc 0x000000000003b7bc 0x000000000003b7bc

(...)

   1063: 0000000000009cd0     0 FUNC    LOCAL  HIDDEN    11 _init
@@ -3691,428 +3691,428 @@
   [  3798]  unknown output-sync type '%s'
   [  37b8]  make
   [  37c0]  true
-  [  37c8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/share/locale
-  [  38b0]  getcwd
-  [  38b8]  .VARIABLES
-  [  38c8]  .RECIPEPREFIX
-  [  38d8]  target-specific order-only second-expansion else-if shortest-stem undefine oneshell nocomment grouped-target extra-prereqs notintermediate shell-export archives jobserver jobserver-fifo output-sync check-symlink load
-  [  39b8]  .FEATURES

(...)

-  [  4160]  shuffle
-  [  4168]  jobserver-style
-  [  4178]    -b, -m                      Ignored for compatibility.\n
-  [  41b8]    -B, --always-make           Unconditionally make all targets.\n
-  [  4200]    -C DIRECTORY, --directory=DIRECTORY\n
+  [  37c8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/521/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/share/locale
+  [  38b8]  getcwd
+  [  38c0]  .VARIABLES
+  [  38d0]  .RECIPEPREFIX
+  [  38e0]  target-specific order-only second-expansion else-if shortest-stem undefine oneshell nocomment grouped-target extra-prereqs notintermediate shell-export archives jobserver jobserver-fifo output-sync check-symlink load
+  [  39c0]  .FEATURES

(...)

-  [  5768]  ...
-  [  5770]  Reading makefiles...\n
-  [  5788]  $(MAKEFILES)
-  [  5798]  GNUmakefile
-  [  57a8]  Makefile
-  [  57b8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/include
-  [  58a0]  /usr/gnu/include
-  [  58b8]  /usr/local/include
-  [  58d0]  /usr/include
-  [  58e0]  stat: 
-  [  58e8]  lstat: 

(...)

-  [  5f78]  Using default commands for '%s'.\n
-  [  5fa0]  Nothing to be done for '%s'.
-  [  5fc0]  '%s' is up to date.
-  [  5fd8]  /lib
-  [  5fe0]  /usr/lib
-  [  5ff0]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/10/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/lib
-  [  60d0]  (%.o)
-  [  60d8]   | 
-  [  60e0]   .WAIT
-  [  60e8]  warning: ignoring prerequisites on suffix rule definition
-  [  6129]  # Implicit Rules

(...)

+  [  5770]  ...
+  [  5778]  Reading makefiles...\n
+  [  5790]  $(MAKEFILES)
+  [  57a0]  GNUmakefile
+  [  57b0]  Makefile
+  [  57c0]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/521/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/include
+  [  58a8]  /usr/gnu/include
+  [  58c0]  /usr/local/include
+  [  58d8]  /usr/include
+  [  58e8]  stat: 
+  [  58f0]  lstat: 

(...)

+  [  5f80]  Using default commands for '%s'.\n
+  [  5fa8]  Nothing to be done for '%s'.
+  [  5fc8]  '%s' is up to date.
+  [  5fe0]  /lib
+  [  5fe8]  /usr/lib
+  [  5ff8]  /postgres/.cache/bazel/_bazel_postgres/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/521/execroot/_main/bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_foreign_cc~/toolchains/private/make/lib
+  [  60d8]  (%.o)
+  [  60e0]   | 
+  [  60e8]   .WAIT
+  [  60f0]  warning: ignoring prerequisites on suffix rule definition
+  [  6131]  # Implicit Rules

(...)

+  [  6a88]  tmpfile: %s
+  [  6b03]   
+  [  6b05]  "
 
 
 Displaying notes found in: .note.ABI-tag
@@ -4123,7 +4123,7 @@
 Displaying notes found in: .note.gnu.build-id
   Owner                Data size       Description
   GNU                  0x00000014      NT_GNU_BUILD_ID (unique build ID bitstring)
-    Build ID: e194acfe778de03c63dca3eb94acd48df4370f97
+    Build ID: b2af58606dd42b02f2128c8bd0a74875a674fac9
 
 Displaying notes found in: .note.gnu.gold-version
   Owner                Data size       Description

jjmaestro · 2024-11-07T16:55:18Z

@nmattia also, I've asked in Bazel's Slack, check this thread, TIL about not just the build ID that I mentioned in my previous comment but also GCC's file-prefix-map / debug-prefix-map and found:

a good resource regarding build path in the documentation section of Reproducible Builds website (there's a bunch of docs in there that are really great regarding reproducible builds).

a flag / option in rules_foreign_cc that I had missed! 😄

rules_foreign_cc/foreign_cc/private/framework.bzl

Lines 204 to 208 in 9edaefe

    
           "set_file_prefix_map": attr.bool( 
        
               doc = ( 
        
                   "Use -ffile-prefix-map with the intention to remove the sandbox path from " + 
        
                   "debug symbols" 
        
               ),

Maybe this flag could be used to make the make build reproducible... :-?

nmattia · 2024-11-26T15:56:55Z

Hi @jjmaestro, sorry for the silence, didn't have time to circle back to this until today.

I investigated a bit and found that it was probably not related to -ffile-prefix-map. While looking at the gnumake source (I'm using rules_foreign_cc version 0.12 which seems to pull gnumake version 4.4.1) it looks like some build-time values are included in the source, namely INCLUDEDIR, LIBDIR and LOCALEDIR.

It turns out that rules_foreign_cc already has a patch to make the make build reproducible, though it is not applied (and not applicable) for the default make version (4.4.1).

Note also that said patch was redefining the values in make's Makefile, though it's not clear to me now whether the makefile is still being used, or whether make's build.sh script is used instead (at least since this commit).

I've ported the original patch to make version 4.4.1 for both the Makefile and the build.sh script:

diff --git a/Makefile.am b/Makefile.am
index c29c2357..d9ff3569 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -74,11 +74,11 @@ make_LDADD =        $(LIBOBJS) $(_GUILE_LIBS) lib/libgnu.a $(GETLOADAVG_LIBS) \
 localedir =    $(datadir)/locale
 
 AM_CPPFLAGS =  -Isrc -I$(top_srcdir)/src -Ilib -I$(top_srcdir)/lib \
-               -DLIBDIR=\"$(libdir)\" -DLOCALEDIR=\"$(localedir)\"
+               -DLIBDIR=\".\" -DLOCALEDIR=\"external/rules_foreign_cc/toolchains/make/share/locale\"
 
 # If prefix is not a standard location, look in prefix as well
 if !KNOWN_PREFIX
-  AM_CPPFLAGS += -DINCLUDEDIR=\"$(includedir)\"
+  AM_CPPFLAGS += -DINCLUDEDIR=\"external/rules_foreign_cc/toolchains/make/include\"
 endif
 
 AM_CFLAGS =    $(_GUILE_CFLAGS)
diff --git a/build.sh b/build.sh
index 96e8f91c..3bf7b293 100755
--- a/build.sh
+++ b/build.sh
@@ -35,7 +35,7 @@ includedir=$prefix/include
 
 localedir=$prefix/share/locale
 
-defines="-DLOCALEDIR=\"$localedir\" -DLIBDIR=\"$libdir\" -DINCLUDEDIR=\"$includedir\""
+defines="-DLOCALEDIR=\"external/rules_foreign_cc/toolchains/make/share/locale\" -DLIBDIR=\".\" -DINCLUDEDIR=\"external/rules_foreign_cc/toolchains/make/include\""
 
 # Look up a make variable value.
 # It can handle simple recursion where variables are separate words.

Unfortunately this doesn't seem to fix the issue and the local sandbox path still appears in the final make executable; I'm not sure why and unfortunately I don't have time to dig deeper in make's build.

What does fix the issue is to replace the problematic values in the C files directly:

diff --git a/src/main.c b/src/main.c
index a9d3a644..404465a0 100644
--- a/src/main.c
+++ b/src/main.c
@@ -1245,7 +1245,7 @@ main (int argc, char **argv, char **envp)
   setlocale (LC_ALL, "");
   /* The cast to void shuts up compiler warnings on systems that
      disable NLS.  */
-  (void)bindtextdomain (PACKAGE, LOCALEDIR);
+  (void)bindtextdomain (PACKAGE, "/usr/share/locale");
   (void)textdomain (PACKAGE);
 
 #ifdef  POSIX
diff --git a/src/read.c b/src/read.c
index b0fc1e1f..c6ca9ec4 100644
--- a/src/read.c
+++ b/src/read.c
@@ -108,7 +108,7 @@ static const char *default_include_directories[] =
 # define INCLUDEDIR "."
 #endif
 #if defined(INCLUDEDIR)
-    INCLUDEDIR,
+    "/usr/include",
 #endif
 #ifndef _AMIGA
     "/usr/gnu/include",
diff --git a/src/remake.c b/src/remake.c
index fe67ab28..cab8d5df 100644
--- a/src/remake.c
+++ b/src/remake.c
@@ -1700,7 +1700,7 @@ library_search (const char *lib, FILE_TIMESTAMP *mtime_ptr)
  */
 #define LIBDIR "."
 #endif
-      LIBDIR,                   /* Defined by configuration.  */
+      "/usr/lib",                   /* Defined by configuration.  */
       0
     };

With this patch applied to make, and the rules_foreign_cc patch mentioned above to wipe the build logs, I'm getting a 100% reproducible make toolchain build.

jjmaestro · 2024-11-26T16:07:44Z

@nmattia wow, thanks!! I also didn't look much further and I'm sure I'll use this soon :D Thanks a lot for the findings!

nmattia mentioned this issue Nov 5, 2024

fix(IDX): make replica build more deterministic dfinity/ic#2441

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make and pkg-config toolchains are not reproducible #1313

Make and pkg-config toolchains are not reproducible #1313

nmattia commented Nov 5, 2024 •

edited

Loading

jjmaestro commented Nov 7, 2024

jjmaestro commented Nov 7, 2024

nmattia commented Nov 26, 2024

jjmaestro commented Nov 26, 2024

Make and pkg-config toolchains are not reproducible #1313

Make and pkg-config toolchains are not reproducible #1313

Comments

nmattia commented Nov 5, 2024 • edited Loading

jjmaestro commented Nov 7, 2024

jjmaestro commented Nov 7, 2024

nmattia commented Nov 26, 2024

jjmaestro commented Nov 26, 2024

nmattia commented Nov 5, 2024 •

edited

Loading