Make mtime of reproducible tarballs dependent on git commit #127050

Kobzol · 2024-06-27T17:11:27Z

Since #123246, our tarballs should be fully reproducible. That means that the mtime of all files and directories in the tarballs is set to the date of the first Rust commit (from 2006). However, this is causing some mtime invalidation issues (#125578 (comment)).

Ideally, we would like to keep the mtime reproducible, but still update it with new versions of Rust. That's what this PR does. It modifies the tarball installer bootstrap invocation so that if the current rustc directory is managed by git, we will set the UTC timestamp of the latest commit as the mtime for all files in the archive. This means that the archive should be still fully reproducible from a given commit SHA, but it will also be changed with new beta bumps and download-rustc versions.

Note that only files are set to this mtime, directories are still set to the year 2006, because the tar library used by rust-installer doesn't allow us to selectively override mtime for directories (or at least I haven't found it). We could work around that by doing all the mtime modifications in bootstrap, but that would require more changes. I think/hope that just modifying the file mtimes should be enough. It should at least fix cargo rustc mtime invalidation.

Fixes: #125578

r? @onur-ozkan

try-job: x86_64-gnu-distcheck

Kobzol · 2024-06-27T17:12:14Z

CC @ehuss

onur-ozkan

Good solution! Just 2 small notes and LGTM other than that.

src/tools/rust-installer/src/tarballer.rs

src/bootstrap/src/utils/tarball.rs

Kobzol · 2024-06-28T13:34:07Z

Thanks, performed the changes.

onur-ozkan · 2024-06-28T14:47:01Z

@bors r+

bors · 2024-06-28T14:47:03Z

📌 Commit 50350f4 has been approved by onur-ozkan

It is now in the queue for this repository.

bors · 2024-06-29T12:37:55Z

☔ The latest upstream changes (presumably #127111) made this pull request unmergeable. Please resolve the merge conflicts.

jieyouxu · 2024-06-29T13:02:58Z

@bors r- (needs rebase)

Kobzol · 2024-06-29T13:08:46Z

Rebased.

@bors r=onur-ozkan

bors · 2024-06-29T13:08:48Z

📌 Commit c54a2a5 has been approved by onur-ozkan

It is now in the queue for this repository.

…llaumeGomez Rollup of 7 pull requests Successful merges: - rust-lang#126805 (Migrate `pdb-alt-path`, `mismatching-target-triples` and `mingw-export-call-convention` `run-make` tests to rmake) - rust-lang#126995 (Migrate `pretty-print-with-dep-file`, `pretty-print-to-file` and `libtest-padding` `run-make` tests to rmake) - rust-lang#127041 (Migrate `run-make/override-aliased-flags` to `rmake.rs`) - rust-lang#127072 (docs: say "includes" instead of "does include") - rust-lang#127073 (Remove unnecessary SeqCst in `impl fmt::Pointer for AtomicPtr`) - rust-lang#127112 (Bootstrap: Don't get output if `lldb --version` errors) - rust-lang#127116 (Migrate `run-make/return-non-c-like-enum` to `rmake.rs`) Failed merges: - rust-lang#127050 (Make mtime of reproducible tarballs dependent on git commit) r? `@ghost` `@rustbot` modify labels: rollup

…-ozkan Make mtime of reproducible tarballs dependent on git commit Since rust-lang#123246, our tarballs should be fully reproducible. That means that the mtime of all files and directories in the tarballs is set to the date of the first Rust commit (from 2006). However, this is causing some mtime invalidation issues (rust-lang#125578 (comment)). Ideally, we would like to keep the mtime reproducible, but still update it with new versions of Rust. That's what this PR does. It modifies the tarball installer bootstrap invocation so that if the current rustc directory is managed by git, we will set the UTC timestamp of the latest commit as the mtime for all files in the archive. This means that the archive should be still fully reproducible from a given commit SHA, but it will also be changed with new beta bumps and `download-rustc` versions. Note that only files are set to this mtime, directories are still set to the year 2006, because the `tar` library used by `rust-installer` doesn't allow us to selectively override mtime for directories (or at least I haven't found it). We could work around that by doing all the mtime modifications in bootstrap, but that would require more changes. I think/hope that just modifying the file mtimes should be enough. It should at least fix cargo `rustc` mtime invalidation. Fixes: rust-lang#125578 r? `@onur-ozkan`

…iaskrgr Rollup of 9 pull requests Successful merges: - rust-lang#123237 (Various rustc_codegen_ssa cleanups) - rust-lang#126960 (Improve error message in tidy) - rust-lang#127002 (Implement `x perf` as a separate tool) - rust-lang#127050 (Make mtime of reproducible tarballs dependent on git commit) - rust-lang#127081 (Add a run-make test that LLD is not being used by default on the x64 beta/stable channel) - rust-lang#127106 (Improve unsafe extern blocks diagnostics) - rust-lang#127110 (Fix a error suggestion for E0121 when using placeholder _ as return types on function signature.) - rust-lang#127114 (fix: prefer `(*p).clone` to `p.clone` if the `p` is a raw pointer) - rust-lang#127118 (Show `used attribute`'s kind for user when find it isn't applied to a `static` variable.) r? `@ghost` `@rustbot` modify labels: rollup

matthiaskrgr · 2024-06-29T20:10:22Z

@bors r-
#127125 (comment)

…ller commands

bors · 2024-06-29T21:35:26Z

⌛ Trying commit e1999f9 with merge 16deff3...

bors · 2024-06-29T22:46:44Z

☀️ Try build successful - checks-actions
Build commit: 16deff3 (16deff398341dea5954d6cedf3be010fe52c1ed6)

Kobzol · 2024-06-30T05:10:42Z

@rustbot ready

onur-ozkan · 2024-06-30T05:12:29Z

Nothing seem suspicious/wrong to me.

@bors r+

bors · 2024-06-30T05:12:32Z

📌 Commit e1999f9 has been approved by onur-ozkan

It is now in the queue for this repository.

bors · 2024-07-02T02:43:14Z

⌛ Testing commit e1999f9 with merge 98108d2...

…zkan Make mtime of reproducible tarballs dependent on git commit Since rust-lang#123246, our tarballs should be fully reproducible. That means that the mtime of all files and directories in the tarballs is set to the date of the first Rust commit (from 2006). However, this is causing some mtime invalidation issues (rust-lang#125578 (comment)). Ideally, we would like to keep the mtime reproducible, but still update it with new versions of Rust. That's what this PR does. It modifies the tarball installer bootstrap invocation so that if the current rustc directory is managed by git, we will set the UTC timestamp of the latest commit as the mtime for all files in the archive. This means that the archive should be still fully reproducible from a given commit SHA, but it will also be changed with new beta bumps and `download-rustc` versions. Note that only files are set to this mtime, directories are still set to the year 2006, because the `tar` library used by `rust-installer` doesn't allow us to selectively override mtime for directories (or at least I haven't found it). We could work around that by doing all the mtime modifications in bootstrap, but that would require more changes. I think/hope that just modifying the file mtimes should be enough. It should at least fix cargo `rustc` mtime invalidation. Fixes: rust-lang#125578 r? `@onur-ozkan` try-job: x86_64-gnu-distcheck

bors · 2024-07-02T02:45:31Z

💔 Test failed - checks-actions

rust-log-analyzer · 2024-07-02T02:51:39Z

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#14 0.347 
#14 0.347 gzip: stdin: not in gzip format
#14 0.347 tar: Child returned status 1
#14 0.348 tar: Error is not recoverable: exiting now
#14 ERROR: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
 > [6/8] RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh:
0.036 + mkdir netbsd

100   245  100   245    0     0    814      0 --:--:-- --:--:-- --:--:--   816
---
  11 | >>> RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh
  12 |     
  13 |     COPY scripts/sccache.sh /scripts/
--------------------
ERROR: failed to solve: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
#0 building with "brave_hopper" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 988B done
---
#11 0.653 
#11 0.653 gzip: stdin: not in gzip format
#11 0.654 tar: Child returned status 1
#11 0.654 tar: Error is not recoverable: exiting now
#11 ERROR: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
 > [6/8] RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh:
0.033 + mkdir netbsd

100   245  100   245    0     0    400      0 --:--:-- --:--:-- --:--:--   399
---
  11 | >>> RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh
  12 |     
  13 |     COPY scripts/sccache.sh /scripts/
--------------------
ERROR: failed to solve: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
#0 building with "brave_hopper" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 988B done
---
#11 0.319 
#11 0.319 gzip: stdin: not in gzip format
#11 0.319 tar: Child returned status 1
#11 0.319 tar: Error is not recoverable: exiting now
#11 ERROR: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
 > [6/8] RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh:

100   245  100   245    0     0    887      0 --:--:-- --:--:-- --:--:--   890
0.319 
---
  11 | >>> RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh
  12 |     
  13 |     COPY scripts/sccache.sh /scripts/
--------------------
ERROR: failed to solve: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
#0 building with "brave_hopper" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 988B done
---
#11 0.392 
#11 0.392 gzip: stdin: not in gzip format
#11 0.392 tar: Child returned status 1
#11 0.392 tar: Error is not recoverable: exiting now
#11 ERROR: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
 > [6/8] RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh:

100   245  100   245    0     0    696      0 --:--:-- --:--:-- --:--:--   698
0.392 
---
  11 | >>> RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh
  12 |     
  13 |     COPY scripts/sccache.sh /scripts/
--------------------
ERROR: failed to solve: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
#0 building with "brave_hopper" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 988B done
---
#11 0.333 
#11 0.333 gzip: stdin: not in gzip format
#11 0.333 tar: Child returned status 1
#11 0.333 tar: Error is not recoverable: exiting now
#11 ERROR: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
 > [6/8] RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh:

100   245  100   245    0     0    836      0 --:--:-- --:--:-- --:--:--   836
0.333 
---
  11 | >>> RUN env HOST_CFLAGS="-O -fcommon" /tmp/build-netbsd-toolchain.sh
  12 |     
  13 |     COPY scripts/sccache.sh /scripts/
--------------------
ERROR: failed to solve: process "/bin/sh -c env HOST_CFLAGS=\"-O -fcommon\" /tmp/build-netbsd-toolchain.sh" did not complete successfully: exit code: 2
##[error]Process completed with exit code 1.
Post job cleanup.

compiler-errors · 2024-07-02T03:02:21Z

@bors retry

…-ozkan Make mtime of reproducible tarballs dependent on git commit Since rust-lang#123246, our tarballs should be fully reproducible. That means that the mtime of all files and directories in the tarballs is set to the date of the first Rust commit (from 2006). However, this is causing some mtime invalidation issues (rust-lang#125578 (comment)). Ideally, we would like to keep the mtime reproducible, but still update it with new versions of Rust. That's what this PR does. It modifies the tarball installer bootstrap invocation so that if the current rustc directory is managed by git, we will set the UTC timestamp of the latest commit as the mtime for all files in the archive. This means that the archive should be still fully reproducible from a given commit SHA, but it will also be changed with new beta bumps and `download-rustc` versions. Note that only files are set to this mtime, directories are still set to the year 2006, because the `tar` library used by `rust-installer` doesn't allow us to selectively override mtime for directories (or at least I haven't found it). We could work around that by doing all the mtime modifications in bootstrap, but that would require more changes. I think/hope that just modifying the file mtimes should be enough. It should at least fix cargo `rustc` mtime invalidation. Fixes: rust-lang#125578 r? `@onur-ozkan` try-job: x86_64-gnu-distcheck

…iaskrgr Rollup of 9 pull requests Successful merges: - rust-lang#126403 (Actually report normalization-based type errors correctly for alias-relate obligations in new solver) - rust-lang#126803 (Change `asm-comments` to `verbose-asm`, always emit user comments) - rust-lang#126917 (Disable rmake test `inaccessible-temp-dir` on riscv64) - rust-lang#127050 (Make mtime of reproducible tarballs dependent on git commit) - rust-lang#127145 (Add `as_lang_item` to `LanguageItems`, new trait solver) - rust-lang#127184 (More refactorings to rustc_interface) - rust-lang#127202 (Remove global error count checks from typeck) - rust-lang#127233 (Some parser cleanups) - rust-lang#127245 (Add a test for `generic_const_exprs`) r? `@ghost` `@rustbot` modify labels: rollup

…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#126803 (Change `asm-comments` to `verbose-asm`, always emit user comments) - rust-lang#127050 (Make mtime of reproducible tarballs dependent on git commit) - rust-lang#127145 (Add `as_lang_item` to `LanguageItems`, new trait solver) - rust-lang#127202 (Remove global error count checks from typeck) - rust-lang#127233 (Some parser cleanups) - rust-lang#127248 (Add parse fail test using safe trait/impl trait) - rust-lang#127264 (Small `run-make-support` API improvements) - rust-lang#127270 (bootstrap: pass correct struct size to winapi) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#127050 - Kobzol:reproducibility-git, r=onur-ozkan Make mtime of reproducible tarballs dependent on git commit Since rust-lang#123246, our tarballs should be fully reproducible. That means that the mtime of all files and directories in the tarballs is set to the date of the first Rust commit (from 2006). However, this is causing some mtime invalidation issues (rust-lang#125578 (comment)). Ideally, we would like to keep the mtime reproducible, but still update it with new versions of Rust. That's what this PR does. It modifies the tarball installer bootstrap invocation so that if the current rustc directory is managed by git, we will set the UTC timestamp of the latest commit as the mtime for all files in the archive. This means that the archive should be still fully reproducible from a given commit SHA, but it will also be changed with new beta bumps and `download-rustc` versions. Note that only files are set to this mtime, directories are still set to the year 2006, because the `tar` library used by `rust-installer` doesn't allow us to selectively override mtime for directories (or at least I haven't found it). We could work around that by doing all the mtime modifications in bootstrap, but that would require more changes. I think/hope that just modifying the file mtimes should be enough. It should at least fix cargo `rustc` mtime invalidation. Fixes: rust-lang#125578 r? ``@onur-ozkan`` try-job: x86_64-gnu-distcheck

rustbot assigned onur-ozkan Jun 27, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 27, 2024

onur-ozkan reviewed Jun 28, 2024

View reviewed changes

src/tools/rust-installer/src/tarballer.rs Outdated Show resolved Hide resolved

src/bootstrap/src/utils/tarball.rs Outdated Show resolved Hide resolved

Kobzol force-pushed the reproducibility-git branch from 96da726 to 50350f4 Compare June 28, 2024 13:33

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 28, 2024

GuillaumeGomez mentioned this pull request Jun 29, 2024

Rollup of 7 pull requests #127121

Merged

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 29, 2024

Make mtime of reproducible tarball dependent on git commit

c54a2a5

Kobzol force-pushed the reproducibility-git branch from 50350f4 to c54a2a5 Compare June 29, 2024 13:08

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 29, 2024

matthiaskrgr mentioned this pull request Jun 29, 2024

Rollup of 9 pull requests #127125

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 29, 2024

Add support for mtime override to generate and combine rust-insta…

e1999f9

…ller commands

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 30, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 30, 2024

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 2, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 2, 2024

matthiaskrgr mentioned this pull request Jul 2, 2024

Rollup of 9 pull requests #127249

Closed

matthiaskrgr mentioned this pull request Jul 3, 2024

Rollup of 8 pull requests #127278

Merged

bors merged commit 444a0ff into rust-lang:master Jul 3, 2024
6 of 7 checks passed

rustbot added this to the 1.81.0 milestone Jul 3, 2024

Kobzol deleted the reproducibility-git branch July 3, 2024 18:55

Veykril mentioned this pull request Jul 18, 2024

Automated pull from upstream master ferrocene/ferrocene#781

Merged

Kobzol mentioned this pull request Sep 3, 2024

don't remove bootstrap on stage0 bump #129924

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make mtime of reproducible tarballs dependent on git commit #127050

Make mtime of reproducible tarballs dependent on git commit #127050

Kobzol commented Jun 27, 2024 •

edited

Loading

Kobzol commented Jun 27, 2024

onur-ozkan left a comment

Kobzol commented Jun 28, 2024

onur-ozkan commented Jun 28, 2024

bors commented Jun 28, 2024

bors commented Jun 29, 2024

jieyouxu commented Jun 29, 2024

Kobzol commented Jun 29, 2024

bors commented Jun 29, 2024

matthiaskrgr commented Jun 29, 2024

bors commented Jun 29, 2024

bors commented Jun 29, 2024

Kobzol commented Jun 30, 2024

onur-ozkan commented Jun 30, 2024

bors commented Jun 30, 2024

bors commented Jul 2, 2024

bors commented Jul 2, 2024

rust-log-analyzer commented Jul 2, 2024

compiler-errors commented Jul 2, 2024

Make mtime of reproducible tarballs dependent on git commit #127050

Make mtime of reproducible tarballs dependent on git commit #127050

Conversation

Kobzol commented Jun 27, 2024 • edited Loading

Kobzol commented Jun 27, 2024

onur-ozkan left a comment

Choose a reason for hiding this comment

Kobzol commented Jun 28, 2024

onur-ozkan commented Jun 28, 2024

bors commented Jun 28, 2024

bors commented Jun 29, 2024

jieyouxu commented Jun 29, 2024

Kobzol commented Jun 29, 2024

bors commented Jun 29, 2024

matthiaskrgr commented Jun 29, 2024

bors commented Jun 29, 2024

bors commented Jun 29, 2024

Kobzol commented Jun 30, 2024

onur-ozkan commented Jun 30, 2024

bors commented Jun 30, 2024

bors commented Jul 2, 2024

bors commented Jul 2, 2024

rust-log-analyzer commented Jul 2, 2024

compiler-errors commented Jul 2, 2024

Kobzol commented Jun 27, 2024 •

edited

Loading