Do not add external libraries to libstd stamp file #72698

mati865 · 2020-05-28T15:31:20Z

Currently they are present in .libstd.stamp and point to target sysroot:

tD:\Projekty\rust\build\x86_64-pc-windows-gnu\stage1\lib\rustlib\x86_64-pc-windows-gnu\lib\crt2.o

while normal objects point to non sysroot locations:

tD:\Projekty\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\liballoc-eb2e01dae1fd9cc7.rlib

Now add_to_sysroot(...) function reads stamp file and copies everything listed to the target or host sysroot. So in case of those objects we are effectively copying the inplace (from target_sysroot/lib/X to target_sysroot/lib/X).

This PR also prepares ground for moving those objects to different location: #68887 (comment)

rust-highfive · 2020-05-28T15:31:24Z

r? @kennytm

(rust_highfive has picked a reviewer for you, use r? to override)

mati865 · 2020-05-28T15:32:47Z

cc @petrochenkov

Enabled try builders:

Linux for sanitizers
Musl for cross compiled target
Windows-gnu for natively compiled target

rust-highfive · 2020-05-28T15:36:30Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

##[section]Starting: Linux mingw-check
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 19'
Agent machine name: 'fv-az578'
Current agent version: '2.168.2'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200517.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200517.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.2)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/9fc82482-72f1-4f64-bd8b-55cfca990109.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/72698/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/72698/merge:refs/remotes/pull/72698/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
   Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
error[E0308]: mismatched types
   --> src/bootstrap/compile.rs:335:20
    |
331 |     fn run(self, builder: &Builder<'_>) {
...
335 |             return vec![];
    |                    ^^^^^^ expected `()`, found struct `std::vec::Vec`
    |
---
  local time: Thu May 28 15:35:56 UTC 2020
  network time: Thu, 28 May 2020 15:35:56 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/72698/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/72698/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3696) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

They are already copied before libstd is built. Copying them again when reading stamp file is unnecessary.

mati865 · 2020-05-28T15:54:38Z

Wrong git stash...
Should be fine now.

tmiasko · 2020-05-28T16:22:23Z

As far as I understand, the content of stamp file also controls what will be
eventually distributed, so those changes effectively stop distributing those
libraries (take a look at how copy_target_libs in dist.rs is using stamp file).

mati865 · 2020-05-28T16:48:08Z

Unfortunately it's something I couldn't test on Windows because ./x.py dist doesn't work with MinGW Python installed from MSYS2. I'll see if ./x.py install works.

rust-highfive · 2020-05-28T17:46:50Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

##[section]Starting: Linux mingw-check
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 2'
Agent machine name: 'fv-az578'
Current agent version: '2.168.2'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200517.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200517.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.2)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/74c1631a-9099-43c5-86da-5924265f264f.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/72698/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/72698/merge:refs/remotes/pull/72698/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
    Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
    Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
    Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
    Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
    Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
    Checking chalk-rust-ir v0.10.0
    Checking chalk-solve v0.10.0
    Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
    Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
    Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
configure: build.submodules     := False
configure: dist.missing-tools   := True
configure: rust.debug-assertions := True
configure: build.locked-deps    := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
Hugepagesize:       2048 kB
DirectMap4k:      143296 kB
DirectMap2M:     4050944 kB
DirectMap1G:     5242880 kB
+ python3 ../x.py test src/tools/expand-yaml-anchors
Ensuring the YAML anchors in the GitHub Actions config were expanded
Ensuring the YAML anchors in the GitHub Actions config were expanded
Building stage0 tool expand-yaml-anchors (x86_64-unknown-linux-gnu)
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.11
   Compiling linked-hash-map v0.5.2
   Compiling lazy_static v1.4.0
   Compiling lazy_static v1.4.0
   Compiling yaml-rust v0.4.3
   Compiling quote v1.0.2
   Compiling thiserror-impl v1.0.5
   Compiling thiserror v1.0.5
   Compiling yaml-merge-keys v0.4.0
   Compiling expand-yaml-anchors v0.1.0 (/checkout/src/tools/expand-yaml-anchors)
Build completed successfully in 0:00:30
+ python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu
    Finished dev [unoptimized] target(s) in 0.22s
Checking rustdoc artifacts (x86_64-unknown-linux-gnu -> i686-pc-windows-gnu)
---
    Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
    Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
    Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
    Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
    Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
    Checking chalk-rust-ir v0.10.0
    Checking chalk-solve v0.10.0
    Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
    Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
    Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
             return;
         }
 
-
         let mut cargo = builder.cargo(compiler, Mode::Std, target, "build");
         std_cargo(builder, target, compiler.stage, &mut cargo);
Diff in /checkout/src/bootstrap/compile.rs at line 87:
Diff in /checkout/src/bootstrap/compile.rs at line 87:
             "Building stage{} std artifacts ({} -> {})",
             compiler.stage, &compiler.host, target
         ));
-        run_cargo(
-            cargo,
-            vec![],
-            vec![],
-            &libstd_stamp(builder, compiler, target),
-        );
-        );
+        run_cargo(builder, cargo, vec![], &libstd_stamp(builder, compiler, target), false);
         builder.ensure(StdLink {
         builder.ensure(StdLink {
             compiler: builder.compiler(compiler.stage, builder.config.build),
Diff in /checkout/src/bootstrap/compile.rs at line 104:
 
 
 /// Copies third party objects needed by various targets.
-fn copy_third_party_objects(
-    builder: &Builder<'_>,
-    compiler: &Compiler,
-    target: Interned<String>,
-) {
+fn copy_third_party_objects(builder: &Builder<'_>, compiler: &Compiler, target: Interned<String>) {
     let libdir = builder.sysroot_libdir(*compiler, target);
 
     let copy = |sourcedir: &Path, name: &str| {
Diff in /checkout/src/bootstrap/compile.rs at line 272:
 
 /// Copies sanitizer runtime libraries into target libdir.
-fn copy_sanitizers(
-    builder: &Builder<'_>,
-    builder: &Builder<'_>,
-    compiler: &Compiler,
-    target: Interned<String>,
-) {
+fn copy_sanitizers(builder: &Builder<'_>, compiler: &Compiler, target: Interned<String>) {
     let runtimes: Vec<native::SanitizerRuntime> = builder.ensure(native::Sanitizers { target });
     if builder.config.dry_run {
Diff in /checkout/src/bootstrap/compile.rs at line 435:
Diff in /checkout/src/bootstrap/compile.rs at line 435:
             "Building stage{} compiler artifacts ({} -> {})",
             compiler.stage, &compiler.host, target
         ));
-        run_cargo(
-            cargo,
-            vec![],
-            vec![],
-            &librustc_stamp(builder, compiler, target),
-        );
-        );
+        run_cargo(builder, cargo, vec![], &librustc_stamp(builder, compiler, target), false);
         builder.ensure(RustcLink {
         builder.ensure(RustcLink {
             compiler: builder.compiler(compiler.stage, builder.config.build),
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/src/bootstrap/compile.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
Build completed unsuccessfully in 0:00:39
== clock drift check ==
  local time: Thu May 28 16:15:10 UTC 2020
  network time: Thu, 28 May 2020 16:15:11 GMT
  network time: Thu, 28 May 2020 16:15:11 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/72698/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/72698/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3358) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

mati865 · 2020-05-28T22:50:58Z

Made dist build on Linux and indeed objects are not copied if they are not present in stamp.
I'll have to find different workaround.

rust-highfive assigned kennytm May 28, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 28, 2020

Mark-Simulacrum assigned Mark-Simulacrum and unassigned kennytm May 28, 2020

mati865 added 2 commits May 28, 2020 17:52

Do not add external libraries to libstd stamp file

00b824a

They are already copied before libstd is built. Copying them again when reading stamp file is unnecessary.

WIP: modify try builders

41ef99a

mati865 force-pushed the no-external-libs-in-stamp branch from c5414eb to 41ef99a Compare May 28, 2020 15:53

mati865 closed this May 28, 2020

mati865 deleted the no-external-libs-in-stamp branch May 28, 2020 22:51

mati865 mentioned this pull request May 29, 2020

Self contained linking option #72738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not add external libraries to libstd stamp file #72698

Do not add external libraries to libstd stamp file #72698

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020

tmiasko commented May 28, 2020

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020

Do not add external libraries to libstd stamp file #72698

Do not add external libraries to libstd stamp file #72698

Conversation

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020

tmiasko commented May 28, 2020

mati865 commented May 28, 2020

rust-highfive commented May 28, 2020

mati865 commented May 28, 2020