Clean up dependency tracking in Rustbuild [2/2] #52036

collin5 · 2018-07-03T22:16:04Z

Make clear_if_dirty calls in Builder::cargo with stamp dependencies for the given Mode.

Continuation of #50904
Ref issue #50509
r? @Mark-Simulacrum

Mark-Simulacrum

This looks great, left some nits but should be mostly minor.

Mark-Simulacrum · 2018-07-03T22:18:46Z

src/bootstrap/builder.rs

+        };
+
+        if cmd == "doc" {
+            if mode == Mode::Rustc || mode == Mode::ToolRustc {


Mark-Simulacrum · 2018-07-03T22:19:50Z

src/bootstrap/builder.rs

+        // This is for the original compiler, but if we're forced to use stage 1, then
+        // std/test/rustc stamps won't exist in stage 2, so we need to get those from stage 1, since
+        // we copy the libs forward.
+        let compiler = if self.force_use_stage1(compiler, target) {


Let's be careful here and only use this compiler for the stamp files -- otherwise I think we'll run into trouble lower down in this function with the compiler path being different than expected.

Right, going to fix this. Thanks!

Mark-Simulacrum · 2018-07-03T22:21:21Z

src/bootstrap/builder.rs

@@ -705,6 +705,68 @@ impl<'a> Builder<'a> {
    ) -> Command {
        let mut cargo = Command::new(&self.initial_cargo);
        let out_dir = self.stage_out(compiler, mode);
+
+        let mut my_out = match cmd {


Can you add a comment here about the difference between this and out_dir defined just above? (Ideally we'd unify the two).

I had thought of unifying it but looks like this (out_dir) is used lower down https://github.com/rust-lang/rust/blob/master/src/bootstrap/builder.rs#L709 and is always the same no matter the passed cmd yet my_out should be different for each passed cmd. Maybe I'm missing something here.

Ah, right -- yes, that's correct. I think we'll eventually want to resolve that to be the same (since presumably that's ultimately what we want) but for now it should be enough to just add the comment -- basically saying that this is a command-specific path, unlike out_dir which is broader in scope.

Mark-Simulacrum · 2018-07-03T22:23:00Z

src/bootstrap/builder.rs

+                    self.clear_if_dirty(&my_out, &self.rustc(compiler));
+                },
+                Mode::Rustc => {
+                    self.clear_if_dirty(&my_out, &libstd_stamp);


These will all want to be gated on self.rustc(compiler), I believe, since we can't guarantee that std's stamp was touched (IIRC, I had some scenario where that was the case, though I could be wrong). If we're repeating anything though we should repeat everything (e.g., rustc would be compiler/std/test).

Mark-Simulacrum · 2018-07-03T22:23:17Z

src/bootstrap/builder.rs

+                Mode::Std => {
+                    self.clear_if_dirty(&my_out, &self.rustc(compiler));
+                },
+                Mode::Rustc => {


If you can change the order here to be std, test, rustc that'd be good

Mark-Simulacrum · 2018-07-03T22:23:54Z

src/bootstrap/builder.rs

+                    self.clear_if_dirty(&my_out, &libtest_stamp);
+                    self.clear_if_dirty(&my_out, &librustc_stamp);
+                }
+                _ => { }


This should be a fully exhaustive list, even if some variants are nothing -- which is probably false; we for example for ToolBoostrap need to clear if the bootstrap compiler changed, as we still go through build/bootstrap/debug/rustc in that case.

Mark-Simulacrum · 2018-07-03T22:26:22Z

src/bootstrap/tool.rs

-    fn run(self, builder: &Builder) {
-        let compiler = self.compiler;
-        let target = self.target;
+    fn run(self, _builder: &Builder) {


Looks like we can completely remove this struct now.

Wasn't sure about this at first, but yeah looks like we won't need it anymore 👍

Mark-Simulacrum · 2018-07-03T22:27:31Z

@alexcrichton I've slowly been moving towards this approach -- wanted to get feedback on the general idea of centralizing the clear_if_dirty calls to the Cargo command construction. What do you think?

rust-highfive · 2018-07-03T23:55:55Z

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[01:37:22] 
[01:37:22] ---- builder::__test::build_default stdout ----
[01:37:22] thread 'builder::__test::build_default' panicked at 'assertion failed: `(left == right)`
[01:37:22] 
[01:37:22] Diff < left / right > :
[01:37:22]      Rustc {
[01:37:22]          target: "A",
[01:37:22]          compiler: Compiler {
[01:37:22]              stage: 0,
---
[01:37:22]      },
[01:37:22]      Rustc {
[01:37:22]          target: "B",
[01:37:22]          compiler: Compiler {
[01:37:22] >            stage: 0,
[01:37:22] >            host: "A"
[01:37:22] >        }
[01:37:22] >    },
[01:37:22] >    Rustc {
[01:37:22] >        target: "B",
[01:37:22] >        compiler: Compiler {
[01:37:22]              host: "A"
[01:37:22]          }
[01:37:22]      },
[01:37:22]      Rustc {
---
[01:37:22] 
[01:37:22] ', bootstrap/builder.rs:1604:9
[01:37:22] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:37:22] 
[01:37:22] ---- builder::__test::build_with_target_flag stdout ----
[01:37:22] thread 'builder::__test::build_with_target_flag' panicked at 'assertion failed: `(left == right)`
[01:37:22] 
[01:37:22] Diff < left / right > :
[01:37:22]      Assemble {
[01:37:22]          target_compiler: Compiler {
[01:37:22]              stage: 0,
[01:37:22]              host: "A"
---
[01:37:22] 
[01:37:22] 
[01:37:22] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:37:22] Build completed unsuccessfully in 1:00:08
[01:37:22] make: *** [check] Error 1
[01:37:22] Makefile:58: recipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:17b31f48
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

alexcrichton · 2018-07-05T15:19:57Z

This looks pretty good to me, thanks!

I'd recommend being sure to try out a few combinations of commands locally, this'll be easy to accidentally introduce a regression as we don't have many workflow-like tests

rust-highfive · 2018-07-11T14:34:45Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.10.55->awscli)
Building wheels for collected packages: awscli
  Running setup.py bdist_wheel for awscli ... - \ | / - \ done
Successfully built awscli
Installing collected packages: docutils, jmespath, python-dateutil, botocore, colorama, pyasn1, rsa, futures, s3transfer, awscli
Successfully installed awscli-1.15.56 botocore-1.10.55 colorama-0.3.9 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.3 python-dateutil-2.7.3 rsa-3.4.2 s3transfer-0.1.13
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
---
[00:02:24]    Compiling bootstrap v0.0.0 (file:///checkout/src/bootstrap)
[00:02:27] error[E0599]: no variant named `ToolTest` found for type `Mode` in the current scope
[00:02:27]    --> bootstrap/builder.rs:765:17
[00:02:27]     |
[00:02:27] 765 |                 Mode::ToolTest => { },
[00:02:27]     |                 ^^^^^^^^^^^^^^ variant not found in `Mode`
[00:02:27]     | 
[00:02:27]    ::: bootstrap/lib.rs:319:1
[00:02:27]     |
[00:02:27] 319 | pub enum Mode {
[00:02:27]     | ------------- variant `ToolTest` not found here
[00:02:28] error: aborting due to previous error
[00:02:28] 
[00:02:28] For more information about this error, try `rustc --explain E0599`.
[00:02:28] For more information about this error, try `rustc --explain E0599`.
[00:02:28] error: Could not compile `bootstrap`.
[00:02:28] To learn more, run the command again with --verbose.
[00:02:28] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:28] Build completed unsuccessfully in 0:01:29
[00:02:28] Makefile:81: recipe for target 'prepare' failed
[00:02:28] Makefile:81: recipe for target 'prepare' failed
[00:02:28] make: *** [prepare] Error 1
[00:02:29] Command failed. Attempt 2/5:
[00:02:29]    Compiling bootstrap v0.0.0 (file:///checkout/src/bootstrap)
[00:02:33] error[E0599]: no variant named `ToolTest` found for type `Mode` in the current scope
[00:02:33]    --> bootstrap/builder.rs:765:17
[00:02:33]     |
[00:02:33] 765 |                 Mode::ToolTest => { },
[00:02:33]     |                 ^^^^^^^^^^^^^^ variant not found in `Mode`
[00:02:33]     | 
[00:02:33]    ::: bootstrap/lib.rs:319:1
[00:02:33]     |
[00:02:33] 319 | pub enum Mode {
[00:02:33]     | ------------- variant `ToolTest` not found here
[00:02:33] error: aborting due to previous error
[00:02:33] 
[00:02:33] For more information about this error, try `rustc --explain E0599`.
[00:02:33] For more information about this error, try `rustc --explain E0599`.
[00:02:33] error: Could not compile `bootstrap`.
[00:02:33] To learn more, run the command again with --verbose.
[00:02:33] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:33] Build completed unsuccessfully in 0:00:04
[00:02:33] Makefile:81: recipe for target 'prepare' failed
[00:02:33] Makefile:81: recipe for target 'prepare' failed
[00:02:33] make: *** [prepare] Error 1
[00:02:35] Command failed. Attempt 3/5:
[00:02:36]    Compiling bootstrap v0.0.0 (file:///checkout/src/bootstrap)
[00:02:40] error[E0599]: no variant named `ToolTest` found for type `Mode` in the current scope
[00:02:40]    --> bootstrap/builder.rs:765:17
[00:02:40]     |
[00:02:40] 765 |                 Mode::ToolTest => { },
[00:02:40]     |                 ^^^^^^^^^^^^^^ variant not found in `Mode`
[00:02:40]     | 
[00:02:40]    ::: bootstrap/lib.rs:319:1
[00:02:40]     |
[00:02:40] 319 | pub enum Mode {
[00:02:40]     | ------------- variant `ToolTest` not found here
[00:02:40] error: aborting due to previous error
[00:02:40] 
[00:02:40] For more information about this error, try `rustc --explain E0599`.
[00:02:40] For more information about this error, try `rustc --explain E0599`.
[00:02:40] error: Could not compile `bootstrap`.
[00:02:40] To learn more, run the command again with --verbose.
[00:02:40] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:40] Build completed unsuccessfully in 0:00:04
[00:02:40] Makefile:81: recipe for target 'prepare' failed
[00:02:40] Makefile:81: recipe for target 'prepare' failed
[00:02:40] make: *** [prepare] Error 1
[00:02:43] Command failed. Attempt 4/5:
[00:02:43]    Compiling bootstrap v0.0.0 (file:///checkout/src/bootstrap)
[00:02:47] error[E0599]: no variant named `ToolTest` found for type `Mode` in the current scope
[00:02:47]    --> bootstrap/builder.rs:765:17
[00:02:47]     |
[00:02:47] 765 |                 Mode::ToolTest => { },
[00:02:47]     |                 ^^^^^^^^^^^^^^ variant not found in `Mode`
[00:02:47]     | 
[00:02:47]    ::: bootstrap/lib.rs:319:1
[00:02:47]     |
[00:02:47] 319 | pub enum Mode {
[00:02:47]     | ------------- variant `ToolTest` not found here
[00:02:48] error: aborting due to previous error
[00:02:48] 
[00:02:48] For more information about this error, try `rustc --explain E0599`.
[00:02:48] For more information about this error, try `rustc --explain E0599`.
[00:02:48] error: Could not compile `bootstrap`.
[00:02:48] To learn more, run the command again with --verbose.
[00:02:48] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:48] Build completed unsuccessfully in 0:00:04
[00:02:48] Makefile:81: recipe for target 'prepare' failed
[00:02:48] Makefile:81: recipe for target 'prepare' failed
[00:02:48] make: *** [prepare] Error 1
[00:02:52] Command failed. Attempt 5/5:
[00:02:52]    Compiling bootstrap v0.0.0 (file:///checkout/src/bootstrap)
[00:02:56] error[E0599]: no variant named `ToolTest` found for type `Mode` in the current scope
[00:02:56]    --> bootstrap/builder.rs:765:17
[00:02:56]     |
[00:02:56] 765 |                 Mode::ToolTest => { },
[00:02:56]     |                 ^^^^^^^^^^^^^^ variant not found in `Mode`
[00:02:56]     | 
[00:02:56]    ::: bootstrap/lib.rs:319:1
[00:02:56]     |
[00:02:56] 319 | pub enum Mode {
[00:02:56]     | ------------- variant `ToolTest` not found here
[00:02:56] error: aborting due to previous error
[00:02:56] 
[00:02:56] For more information about this error, try `rustc --explain E0599`.
[00:02:56] For more information about this error, try `rustc --explain E0599`.
[00:02:56] error: Could not compile `bootstrap`.
[00:02:56] To learn more, run the command again with --verbose.
[00:02:56] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:56] Build completed unsuccessfully in 0:00:04
[00:02:56] make: *** [prepare] Error 1
---
travis_time:end:0568e832:start=1531319476021543158,finish=1531319476029387059,duration=7843901
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:08d4ba0e
$ head -30 ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
head: cannot open ‘./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers’ for reading: No such file or directory
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0e36f778
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2018-07-13T14:29:31Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:33:10] Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:33:10] Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
[00:33:10] Uplifting stage1 test (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:33:10] Copying stage2 test from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
[00:33:10] thread 'main' panicked at 'File::open(stamp) failed with No such file or directory (os error 2)', bootstrap/lib.rs:1108:12
[00:33:10] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:33:10] Build completed unsuccessfully in 0:29:36
[00:33:10] Build completed unsuccessfully in 0:29:36
[00:33:10] make: *** [all] Error 1
[00:33:10] Makefile:28: recipe for target 'all' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:04942aa0
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:08bffda6:start=1531492167650324644,finish=1531492167656424917,duration=6100273
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:010bae86
$ head -30 ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
head: cannot open ‘./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers’ for reading: No such file or directory
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:06cb5830
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

collin5 · 2018-07-14T02:14:35Z

You can now take another look at this. Thank you!
r? @Mark-Simulacrum

Mark-Simulacrum · 2018-07-14T13:59:04Z

src/bootstrap/builder.rs

+                    self.clear_if_dirty(&my_out, &libstd_stamp);
+                    self.clear_if_dirty(&my_out, &libtest_stamp);
+                },
+                Mode::Codegen => { },


Codegen also needs to be cleared if rustc/std/test/rustc are dirty.

Going to fix this. Thanks 👍

Mark-Simulacrum · 2018-07-14T14:00:11Z

src/bootstrap/compile.rs

-            target,
-            cause: Mode::Std,
-        });
+        builder.cargo(target_compiler, Mode::ToolStd, target, "clean");


Hm, is this (and similar statements below) actually necessary? I would've hoped this would happen on its own.

I think so. It looks like some steps are missed when we don't explicitly call Builder::cargo here. i.e running ./x.py test src/bootstrap without doing this shows that some steps were actually not called. I'm not completely sure about this though, maybe I'm missing something.

Step resolution shouldn't depend on files existing on the system - do you happen to have a concrete example? If we actually do need this then can we factor out the cleaning logic from Builder::cargo and call that separately here? It feels a little odd to be creating the full command and then not running it (i.e., using implicit side-effects).

If you don't feel like dealing with this though feel free to let me know -- I'm fine with leaving these in for now

Yeah, i.e removing both Builder::cargo https://github.com/rust-lang/rust/pull/52036/files#diff-fa9668c926a2788f7849514fe3ed66a9R248 for StdLink and https://github.com/rust-lang/rust/pull/52036/files#diff-fa9668c926a2788f7849514fe3ed66a9R446 for TestLink then doing ./x.py test src/bootstrap, Assertion

rust/src/bootstrap/builder.rs

Line 1660 in 1ecf692

assert_eq!(

will fail and looking at diff shows one of steps was missed. Perhaps we could factor out cleaning logic as you have suggested above.

Okay, let's leave this as is for now and look into the details later -- it seems fine to leave this somewhat half done pending a more thorough investigation as to how exactly the dependency edge is created.

rust-highfive · 2018-07-16T20:01:25Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:29abd7b0:start=1531771282486420370,finish=1531771282494725326,duration=8304956
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0dc900d6
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:015caf2e
travis_time:start:015caf2e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:22eaab3c

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Mark-Simulacrum · 2018-07-17T01:45:58Z

@collin5 If you can try out a few workflows/commands and make sure they continue to work (i.e., no weird linker/compiler errors related to duplicate deps or unexpected rebuilding), such as:

x.py build --stage 1 src/libstd with edits to compiler and std (i.e., compiler successfully builds after such changes)
x.py test --stage 1 src/test/compile-fail with an edit to std (making sure that tools build properly)
x.py build --stage 2 with edit to rustc (basically, ensure that editing rustc doesn't cause a later step to fail due to deps being outdated).

@bors delegate+

If you feel about the above "workflows" being correct then feel free to r=me, you should have permission now.

bors · 2018-07-17T01:45:59Z

✌️ @collin5 can now approve this pull request

collin5 · 2018-07-17T14:08:37Z

@bors r=Mark-Simulacrum

bors · 2018-07-17T14:08:38Z

📌 Commit 0d26932a19e60014da2ee02d17f00167b4952fc4 has been approved by Mark-Simulacrum

bors · 2018-07-17T17:52:10Z

⌛ Testing commit 0d26932a19e60014da2ee02d17f00167b4952fc4 with merge d1abe7496cbf9edf09a646b7d17f54c72958c553...

bors · 2018-07-17T18:54:07Z

💔 Test failed - status-travis

rust-highfive · 2018-07-17T18:54:09Z

The job wasm32-unknown of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/4a/98/035298bacd2d0041f0a1c919ebd81c8e7d5211c4282b9e5bf97bb9f5c416/awscli-1.15.60-py2.py3-none-any.whl (1.3MB)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Requirement already satisfied: PyYAML<=3.13,>=3.10 in /usr/lib/python2.7/dist-packages (from awscli)
---
  Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
Collecting botocore==1.10.59 (from awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/bf/77/26dab42032978b5f547869836523701279fc207d4affcad74b4b6d65f13f/botocore-1.10.59-py2.py3-none-any.whl (4.4MB)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl (543kB)
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.12->awscli)
---
[00:57:58]    Compiling rand v0.4.2
[00:57:58]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:00] [RUSTC-TIMING] rand test:false 2.143
[00:58:31] [RUSTC-TIMING] core test:false 32.973
[00:58:31] error[E0523]: found two different crates with name `core` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two.
[00:58:31]   --> libcore/../libcore/tests/lib.rs:49:1
[00:58:31] 49 | extern crate core;
[00:58:31]    | ^^^^^^^^^^^^^^^^^^
[00:58:31] 
[00:58:31] error: aborting due to previous error
---
[00:58:31] 
[00:58:31] To learn more, run the command again with --verbose.
[00:58:31] 
[00:58:31] 
[00:58:31] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "wasm32-unknown-unknown" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "core" "--"
[00:58:31] 
[00:58:31] 
[00:58:31] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target wasm32-unknown-unknown src/test/run-make src/test/ui src/test/run-pass src/test/compile-fail src/test/parse-fail src/test/mir-opt src/test/codegen-units src/libcore
[00:58:31] Build completed unsuccessfully in 0:55:42
---
travis_time:end:0d21902c:start=1531853588664730574,finish=1531853588672240957,duration=7510383
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0049b138
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:091bae04
travis_time:start:091bae04
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0c694d64
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2018-07-17T18:54:14Z

The job wasm32-unknown of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/4a/98/035298bacd2d0041f0a1c919ebd81c8e7d5211c4282b9e5bf97bb9f5c416/awscli-1.15.60-py2.py3-none-any.whl (1.3MB)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Requirement already satisfied: PyYAML<=3.13,>=3.10 in /usr/lib/python2.7/dist-packages (from awscli)
---
  Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
Collecting botocore==1.10.59 (from awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/bf/77/26dab42032978b5f547869836523701279fc207d4affcad74b4b6d65f13f/botocore-1.10.59-py2.py3-none-any.whl (4.4MB)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl (543kB)
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.12->awscli)
---
[00:57:58]    Compiling rand v0.4.2
[00:57:58]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:00] [RUSTC-TIMING] rand test:false 2.143
[00:58:31] [RUSTC-TIMING] core test:false 32.973
[00:58:31] error[E0523]: found two different crates with name `core` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two.
[00:58:31]   --> libcore/../libcore/tests/lib.rs:49:1
[00:58:31] 49 | extern crate core;
[00:58:31]    | ^^^^^^^^^^^^^^^^^^
[00:58:31] 
[00:58:31] error: aborting due to previous error
---
[00:58:31] 
[00:58:31] To learn more, run the command again with --verbose.
[00:58:31] 
[00:58:31] 
[00:58:31] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "wasm32-unknown-unknown" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "core" "--"
[00:58:31] 
[00:58:31] 
[00:58:31] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target wasm32-unknown-unknown src/test/run-make src/test/ui src/test/run-pass src/test/compile-fail src/test/parse-fail src/test/mir-opt src/test/codegen-units src/libcore
[00:58:31] Build completed unsuccessfully in 0:55:42
---
travis_time:end:0d21902c:start=1531853588664730574,finish=1531853588672240957,duration=7510383
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0049b138
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:091bae04
travis_time:start:091bae04
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0c694d64
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-09-12T08:08:09Z

💔 Test failed - status-travis

rust-highfive · 2018-09-12T08:08:11Z

The job wasm32-unknown of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[01:14:07] [RUSTC-TIMING] rand_core test:false 0.482
[01:14:07]    Compiling rand v0.5.4
[01:14:11] [RUSTC-TIMING] rand test:false 3.587
[01:14:30] [RUSTC-TIMING] core test:false 23.685
[01:14:30] error[E0523]: found two different crates with name `core` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two.
[01:14:30]   --> libcore/../libcore/tests/lib.rs:44:1
[01:14:30] 44 | extern crate core;
[01:14:30]    | ^^^^^^^^^^^^^^^^^^
[01:14:30] 
[01:14:30] error: aborting due to previous error
---
[01:14:30] 
[01:14:30] To learn more, run the command again with --verbose.
[01:14:30] 
[01:14:30] 
[01:14:30] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "wasm32-unknown-unknown" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "core" "--"
[01:14:30] 
[01:14:30] 
[01:14:30] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target wasm32-unknown-unknown src/test/run-make src/test/ui src/test/run-pass src/test/compile-fail src/test/parse-fail src/test/mir-opt src/test/codegen-units src/libcore
[01:14:30] Build completed unsuccessfully in 1:10:47
---
travis_time:end:098e865a:start=1536739607225524569,finish=1536739607230683831,duration=5159262
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0dac793b
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:05fcbc5b
travis_time:start:05fcbc5b
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0752a3ea
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

collin5 · 2018-09-12T16:16:56Z

Hmm, Looks like I've failed to reproduce this. Running ./src/ci/docker/run.sh wasm32-unknown on Ubuntu 16.04.2 LTS always exits with

[TIMING] Crate { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "wasm32-unknown-unknown", mode: Std, test_kind: Test, krate: "core" } -- 1.078
Build completed successfully in 0:02:33

Do you please mind taking a look at this? Maybe there is something I'm missing here. Thank you! cc @Mark-Simulacrum

Mark-Simulacrum · 2018-09-12T16:25:36Z

Hm, have you tried clearing out the build directory (obj I think by default) and retrying? I can queue a build later today or tomorrow on my local hardware to try and reproduce as well.

collin5 · 2018-09-12T16:37:27Z

Yeah, I did that. Even removed all the docker images and cache but still getting the same result.
Thank you, I would really appreciate that.

also indicate difference between out_dir and my_out

Mark-Simulacrum · 2018-09-12T22:26:08Z

I rebased and force pushed, I think I can reproduce this now.

collin5 · 2018-09-13T17:07:26Z

Awesome! thanks a lot. Going to pull the changes in a few.

collin5 · 2018-09-17T14:58:54Z

😌 Whew! . Finally this should do it.
@bors r+

bors · 2018-09-17T14:58:55Z

📌 Commit 5ae40be has been approved by collin5

bors · 2018-09-17T21:15:18Z

⌛ Testing commit 5ae40be with merge 2224a42...

@Mark-Simulacrum

Clean up dependency tracking in Rustbuild [2/2] Make `clear_if_dirty` calls in `Builder::cargo` with stamp dependencies for the given Mode. Continuation of #50904 Ref issue #50509 r? @Mark-Simulacrum

bors · 2018-09-17T23:43:12Z

☀️ Test successful - status-appveyor, status-travis
Approved by: collin5
Pushing 2224a42 to master...

rust-highfive assigned Mark-Simulacrum Jul 3, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 3, 2018

Mark-Simulacrum reviewed Jul 3, 2018

View reviewed changes

pietroalbini added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 9, 2018

collin5 force-pushed the b50509-2 branch from d8452ed to 4729bc6 Compare July 13, 2018 13:54

collin5 force-pushed the b50509-2 branch from 4729bc6 to ee6f8af Compare July 13, 2018 23:08

Mark-Simulacrum reviewed Jul 14, 2018

View reviewed changes

collin5 force-pushed the b50509-2 branch from ee6f8af to 9bcfde9 Compare July 16, 2018 19:25

collin5 force-pushed the b50509-2 branch from 9bcfde9 to 0d26932 Compare July 17, 2018 00:25

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 17, 2018

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 17, 2018

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 12, 2018

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 12, 2018

collin5 and others added 7 commits September 12, 2018 13:31

clear_if_dirty in Builder::cargo with passed mode

c0af0b0

use stage 1 compiler only for stamps

c22c709

clear_if_dirty with self.rustc(compiler) for Mode::Rustc

86e34cd

nit: rearrange and make match exhaustive

de3ec8d

also indicate difference between out_dir and my_out

remove struct CleanTools

e792d1d

Do not print checking messages in dry run mode

b585893

Add check build dependencies

9681e13

Mark-Simulacrum force-pushed the b50509-2 branch from 5653421 to 9681e13 Compare September 12, 2018 22:18

refactor Builder::cargo, clean deps for cmd!=test

5ae40be

collin5 force-pushed the b50509-2 branch from 8586a1d to 5ae40be Compare September 17, 2018 11:35

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 17, 2018

bors merged commit 5ae40be into rust-lang:master Sep 17, 2018

collin5 deleted the b50509-2 branch September 19, 2018 18:06

Clean up dependency tracking in Rustbuild [2/2] #52036

Clean up dependency tracking in Rustbuild [2/2] #52036

Conversation

collin5 commented Jul 3, 2018 • edited Loading

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 Jul 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 Jul 4, 2018 • edited Loading

Choose a reason for hiding this comment

Mark-Simulacrum commented Jul 3, 2018

rust-highfive commented Jul 3, 2018

alexcrichton commented Jul 5, 2018

rust-highfive commented Jul 11, 2018

rust-highfive commented Jul 13, 2018

collin5 commented Jul 14, 2018

Choose a reason for hiding this comment

collin5 Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 Jul 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rust-highfive commented Jul 16, 2018

Mark-Simulacrum commented Jul 17, 2018

bors commented Jul 17, 2018

collin5 commented Jul 17, 2018

bors commented Jul 17, 2018

bors commented Jul 17, 2018

bors commented Jul 17, 2018

rust-highfive commented Jul 17, 2018

rust-highfive commented Jul 17, 2018

bors commented Sep 12, 2018

rust-highfive commented Sep 12, 2018

collin5 commented Sep 12, 2018

Mark-Simulacrum commented Sep 12, 2018

collin5 commented Sep 12, 2018

Mark-Simulacrum commented Sep 12, 2018

collin5 commented Sep 13, 2018

collin5 commented Sep 17, 2018

bors commented Sep 17, 2018

bors commented Sep 17, 2018

bors commented Sep 17, 2018

collin5 commented Jul 3, 2018 •

edited

Loading

collin5 Jul 4, 2018 •

edited

Loading

collin5 Jul 4, 2018 •

edited

Loading

collin5 Jul 16, 2018 •

edited

Loading

collin5 Jul 17, 2018 •

edited

Loading