Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std: Depend directly on crates.io crates #56092

Merged
merged 1 commit into from
Dec 12, 2018

Conversation

alexcrichton
Copy link
Member

Ever since we added a Cargo-based build system for the compiler the
standard library has always been a little special, it's never been able
to depend on crates.io crates for runtime dependencies. This has been a
result of various limitations, namely that Cargo doesn't understand that
crates from crates.io depend on libcore, so Cargo tries to build crates
before libcore is finished.

I had an idea this afternoon, however, which lifts the strategy
from #52919 to directly depend on crates.io crates from the standard
library. After all is said and done this removes a whopping three
submodules that we need to manage!

The basic idea here is that for any crate std depends on it adds an
optional dependency on an empty crate on crates.io, in this case named
rustc-std-workspace-core. This crate is overridden via [patch] in
this repository to point to a local crate we write, and that has a
path dependency on libcore.

Note that all no_std crates also depend on compiler_builtins, but if
we're not using submodules we can publish compiler_builtins to
crates.io and all crates can depend on it anyway! The basic strategy
then looks like:

  • The standard library (or some transitive dep) decides to depend on a
    crate foo.

  • The standard library adds

    [dependencies]
    foo = { version = "0.1", features = ['rustc-dep-of-std'] }
  • The crate foo has an optional dependency on rustc-std-workspace-core

  • The crate foo has an optional dependency on compiler_builtins

  • The crate foo has a feature rustc-dep-of-std which activates these
    crates and any other necessary infrastructure in the crate.

A sample commit for dlmalloc turns out to be quite simple.
After that all no_std crates should largely build "as is" and still be
publishable on crates.io! Notably they should be able to continue to use
stable Rust if necessary, since the rename-dependency feature of Cargo
is soon stabilizing.

As a proof of concept, this commit removes the dlmalloc,
libcompiler_builtins, and libc submodules from this repository. Long
thorns in our side these are now gone for good and we can directly
depend on crates.io! It's hoped that in the long term we can bring in
other crates as necessary, but for now this is largely intended to
simply make it easier to manage these crates and remove submodules.

This should be a transparent non-breaking change for all users, but one
possible stickler is that this almost for sure breaks out-of-tree
std-building tools like xargo and cargo-xbuild. I think it should
be relatively easy to get them working, however, as all that's needed is
an entry in the [patch] section used to build the standard library.
Hopefully we can work with these tools to solve this problem!

@rust-highfive
Copy link
Collaborator

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 20, 2018
@alexcrichton
Copy link
Member Author

r? @Mark-Simulacrum

cc @rust-lang/libs, y'all are likely interested in this!

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0514d240:start=1542695844332022153,finish=1542695901405956600,duration=57073934447
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:01:59]    Compiling proc-macro2 v0.4.13
[00:01:59]    Compiling unicode-xid v0.1.0
[00:01:59]    Compiling ryu v0.2.6
[00:01:59]    Compiling serde v1.0.75
[00:01:59]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:01:59]    Compiling cc v1.0.25
[00:01:59]    Compiling ordermap v0.3.5
[00:02:00]    Compiling cfg-if v0.1.5
[00:02:00]    Compiling itoa v0.4.3
---
tidy check
[00:03:45] * 568 error codes
[00:03:45] * highest error code: E0721
[00:03:46] * 239 features
[00:03:46] invalid source: "git+https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9eed37338be41ef455cb1c43b49a4b786f"
[00:03:46] invalid source: "git+https://github.com/alexcrichton/dlmalloc-rs?branch=rustc-test#28ee12db813a3b650a7c25d1c36d2c17dcb88ae3"
[00:03:46] invalid source: "git+https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730f2b3939eaef106baa9e63449fc64da5e"

[00:03:46] travis_time:end:tidy:start=1542696135120865914,finish=1542696136742693862,duration=1621827948

[00:03:46] Build completed successfully in 0:00:47
---
[00:03:48] travis_fold:start:stage0-std
travis_time:start:stage0-std
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:03:48]    Compiling cc v1.0.25
[00:03:48]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:03:48]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:03:49]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:03:49]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:03:55]    Compiling compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:03:55]    Compiling std v0.0.0 (/checkout/src/libstd)
[00:03:59]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:03:59]    Compiling rustc_tsan v0.0.0 (/checkout/src/librustc_tsan)
[00:03:59]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:03:59]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:03:59]    Compiling rustc_msan v0.0.0 (/checkout/src/librustc_msan)
[00:04:13]    Compiling rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:04:14]    Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:04:19]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:04:37]     Finished release [optimized] target(s) in 49.20s
[00:04:37] Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
travis_time:start:stage0-rustc
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:04:51]    Compiling version_check v0.1.5
[00:04:51]    Compiling cfg-if v0.1.5
[00:04:51]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:04:51]    Compiling scopeguard v0.3.3
[00:04:51]    Compiling void v1.0.2
[00:04:51]    Compiling memoffset v0.2.1
[00:04:51]    Compiling rand_core v0.2.1
---

[00:18:28] travis_time:end:stage0-rustc:start=1542696200952004900,finish=1542697018716184898,duration=817764179998

[00:18:28] Building stage0 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
[00:18:29]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:18:29]    Compiling cc v1.0.25
[00:18:29]    Compiling rustc_codegen_llvm v0.0.0 (/checkout/src/librustc_codegen_llvm)
[00:18:35]    Compiling rustc-demangle v0.1.9
[00:18:36]    Compiling rustc_llvm v0.0.0 (/checkout/src/librustc_llvm)
---
[00:19:34] travis_fold:start:stage1-std
travis_time:start:stage1-std
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:19:34]    Compiling cc v1.0.25
[00:19:34]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:19:34]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:19:36]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:19:36]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:19:41]    Compiling compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:19:41]    Compiling std v0.0.0 (/checkout/src/libstd)
[00:19:44]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:19:44]    Compiling rustc_tsan v0.0.0 (/checkout/src/librustc_tsan)
[00:19:45]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:19:45]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:19:45]    Compiling rustc_msan v0.0.0 (/checkout/src/librustc_msan)
[00:20:19]    Compiling rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:20:21]    Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:20:28]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:20:49]     Finished release [optimized] target(s) in 1m 15s
[00:20:49] Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
travis_time:start:stage1-rustc
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:21:05]    Compiling version_check v0.1.5
[00:21:05]    Compiling cfg-if v0.1.5
[00:21:05]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:21:05]    Compiling memoffset v0.2.1
[00:21:05]    Compiling void v1.0.2
[00:21:05]    Compiling scopeguard v0.3.3
[00:21:05]    Compiling rustc-rayon-core v0.1.1
---

[00:36:54] travis_time:end:stage1-rustc:start=1542697174665881242,finish=1542698124636002080,duration=949970120838

[00:36:54] Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
[00:36:55]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:36:55]    Compiling cc v1.0.25
[00:36:55]    Compiling rustc_codegen_llvm v0.0.0 (/checkout/src/librustc_codegen_llvm)
[00:36:55]    Compiling rustc-demangle v0.1.9
[00:37:03]    Compiling num_cpus v1.8.0
---
[00:38:10] Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
[00:38:10] travis_fold:start:stage2-rustdoc
travis_time:start:stage2-rustdoc
Building rustdoc for stage2 (x86_64-unknown-linux-gnu)
[00:38:10]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:38:10]    Compiling rand_core v0.2.1
[00:38:10]    Compiling stable_deref_trait v1.1.0
[00:38:11]    Compiling scopeguard v0.3.3
[00:38:11]    Compiling pulldown-cmark v0.1.2
---

[00:40:13] travis_fold:start:stage0-rustbook
travis_time:start:stage0-rustbook
Building stage0 tool rustbook (x86_64-unknown-linux-gnu)
[00:40:13]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:40:13]    Compiling string_cache_shared v0.3.0
[00:40:13]    Compiling cc v1.0.25
[00:40:13]    Compiling version_check v0.1.5
[00:40:14]    Compiling void v1.0.2
---
[00:44:03] Documenting book index (x86_64-unknown-linux-gnu)
[00:44:03] Documenting book redirect pages (x86_64-unknown-linux-gnu)
[00:44:04] Documenting stage2 std (x86_64-unknown-linux-gnu)
[00:44:04]     Checking core v0.0.0 (/checkout/src/libcore)
[00:44:39]     Checking rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:44:39]     Checking compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:44:45]     Finished release [optimized] target(s) in 41.08s
[00:44:45]  Documenting core v0.0.0 (/checkout/src/libcore)
[00:45:04] warning: `[chunks]` cannot be resolved, ignoring it...
[00:45:04]    --> libcore/slice/mod.rs:860:52
---
[00:45:04]     |
[00:45:04]     = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`
[00:45:04] 
[00:45:07]     Finished release [optimized] target(s) in 22.54s
[00:45:08]     Checking libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:45:08]     Checking alloc v0.0.0 (/checkout/src/liballoc)
[00:45:09]     Checking unwind v0.0.0 (/checkout/src/libunwind)
[00:45:09]     Checking panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:45:13]     Checking rustc_msan v0.0.0 (/checkout/src/librustc_msan)
---
[00:45:31]     Checking log v0.4.5
[00:45:31]     Checking arrayvec v0.4.7
[00:45:32]     Checking unreachable v1.0.0
[00:45:32]     Checking owning_ref v0.3.3
[00:45:32]     Checking libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730)
[00:45:32]     Checking rustc_cratesio_shim v0.0.0 (/checkout/src/librustc_cratesio_shim)
[00:45:32]     Checking rustc-hash v1.0.1
[00:45:32]     Checking lazy_static v1.1.0
[00:45:32]     Checking smallvec v0.6.5
---
tidy check
[00:46:14] * 568 error codes
[00:46:14] * highest error code: E0721
[00:46:15] * 239 features
[00:46:15] invalid source: "git+https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9eed37338be41ef455cb1c43b49a4b786f"
[00:46:15] invalid source: "git+https://github.com/alexcrichton/dlmalloc-rs?branch=rustc-test#28ee12db813a3b650a7c25d1c36d2c17dcb88ae3"
[00:46:15] invalid source: "git+https://github.com/alexcrichton/libc?branch=rustc-test#d3aa5730f2b3939eaef106baa9e63449fc64da5e"

[00:46:15] travis_time:end:tidy:start=1542698683887190144,finish=1542698685666461457,duration=1779271313

[00:46:15] travis_fold:start:stage0-std
---
[00:47:39] ..................................................................................i...........i..... 800/5041
[00:47:42] .................................................................................................... 900/5041
[00:47:46] .iiiii.............................................................................................. 1000/5041
[00:47:48] .................................................................................................... 1100/5041
[00:47:50] .................F.................................................................................. 1200/5041
[00:47:53] .................................................................................................... 1300/5041
[00:47:55] .......F............................................................................................ 1400/5041
[00:48:00] ..i.....................................................................i........................... 1600/5041
[00:48:04] .................................................................................................... 1700/5041
[00:48:07] .................................................................................................... 1800/5041
[00:48:07] .................................................................................................... 1800/5041
[00:48:11] ............F....................................................................................... 1900/5041
[00:48:14] ....i............................................................................................... 2000/5041
[00:48:17] ..............................................................................................F..... 2100/5041
[00:48:26] .................................................................................................... 2300/5041
[00:48:29] .................................................................................................... 2400/5041
[00:48:29] .................................................................................................... 2400/5041
[00:48:32] .....................................................F.............................................. 2500/5041
[00:48:40] .................................................................................................... 2700/5041
[00:48:43] .................................................................................................... 2800/5041
[00:48:46] .................................................................................................... 2900/5041
[00:48:46] .................................................................................................... 2900/5041
[00:48:50] .......F............................................................................................ 3000/5041
[00:48:57] .................................................................................................... 3200/5041
[00:48:59] .........ii..i..ii.................................................................................. 3300/5041
[00:49:03] .................................................................................................... 3400/5041
[00:49:03] .................................................................................................... 3400/5041
[00:49:07] ........................................................................................F.ii........ 3500/5041
[00:49:11] .........i.......................................................................................... 3700/5041
[00:49:12] ................................................................i................................... 3800/5041
[00:49:14] .................................................................................................... 3900/5041
[00:49:17] .................................................................................................... 4000/5041

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@SimonSapin
Copy link
Contributor

Notably they should be able to continue to use stable Rust if necessary, since the rename-dependency feature of Cargo is soon stabilizing.

Does this raise the minimum Rust version required by libc?

Copy link
Member

@Mark-Simulacrum Mark-Simulacrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I think I'm not a fan of this approach because it requires modifying crates.io crates and bumping their minimum version of Rust to 1.31.

Could we perhaps inject the dependency via an unstable Cargo feature? I'm okay with landing the groundwork here for now, though I don't think it's long term viable...

src/libpanic_unwind/Cargo.toml Show resolved Hide resolved
profiler_builtins = { path = "../libprofiler_builtins", optional = true }
unwind = { path = "../libunwind" }

[dev-dependencies]
rand = "0.5"
#rand = "0.5"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this is because uncommenting rand here causes problems down the line?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry forgot to add a comment about this. This is a temporary holdover until rust-random/rand#647 is merged, published, and integerated here.

@alexcrichton
Copy link
Member Author

Does this raise the minimum Rust version required by libc?

A good question!

Overall I think I'm not a fan of this approach because it requires modifying crates.io crates and bumping their minimum version of Rust to 1.31.

Don't worry, as the maintainer of libc since 1.0 and as the person who's maintained its compatibility back to basically 1.0, this is not something I would forget.

The libc-specific patch is much different than the patch applied to dlmalloc. It's far less ergonomic and I wouldn't recommend it for any other crate. It avoids using the rename-dependency feature in Cargo.

@alexcrichton
Copy link
Member Author

it requires modifying crates.io crates and bumping their minimum version of Rust to 1.31

To be clear, we have basically no other candidates for adding as a dependency to the standard library in the near future. Crates added here are often nightly and unstable, and libc is the only exception with a huge compatibility range (and thus has a special patch). Rust 1.31 will quickly become the norm and for the very rare case that we integrate into libstd I think it's pretty reasonable to say that Rust 1.31 is required.

In any case all crates affected here are barely used on crates.io (modulo libc). I would like to land this to pave the way for using crates in the future, even if crates we want to use don't want to require 1.31 yet.

@Mark-Simulacrum
Copy link
Member

Okay - yeah, the version requirement matters less to me than three general approach of pushing this out into crates.io but I agree that the set of crates is limited enough for now that this is the practical decision.

@alexcrichton
Copy link
Member Author

@Mark-Simulacrum hm sorry I may be misunderstanding, but do you think we should not pursue a change like this?

@gnzlbg
Copy link
Contributor

gnzlbg commented Nov 20, 2018

I think this is awesome, but I worry about the ergonomics.

@alexcrichton IIUC, we only need to add hacks for this to crates in crates.io that we want to use from liballoc, libstd, etc. Is that correct?

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:113b1a20:start=1542735617475246872,finish=1542735672224375716,duration=54749128844
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:02:01]   Downloaded quote v0.6.8
[00:02:01]   Downloaded ordermap v0.3.5
[00:02:01]    Compiling proc-macro2 v0.4.13
[00:02:01]    Compiling unicode-xid v0.1.0
[00:02:01]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:02:01]    Compiling serde v1.0.75
[00:02:01]    Compiling itoa v0.4.3
[00:02:02]    Compiling ordermap v0.3.5
[00:02:02]    Compiling cc v1.0.25
---
tidy check
[00:03:51] * 568 error codes
[00:03:51] * highest error code: E0721
[00:03:52] * 239 features
[00:03:52] invalid source: "git+https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9eed37338be41ef455cb1c43b49a4b786f"
[00:03:52] invalid source: "git+https://github.com/alexcrichton/dlmalloc-rs?branch=rustc-test#28ee12db813a3b650a7c25d1c36d2c17dcb88ae3"
[00:03:52] invalid source: "git+https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db29c691242c394f45f71eb55ac28a9057"

[00:03:52] travis_time:end:tidy:start=1542735912563234351,finish=1542735914104495051,duration=1541260700

[00:03:52] Build completed successfully in 0:00:45
---
[00:03:54] travis_fold:start:stage0-std
travis_time:start:stage0-std
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:03:54]    Compiling cc v1.0.25
[00:03:54]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:03:54]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:03:57]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:03:57]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:04:01]    Compiling compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:04:01]    Compiling std v0.0.0 (/checkout/src/libstd)
[00:04:05]    Compiling rustc_msan v0.0.0 (/checkout/src/librustc_msan)
[00:04:05]    Compiling rustc_tsan v0.0.0 (/checkout/src/librustc_tsan)
[00:04:05]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:04:05]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:04:05]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:04:18]    Compiling rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:04:20]    Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:04:25]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:04:42]     Finished release [optimized] target(s) in 48.45s
[00:04:42] Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
[00:04:55] travis_fold:start:stage0-rustc
travis_time:start:stage0-rustc
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:04:56]    Compiling version_check v0.1.5
[00:04:56]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:04:56]    Compiling nodrop v0.1.12
[00:04:56]    Compiling void v1.0.2
[00:04:56]    Compiling memoffset v0.2.1
[00:04:56]    Compiling scopeguard v0.3.3
---

[00:18:12] travis_time:end:stage0-rustc:start=1542735976922803667,finish=1542736773729254016,duration=796806450349

[00:18:12] Building stage0 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
[00:18:12]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:18:12]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:18:16]    Compiling rustc_codegen_llvm v0.0.0 (/checkout/src/librustc_codegen_llvm)
[00:18:16]    Compiling rustc-demangle v0.1.9
[00:18:21]    Compiling num_cpus v1.8.0
---
travis_time:start:stage1-std
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:19:17]    Compiling cc v1.0.25
[00:19:17]    Compiling core v0.0.0 (/checkout/src/libcore)
[00:19:17]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:19:21]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:19:21]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:19:24]    Compiling compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:19:24]    Compiling std v0.0.0 (/checkout/src/libstd)
[00:19:27]    Compiling rustc_tsan v0.0.0 (/checkout/src/librustc_tsan)
[00:19:27]    Compiling rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
[00:19:28]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:19:28]    Compiling rustc_asan v0.0.0 (/checkout/src/librustc_asan)
[00:19:28]    Compiling rustc_msan v0.0.0 (/checkout/src/librustc_msan)
[00:20:00]    Compiling rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:20:02]    Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:20:09]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:20:29]     Finished release [optimized] target(s) in 1m 12s
[00:20:29] Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:20:44]    Compiling version_check v0.1.5
[00:20:44]    Compiling nodrop v0.1.12
[00:20:44]    Compiling cfg-if v0.1.5
[00:20:44]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:20:44]    Compiling scopeguard v0.3.3
[00:20:44]    Compiling void v1.0.2
[00:20:44]    Compiling rustc-rayon-core v0.1.1
[00:20:45]    Compiling stable_deref_trait v1.1.0
---

[00:36:16] travis_time:end:stage1-rustc:start=1542736925340611536,finish=1542737857523448719,duration=932182837183

[00:36:16] Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
[00:36:16]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:36:16]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:36:16]    Compiling rustc-demangle v0.1.9
[00:36:18]    Compiling rustc_codegen_llvm v0.0.0 (/checkout/src/librustc_codegen_llvm)
[00:36:24]    Compiling memmap v0.6.2
---
[00:37:30] Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
[00:37:30] travis_fold:start:stage2-rustdoc
travis_time:start:stage2-rustdoc
Building rustdoc for stage2 (x86_64-unknown-linux-gnu)
[00:37:30]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:37:30]    Compiling rand_core v0.2.1
[00:37:30]    Compiling stable_deref_trait v1.1.0
[00:37:30]    Compiling scopeguard v0.3.3
[00:37:30]    Compiling pulldown-cmark v0.1.2
---

[00:39:30] travis_fold:start:stage0-rustbook
travis_time:start:stage0-rustbook
Building stage0 tool rustbook (x86_64-unknown-linux-gnu)
[00:39:30]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:39:30]    Compiling string_cache_shared v0.3.0
[00:39:30]    Compiling void v1.0.2
[00:39:30]    Compiling cc v1.0.25
[00:39:30]    Compiling version_check v0.1.5
---
[00:43:14] Documenting book index (x86_64-unknown-linux-gnu)
[00:43:14] Documenting book redirect pages (x86_64-unknown-linux-gnu)
[00:43:15] Documenting stage2 std (x86_64-unknown-linux-gnu)
[00:43:15]     Checking core v0.0.0 (/checkout/src/libcore)
[00:43:48]     Checking rustc-std-workspace-core v1.0.0 (/checkout/src/tools/rustc-std-workspace-core)
[00:43:48]     Checking compiler_builtins v0.1.0 (https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9e)
[00:43:54]     Finished release [optimized] target(s) in 38.88s
[00:43:54]  Documenting core v0.0.0 (/checkout/src/libcore)
[00:44:12] warning: `[chunks]` cannot be resolved, ignoring it...
[00:44:12]    --> libcore/slice/mod.rs:860:52
---
[00:44:12]     |
[00:44:12]     = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`
[00:44:12] 
[00:44:15]     Finished release [optimized] target(s) in 21.66s
[00:44:16]     Checking libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:44:16]     Checking alloc v0.0.0 (/checkout/src/liballoc)
[00:44:17]     Checking unwind v0.0.0 (/checkout/src/libunwind)
[00:44:17]     Checking panic_abort v0.0.0 (/checkout/src/libpanic_abort)
[00:44:21]     Checking rustc_lsan v0.0.0 (/checkout/src/librustc_lsan)
---
[00:44:38]     Checking log v0.4.5
[00:44:38]     Checking unreachable v1.0.0
[00:44:38]     Checking arrayvec v0.4.7
[00:44:38]     Checking owning_ref v0.3.3
[00:44:38]     Checking libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[00:44:38]     Checking smallvec v0.6.5
[00:44:38]     Checking lazy_static v1.1.0
[00:44:38]     Checking rustc_cratesio_shim v0.0.0 (/checkout/src/librustc_cratesio_shim)
[00:44:38]     Checking ena v0.9.3
---
tidy check
[00:45:19] * 568 error codes
[00:45:19] * highest error code: E0721
[00:45:20] * 239 features
[00:45:20] invalid source: "git+https://github.com/alexcrichton/compiler-builtins?branch=rustc-test#1de78e9eed37338be41ef455cb1c43b49a4b786f"
[00:45:20] invalid source: "git+https://github.com/alexcrichton/dlmalloc-rs?branch=rustc-test#28ee12db813a3b650a7c25d1c36d2c17dcb88ae3"
[00:45:20] invalid source: "git+https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db29c691242c394f45f71eb55ac28a9057"

[00:45:20] travis_time:end:tidy:start=1542738400457607969,finish=1542738402145331449,duration=1687723480

[00:45:20] travis_fold:start:stage0-std
---
[00:48:52] .................................................................................................... 100/5041
[00:48:55] .................................................................................................... 200/5041
[00:48:58] .............................ii............................................ii...................ii.. 300/5041
[00:49:00] ..............................................................................................iii... 400/5041
[00:49:03] .....iiiiiiii.iii............................iii...........................................i........ 500/5041
[00:49:10] .................................................................................................... 700/5041
[00:49:16] ..................................................................................i...........i..... 800/5041
[00:49:19] .................................................................................................... 900/5041
[00:49:23] .iiiii..................ii.iiii..................................................................... 1000/5041
---
[00:49:57] .................................................................................................... 2200/5041
[00:50:01] .................................................................................................... 2300/5041
[00:50:04] .................................................................................................... 2400/5041
[00:50:08] .................................................................................................... 2500/5041
[00:50:11] .....................................................................................iiiiiiiii...... 2600/5041
[00:50:18] ...................................................ii............................................... 2800/5041
[00:50:21] .................................................................................................... 2900/5041
[00:50:24] .................................................................................................... 3000/5041
[00:50:28] ..............................................i..................................................... 3100/5041
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:03:55] 
[01:03:55] running 117 tests
[01:03:58] i..ii...iii..iiii.....i...i.........i..iii...........i.....i.....ii...i..i.ii..............i...ii..i 100/117
[01:03:58] i.i.....iiii.....
[01:03:58] 
[01:03:58]  finished in 3.277
[01:03:58] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:04:12] 
[01:04:12] running 118 tests
[01:04:35] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[01:04:38] ......iii.i.....ii
[01:04:38] 
[01:04:38]  finished in 26.723
[01:04:38] travis_fold:end:test_debuginfo

---

[01:14:12] travis_fold:start:test_stage1-alloc
travis_time:start:test_stage1-alloc
Testing alloc stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:14:12]    Compiling libc v0.2.43 (https://github.com/alexcrichton/libc?branch=rustc-test#5e44d7db)
[01:14:13] error[E0259]: the name `core` is defined multiple times
[01:14:13] error[E0259]: the name `core` is defined multiple times
[01:14:13]    --> /cargo/git/checkouts/libc-ada9724775c1b512/5e44d7d/src/lib.rs:105:1
[01:14:13] 102 | extern crate std as core;
[01:14:13] 102 | extern crate std as core;
[01:14:13]     | ------------------------- previous import of the extern crate `core` here
[01:14:13] ...
[01:14:13] 105 | extern crate rustc_std_workspace_core as core;
[01:14:13]     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `core` reimported here
[01:14:13]     |
[01:14:13]     = note: `core` must be defined only once in the type namespace of this module
[01:14:13] help: you can use `as` to change the binding name of the import
[01:14:13]     |
[01:14:13] 105 | extern crate rustc_std_workspace_core as other_core;
[01:14:13] 
[01:14:14] error: aborting due to previous error
[01:14:14] 
[01:14:14] For more information about this error, try `rustc --explain E0259`.
[01:14:14] For more information about this error, try `rustc --explain E0259`.
[01:14:14] error: Could not compile `libc`.
[01:14:14] 
[01:14:14] To learn more, run the command again with --verbose.
[01:14:14] 
[01:14:14] 
[01:14:14] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "alloc" "--" "--quiet"
[01:14:14] 
[01:14:14] 
[01:14:14] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:14:14] Build completed unsuccessfully in 0:28:56
[01:14:14] Build completed unsuccessfully in 0:28:56
[01:14:14] Makefile:58: recipe for target 'check' failed
[01:14:14] make: *** [check] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:005f9bf1
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Tue Nov 20 18:55:35 UTC 2018

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@alexcrichton
Copy link
Member Author

@gnzlbg can you elaborate on the ergonomics point? To be clear we have very few dependencies of the standard library, and we need to be extremely careful about picking up dependencies in libstd because those crates are shipped with basically all code Rust builds against. To that end the fact that anything at all is required to change to get used by libstd seems reasonable, and the 4-5 lines in Cargo.toml really seems quite minimal to me at least

@gnzlbg
Copy link
Contributor

gnzlbg commented Nov 20, 2018

@alexcrichton I agree with everything you said.

With ergonomics I meant that IMO the place to be careful about what goes into libstd is libstd's Cargo.toml. Having to add 4-5 lines of "conventions" to the Cargo.toml of each dependency just feels a bit "hacky", and my worry is that if we end up doing this a lot, we have to carry this convention everywhere.

But this works, adding these lines is not much extra work and I agree that as you said this won't happen often. Finding a better way to do this might not be a problem worth solving, and it should definitely not block this PR.

@alexcrichton
Copy link
Member Author

Heh the Cargo.toml of libstd and other crates in rust-lang/rust are already a comedy of workarounds, so that's sort of the least of my worries. I also feel like none of us can say with a straight face that src/rustc/libc_shim is better than depending directly on crates.io.

@gnzlbg
Copy link
Contributor

gnzlbg commented Nov 20, 2018

Heh the Cargo.toml of libstd and other crates in rust-lang/rust are already a comedy of workarounds, so that's sort of the least of my worries.

That's true, at least this workaround is consistent for all crates involved.

@Mark-Simulacrum
Copy link
Member

Mark-Simulacrum commented Nov 21, 2018

@alexcrichton I think we should go ahead with this; I was suggesting that there might be a better solution somewhere down the line, I don't think there's any point in waiting though.

(Edit: to be clear r=me with the relevant PRs filed/merged in upstream crates)

alexcrichton added a commit to alexcrichton/compiler-builtins that referenced this pull request Nov 21, 2018
This commit prepares to publish the compiler-builtins crate to crates.io
in order for the standard library to directly depend on it from
crates.io in rust-lang/rust#56092
alexcrichton added a commit to alexcrichton/dlmalloc-rs that referenced this pull request Nov 21, 2018
alexcrichton added a commit to alexcrichton/libc that referenced this pull request Nov 21, 2018
This commit prepares the `libc` crate to be included directly into the
standard library via crates.io. More details about this can be found on
rust-lang/rust#56092, but the main idea is that this crate now depends
on core/compiler-builtins explicitly (but off-by-default).

The main caveat here is that this activates `no_core` when building as
part of libstd, which means that it needs to explicitly have an `iter`
and `option` module for the expansion of `for` loops to work.
alexcrichton added a commit to alexcrichton/libc that referenced this pull request Nov 21, 2018
This commit prepares the `libc` crate to be included directly into the
standard library via crates.io. More details about this can be found on
rust-lang/rust#56092, but the main idea is that this crate now depends
on core/compiler-builtins explicitly (but off-by-default).

The main caveat here is that this activates `no_core` when building as
part of libstd, which means that it needs to explicitly have an `iter`
and `option` module for the expansion of `for` loops to work.
alexcrichton added a commit to alexcrichton/libc that referenced this pull request Nov 21, 2018
This commit prepares the `libc` crate to be included directly into the
standard library via crates.io. More details about this can be found on
rust-lang/rust#56092, but the main idea is that this crate now depends
on core/compiler-builtins explicitly (but off-by-default).

The main caveat here is that this activates `no_core` when building as
part of libstd, which means that it needs to explicitly have an `iter`
and `option` module for the expansion of `for` loops to work.
@jethrogb
Copy link
Contributor

Is there a reason stdsimd is still a submodule?

@gnzlbg
Copy link
Contributor

gnzlbg commented Dec 13, 2018

Is there a reason stdsimd is still a submodule?

Why wouldn't it be? Makes it easy to test it, implement new features, reduces testing load (no need to run stdsimd tests in CI, no need to test every single stdsimd PR, etc.) in rust-lang/rust, etc.

@jethrogb
Copy link
Contributor

All those arguments also apply to all the submodules that were removed by the PR.

@gnzlbg
Copy link
Contributor

gnzlbg commented Dec 13, 2018

Ah, I see what you mean. The stdsimd git submodule contains two Rust modules: coresimd, and stdsimd. These modules are not crates. For example, the coresimd::arch module is re-exported by libcore as core::arch, but coresimd::arch depends on other parts of core like core::mem. That is, we cannot add coresimd as a libcore crate dependency, because the dependency would need to be cyclic. (EDIT: something similar applies to the relationship between stdsimd and libstd)

@Xanewok Xanewok mentioned this pull request Dec 13, 2018
@alexcrichton
Copy link
Member Author

@jethrogb stdsimd is special a bit because it actually is included in libcore itself, and we definitely can't have libcore depend on something. Because of that we still use a submodule to include it into libcore's build directly.

I suspect we could build stdsimd as a crate (not coresimd), however, and link that into libstd! I'm not sure it'd have much benefit though as all the real meat is in libcore

@alexcrichton
Copy link
Member Author

er sorry, just ignore what I said and it's all what @gnzlbg said

@gnzlbg
Copy link
Contributor

gnzlbg commented Dec 13, 2018

I suspect we could build stdsimd as a crate (not coresimd), however, and link that into libstd! I'm not sure it'd have much benefit though as all the real meat is in libcore

I think we should do that. Some people have expressed interest into using the same run-time feature detection as libstd in #[no_std] crates, and there is no real reason for the hacky #[path] includes of stdsimd into libstd. In particular, IIUC, we re-export coresimd in libstd again instead of doing so from core::arch directly. So we end up with two copies, one in core::arch, and one in std::arch, instead of just one.

(I can prepare the stdsimd crate for that, but until I'm able to re-compile rust-lang/rust again, I'll need some help on this side).

@alexcrichton
Copy link
Member Author

Oh good point about the runtime feature detection, seems plausible to me to game that out and see what it would look like!

One of the main points of being on crates.io though is offering a stable interface, and I do fear a bit that providing a stable interface might be hard especially for the stability attributes and all that... Seems fine to try though!

bors added a commit that referenced this pull request Dec 14, 2018
Update Clippy

Hopefully unbreaks toolstate: #56092 (comment)
alexcrichton added a commit to alexcrichton/rust that referenced this pull request Dec 14, 2018
This was an accidental regression from rust-lang#56092, but for `no_std` targets
being built and distributed we want to be sure to activate the
compiler-builtins `mem` feature which demangles important memory-related
intrinsics.
bors added a commit that referenced this pull request Dec 15, 2018
std: Activate compiler_builtins `mem` feature for no_std targets

This was an accidental regression from #56092, but for `no_std` targets
being built and distributed we want to be sure to activate the
compiler-builtins `mem` feature which demangles important memory-related
intrinsics.
@alexcrichton alexcrichton restored the no-more-std-subodules branch December 15, 2018 19:56
@alexcrichton alexcrichton deleted the no-more-std-subodules branch December 17, 2018 02:59
@est31 est31 mentioned this pull request May 22, 2019
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.