Skip to content

Commit

Permalink
std: Depend directly on crates.io crates
Browse files Browse the repository at this point in the history
Ever since we added a Cargo-based build system for the compiler the
standard library has always been a little special, it's never been able
to depend on crates.io crates for runtime dependencies. This has been a
result of various limitations, namely that Cargo doesn't understand that
crates from crates.io depend on libcore, so Cargo tries to build crates
before libcore is finished.

I had an idea this afternoon, however, which lifts the strategy
from rust-lang#52919 to directly depend on crates.io crates from the standard
library. After all is said and done this removes a whopping three
submodules that we need to manage!

The basic idea here is that for any crate `std` depends on it adds an
*optional* dependency on an empty crate on crates.io, in this case named
`rustc-std-workspace-core`. This crate is overridden via `[patch]` in
this repository to point to a local crate we write, and *that* has a
`path` dependency on libcore.

Note that all `no_std` crates also depend on `compiler_builtins`, but if
we're not using submodules we can publish `compiler_builtins` to
crates.io and all crates can depend on it anyway! The basic strategy
then looks like:

* The standard library (or some transitive dep) decides to depend on a
  crate `foo`.
* The standard library adds

  ```toml
  [dependencies]
  foo = { version = "0.1", features = ['rustc-dep-of-std'] }
  ```
* The crate `foo` has an optional dependency on `rustc-std-workspace-core`
* The crate `foo` has an optional dependency on `compiler_builtins`
* The crate `foo` has a feature `rustc-dep-of-std` which activates these
  crates and any other necessary infrastructure in the crate.

A sample commit for `dlmalloc` [turns out to be quite simple][commit].
After that all `no_std` crates should largely build "as is" and still be
publishable on crates.io! Notably they should be able to continue to use
stable Rust if necessary, since the `rename-dependency` feature of Cargo
is soon stabilizing.

As a proof of concept, this commit removes the `dlmalloc`,
`libcompiler_builtins`, and `libc` submodules from this repository. Long
thorns in our side these are now gone for good and we can directly
depend on crates.io! It's hoped that in the long term we can bring in
other crates as necessary, but for now this is largely intended to
simply make it easier to manage these crates and remove submodules.

This should be a transparent non-breaking change for all users, but one
possible stickler is that this almost for sure breaks out-of-tree
`std`-building tools like `xargo` and `cargo-xbuild`. I think it should
be relatively easy to get them working, however, as all that's needed is
an entry in the `[patch]` section used to build the standard library.
Hopefully we can work with these tools to solve this problem!

[commit]: alexcrichton/dlmalloc-rs@28ee12d
  • Loading branch information
alexcrichton committed Dec 12, 2018
1 parent 4c0116e commit 4c21a3b
Show file tree
Hide file tree
Showing 99 changed files with 231 additions and 314 deletions.
12 changes: 0 additions & 12 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@
[submodule "src/rust-installer"]
path = src/tools/rust-installer
url = https://github.com/rust-lang/rust-installer.git
[submodule "src/liblibc"]
path = src/liblibc
url = https://github.com/rust-lang/libc.git
[submodule "src/doc/nomicon"]
path = src/doc/nomicon
url = https://github.com/rust-lang-nursery/nomicon.git
Expand All @@ -23,9 +20,6 @@
[submodule "src/tools/rls"]
path = src/tools/rls
url = https://github.com/rust-lang-nursery/rls.git
[submodule "src/libcompiler_builtins"]
path = src/libcompiler_builtins
url = https://github.com/rust-lang-nursery/compiler-builtins.git
[submodule "src/tools/clippy"]
path = src/tools/clippy
url = https://github.com/rust-lang-nursery/rust-clippy.git
Expand All @@ -35,9 +29,6 @@
[submodule "src/tools/miri"]
path = src/tools/miri
url = https://github.com/solson/miri.git
[submodule "src/dlmalloc"]
path = src/dlmalloc
url = https://github.com/alexcrichton/dlmalloc-rs.git
[submodule "src/doc/rust-by-example"]
path = src/doc/rust-by-example
url = https://github.com/rust-lang/rust-by-example.git
Expand Down Expand Up @@ -67,6 +58,3 @@
[submodule "src/doc/edition-guide"]
path = src/doc/edition-guide
url = https://github.com/rust-lang-nursery/edition-guide
[submodule "src/rust-sgx"]
path = src/rust-sgx
url = https://github.com/fortanix/rust-sgx
85 changes: 47 additions & 38 deletions Cargo.lock
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ dependencies = [
name = "alloc"
version = "0.0.0"
dependencies = [
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"rand 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
Expand Down Expand Up @@ -407,10 +407,11 @@ dependencies = [

[[package]]
name = "compiler_builtins"
version = "0.0.0"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cc 1.0.25 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"rustc-std-workspace-core 1.0.0",
]

[[package]]
Expand Down Expand Up @@ -456,7 +457,7 @@ dependencies = [
name = "core"
version = "0.0.0"
dependencies = [
"rand 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
Expand Down Expand Up @@ -658,10 +659,12 @@ dependencies = [

[[package]]
name = "dlmalloc"
version = "0.0.0"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"compiler_builtins 0.0.0",
"core 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.45 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-std-workspace-core 1.0.0",
]

[[package]]
Expand Down Expand Up @@ -814,10 +817,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"

[[package]]
name = "fortanix-sgx-abi"
version = "0.0.0"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"compiler_builtins 0.0.0",
"core 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-std-workspace-core 1.0.0",
]

[[package]]
Expand Down Expand Up @@ -1138,18 +1142,13 @@ name = "lazycell"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"

[[package]]
name = "libc"
version = "0.0.0"
dependencies = [
"compiler_builtins 0.0.0",
"core 0.0.0",
]

[[package]]
name = "libc"
version = "0.2.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rustc-std-workspace-core 1.0.0",
]

[[package]]
name = "libgit2-sys"
Expand Down Expand Up @@ -1520,19 +1519,19 @@ dependencies = [
name = "panic_abort"
version = "0.0.0"
dependencies = [
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"libc 0.0.0",
"libc 0.2.45 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
name = "panic_unwind"
version = "0.0.0"
dependencies = [
"alloc 0.0.0",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"libc 0.0.0",
"libc 0.2.45 (registry+https://github.com/rust-lang/crates.io-index)",
"unwind 0.0.0",
]

Expand Down Expand Up @@ -1683,7 +1682,7 @@ name = "profiler_builtins"
version = "0.0.0"
dependencies = [
"cc 1.0.25 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
]

Expand Down Expand Up @@ -1814,7 +1813,7 @@ name = "rand_chacha"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand_core 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
]

Expand All @@ -1836,7 +1835,7 @@ name = "rand_hc"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand_core 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
Expand All @@ -1861,7 +1860,7 @@ name = "rand_xorshift"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand_core 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
Expand Down Expand Up @@ -2242,6 +2241,13 @@ name = "rustc-serialize"
version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"

[[package]]
name = "rustc-std-workspace-core"
version = "1.0.0"
dependencies = [
"core 0.0.0",
]

[[package]]
name = "rustc-workspace-hack"
version = "1.0.0"
Expand Down Expand Up @@ -2284,7 +2290,7 @@ dependencies = [
"alloc 0.0.0",
"build_helper 0.1.0",
"cmake 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
]

Expand Down Expand Up @@ -2479,7 +2485,7 @@ dependencies = [
"alloc 0.0.0",
"build_helper 0.1.0",
"cmake 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
]

Expand Down Expand Up @@ -2531,7 +2537,7 @@ dependencies = [
"alloc 0.0.0",
"build_helper 0.1.0",
"cmake 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
]

Expand Down Expand Up @@ -2644,7 +2650,7 @@ dependencies = [
"alloc 0.0.0",
"build_helper 0.1.0",
"cmake 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
]

Expand Down Expand Up @@ -2877,15 +2883,15 @@ dependencies = [
"alloc 0.0.0",
"build_helper 0.1.0",
"cc 1.0.25 (registry+https://github.com/rust-lang/crates.io-index)",
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"dlmalloc 0.0.0",
"fortanix-sgx-abi 0.0.0",
"libc 0.0.0",
"dlmalloc 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"fortanix-sgx-abi 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.45 (registry+https://github.com/rust-lang/crates.io-index)",
"panic_abort 0.0.0",
"panic_unwind 0.0.0",
"profiler_builtins 0.0.0",
"rand 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc_asan 0.0.0",
"rustc_lsan 0.0.0",
"rustc_msan 0.0.0",
Expand Down Expand Up @@ -3217,9 +3223,9 @@ dependencies = [
name = "unwind"
version = "0.0.0"
dependencies = [
"compiler_builtins 0.0.0",
"compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"core 0.0.0",
"libc 0.0.0",
"libc 0.2.45 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
Expand Down Expand Up @@ -3397,6 +3403,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
"checksum colored 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b0aa3473e85a3161b59845d6096b289bb577874cafeaf75ea1b1beaa6572c7fc"
"checksum commoncrypto 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d056a8586ba25a1e4d61cb090900e495952c7886786fc55f909ab2f819b69007"
"checksum commoncrypto-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1fed34f46747aa73dfaa578069fd8279d2818ade2b55f38f22a9401c7f4083e2"
"checksum compiler_builtins 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "8ad611263b9f31bdb66e66227d3b781600fd1e68d5deee29b23f5e2ac9cb4892"
"checksum compiletest_rs 0.3.17 (registry+https://github.com/rust-lang/crates.io-index)" = "89747fe073b7838343bd2c2445e7a7c2e0d415598f8925f0fa9205b9cdfc48cb"
"checksum core-foundation 0.6.3 (registry+https://github.com/rust-lang/crates.io-index)" = "4e2640d6d0bf22e82bed1b73c6aef8d5dd31e5abe6666c57e6d45e2649f4f887"
"checksum core-foundation-sys 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e7ca8a5221364ef15ce201e8ed2f609fc312682a8f4e0e3d4aa5879764e0fa3b"
Expand All @@ -3418,6 +3425,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
"checksum diff 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "3c2b69f912779fbb121ceb775d74d51e915af17aaebc38d28a592843a2dd0a3a"
"checksum difference 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "524cbf6897b527295dff137cec09ecf3a05f4fddffd7dfcd1585403449e74198"
"checksum directories 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "72d337a64190607d4fcca2cb78982c5dd57f4916e19696b48a575fa746b6cb0f"
"checksum dlmalloc 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4c46c65de42b063004b31c67a98abe071089b289ff0919c660ed7ff4f59317f8"
"checksum either 1.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "3be565ca5c557d7f59e7cfcf1844f9e3033650c929c6566f511e8005f205c1d0"
"checksum elasticlunr-rs 2.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "a99a310cd1f9770e7bf8e48810c7bcbb0e078c8fb23a8c7bcf0da4c2bf61a455"
"checksum ena 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f56c93cc076508c549d9bb747f79aa9b4eb098be7b8cad8830c3137ef52d1e00"
Expand All @@ -3434,6 +3442,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
"checksum fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)" = "2fad85553e09a6f881f739c29f0b00b0f01357c743266d478b68951ce23285f3"
"checksum foreign-types 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
"checksum foreign-types-shared 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
"checksum fortanix-sgx-abi 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "26105e20b4c3f7a319db1376b54ac9a46e5761e949405553375095d05a0cee4d"
"checksum fs2 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)" = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213"
"checksum fs_extra 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5f2a4a2034423744d2cc7ca2068453168dcdb82c438419e639a26bd87839c674"
"checksum fst 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d94485a00b1827b861dd9d1a2cc9764f9044d4c535514c0760a5a2012ef3399f"
Expand Down
4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ rustfmt-nightly = { path = "src/tools/rustfmt" }
# here
rustc-workspace-hack = { path = 'src/tools/rustc-workspace-hack' }

# See comments in `tools/rustc-std-workspace-core/README.md` for what's going on
# here
rustc-std-workspace-core = { path = 'src/tools/rustc-std-workspace-core' }

[patch."https://github.com/rust-lang/rust-clippy"]
clippy_lints = { path = "src/tools/clippy/clippy_lints" }
rustc_tools_util = { path = "src/tools/clippy/rustc_tools_util" }
5 changes: 2 additions & 3 deletions src/bootstrap/compile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,10 @@ pub fn std_cargo(builder: &Builder,

if builder.no_std(target) == Some(true) {
// for no-std targets we only compile a few no_std crates
cargo.arg("--features").arg("c mem")
cargo
.args(&["-p", "alloc"])
.args(&["-p", "compiler_builtins"])
.arg("--manifest-path")
.arg(builder.src.join("src/rustc/compiler_builtins_shim/Cargo.toml"));
.arg(builder.src.join("src/liballoc/Cargo.toml"));
} else {
let features = builder.std_features();

Expand Down
15 changes: 3 additions & 12 deletions src/bootstrap/dist.rs
Original file line number Diff line number Diff line change
Expand Up @@ -857,12 +857,9 @@ impl Step for Src {
// (essentially libstd and all of its path dependencies)
let std_src_dirs = [
"src/build_helper",
"src/dlmalloc",
"src/liballoc",
"src/libbacktrace",
"src/libcompiler_builtins",
"src/libcore",
"src/liblibc",
"src/libpanic_abort",
"src/libpanic_unwind",
"src/librustc_asan",
Expand All @@ -871,21 +868,15 @@ impl Step for Src {
"src/librustc_tsan",
"src/libstd",
"src/libunwind",
"src/rustc/compiler_builtins_shim",
"src/rustc/libc_shim",
"src/rustc/dlmalloc_shim",
"src/rustc/fortanix-sgx-abi_shim",
"src/libtest",
"src/libterm",
"src/libprofiler_builtins",
"src/stdsimd",
"src/libproc_macro",
];
let std_src_dirs_exclude = [
"src/libcompiler_builtins/compiler-rt/test",
"src/tools/rustc-std-workspace-core",
];

copy_src_dirs(builder, &std_src_dirs[..], &std_src_dirs_exclude[..], &dst_src);
copy_src_dirs(builder, &std_src_dirs[..], &[], &dst_src);
for file in src_files.iter() {
builder.copy(&builder.src.join(file), &dst_src.join(file));
}
Expand All @@ -909,7 +900,7 @@ impl Step for Src {
}
}

const CARGO_VENDOR_VERSION: &str = "0.1.19";
const CARGO_VENDOR_VERSION: &str = "0.1.22";

#[derive(Debug, PartialOrd, Ord, Copy, Clone, Hash, PartialEq, Eq)]
pub struct PlainSourceTarball;
Expand Down
8 changes: 4 additions & 4 deletions src/bootstrap/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1134,10 +1134,10 @@ impl Build {
let krate = &self.crates[&krate];
if krate.is_local(self) {
ret.push(krate);
for dep in &krate.deps {
if visited.insert(dep) && dep != "build_helper" {
list.push(*dep);
}
}
for dep in &krate.deps {
if visited.insert(dep) && dep != "build_helper" {
list.push(*dep);
}
}
}
Expand Down
5 changes: 1 addition & 4 deletions src/bootstrap/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1558,10 +1558,7 @@ impl Step for Crate {
let builder = run.builder;
run = run.krate("test");
for krate in run.builder.in_tree_crates("std") {
if krate.is_local(&run.builder)
&& !(krate.name.starts_with("rustc_") && krate.name.ends_with("san"))
&& krate.name != "dlmalloc"
{
if !(krate.name.starts_with("rustc_") && krate.name.ends_with("san")) {
run = run.path(krate.local_path(&builder).to_str().unwrap());
}
}
Expand Down
Loading

0 comments on commit 4c21a3b

Please sign in to comment.