bootstrap panic generating rustc docs while cross-compiling with x.py dist

[catamorphism]

With RUST_BACKTRACE=full ./x.py dist --target riscv64gc-unknown-linux-musl --verbose and the following config.toml file:

profile = "codegen"

[target.riscv64gc-unknown-linux-musl]
ar = "riscv64-unknown-linux-musl-ar"
linker = "riscv64-unknown-linux-musl-gcc"
cc = "riscv64-unknown-linux-musl-gcc"
cxx = "riscv64-unknown-linux-musl-g++"
musl-root = "/home/tjc/riscv64-linux-musl-cross/riscv64-linux-musl"

the rust-installer command fails while installing docs because it can't find the target's compiler-doc directory. This makes sense because the compiler-doc directory is created in one of two places:

https://github.com/rust-lang/rust/blob/master/src/bootstrap/doc.rs#L631
https://github.com/rust-lang/rust/blob/master/src/bootstrap/doc.rs#L780

and both of these are only invoked for the host (in the enclosing impls of each of those call sites, ONLY_HOSTS is set to true).

So the problem is that the dist::RustcDocs step, which relies on the existence of the compiler-doc directory (see https://github.com/rust-lang/rust/blob/master/src/bootstrap/dist.rs#L150 ), is run for the target ( https://github.com/rust-lang/rust/blob/master/src/bootstrap/dist.rs#L140 ), but the steps that would create that directory have only been run for the host.

I was able to fix this by adding const ONLY_HOSTS: bool = true inside the impl Step for RustcDocs ( https://github.com/rust-lang/rust/blob/master/src/bootstrap/dist.rs#L130 ), so that rustc docs aren't generated for the target. I'm not sure if that's the right solution, and in general I'm unsure about the logic for which pieces of documentation are generated for the host and which ones for the target under cross-compilation.

The end of the output, showing that rust-installer fails due to reading a nonexistent directory, is:

Dist rust-docs-1.70.0-dev-riscv64gc-unknown-linux-musl
running: LD_LIBRARY_PATH="/home/tjc/rust/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/home/tjc/rust/build/x86_64-unknown-linux-gnu/stage0/lib:/home/tjc/WebKit/WebKitBuild/Debug/lib:/home/tjc/WebKit/WebKitBuild/GTK/Debug/lib:/home/tjc/WebKit/WebKitBuild/Debug/lib:/home/tjc/WebKit/WebKitBuild/GTK/Debug/lib:/home/tjc/WebKit/WebKitBuild/Debug/lib:/home/tjc/WebKit/WebKitBuild/GTK/Debug/lib:" RUSTC="/home/tjc/rust/build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "/home/tjc/rust/build/x86_64-unknown-linux-gnu/stage0-tools-bin/rust-installer" "generate" "--image-dir" "/home/tjc/rust/build/tmp/tarball/rust-docs/riscv64gc-unknown-linux-musl/image" "--component-name=rust-docs" "--bulk-dirs" "share/doc/rust/html" "--rel-manifest-dir=rustlib" "--legacy-manifest-dirs=rustlib,cargo" "--product-name=Rust Documentation" "--success-message=rust-docs installed." "--package-name=rust-docs-1.70.0-dev-riscv64gc-unknown-linux-musl" "--non-installed-overlay" "/home/tjc/rust/build/tmp/tarball/rust-docs/riscv64gc-unknown-linux-musl/overlay" "--output-dir" "/home/tjc/rust/build/dist" "--work-dir" "/home/tjc/rust/build/tmp/tarball/rust-docs/riscv64gc-unknown-linux-musl" "--compression-profile" "fast"
	finished in 3.464 seconds
thread 'main' panicked at 'could not read dir "/home/tjc/rust/build/riscv64gc-unknown-linux-musl/compiler-doc": Os { code: 2, kind: NotFound, message: "No such file or directory" }', lib.rs:1538:25
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b955c8271da80a1af8a1d54c4e1bbdaf51b032e9/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/b955c8271da80a1af8a1d54c4e1bbdaf51b032e9/library/core/src/panicking.rs:64:14
   2: bootstrap::Build::read_dir
             at ./src/bootstrap/lib.rs:1538:25
   3: bootstrap::Build::cp_r
             at ./src/bootstrap/lib.rs:1450:18
   4: bootstrap::tarball::Tarball::add_dir
             at ./src/bootstrap/tarball.rs:205:9
   5: bootstrap::tarball::Tarball::add_bulk_dir
             at ./src/bootstrap/tarball.rs:210:9
   6: <bootstrap::dist::RustcDocs as bootstrap::builder::Step>::run
             at ./src/bootstrap/dist.rs:150:9
   7: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:2019:23
   8: <bootstrap::dist::RustcDocs as bootstrap::builder::Step>::make_run
             at ./src/bootstrap/dist.rs:140:9
   9: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:301:13
  10: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:338:21
  11: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:924:9
  12: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:904:9
  13: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:683:13
  14: bootstrap::main
             at ./src/bootstrap/bin/main.rs:58:5
  15: core::ops::function::FnOnce::call_once
             at /rustc/b955c8271da80a1af8a1d54c4e1bbdaf51b032e9/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "/home/tjc/rust/./x.py", line 29, in <module>
    bootstrap.main()
  File "/home/tjc/rust/src/bootstrap/bootstrap.py", line 949, in main
    bootstrap(args)
  File "/home/tjc/rust/src/bootstrap/bootstrap.py", line 924, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/home/tjc/rust/src/bootstrap/bootstrap.py", line 167, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /home/tjc/rust/build/bootstrap/debug/bootstrap dist --target riscv64gc-unknown-linux-musl --verbose

[catamorphism]

n.b. I'm not sure I actually was using x.py dist correctly -- my goal is to cross-compile snapshots that I can use for bootstrapping Rust on a riscv64 system, and based on the output, I think I probably should have used:

./x.py dist --host riscv64gc-unknown-linux-musl --target riscv64gc-unknown-linux-musl

instead. But this still shouldn't happen.

[jyn514]

I think this will be fixed by #109880.

[jyn514]

In general RustcDocs is pretty broken, see #108535

[tgross35]

#109880 merged but this is still unfortunately still an issue :( reproduced with ./x dist --target mips64-openwrt-linux-musl --host x86_64-unknown-linux-gnu

@catamorphism did you ever find a quick workaround?