reorganize/refactor bootstrap codebase

[onur-ozkan]

Currently, bootstrap stores everything on the root path, including very large modules, which makes things very hard to scale and adds too much complexity.

This PR has the following objectives:

• Improving scalability.
• Making bootstrap source more understandable for the new contributors(or for everyone).
• Improving the development experience and making maintenance easier for the bootstrap team.

The new source structure:

.
├── defaults
│   ├── README.md
│   ├── config.codegen.toml
│   ├── config.compiler.toml
│   ├── config.dist.toml
│   ├── config.library.toml
│   └── config.tools.toml
├── mk
│   └── Makefile.in
├── src
│   ├── bin
│   │   ├── main.rs
│   │   ├── rustc.rs
│   │   ├── rustdoc.rs
│   │   └── sccache-plus-cl.rs
│   ├── core
│   │   ├── build_steps
│   │   │   ├── check.rs
│   │   │   ├── clean.rs
│   │   │   ├── compile.rs
│   │   │   ├── dist.rs
│   │   │   ├── doc.rs
│   │   │   ├── format.rs
│   │   │   ├── install.rs
│   │   │   ├── llvm.rs
│   │   │   ├── mod.rs
│   │   │   ├── run.rs
│   │   │   ├── setup.rs
│   │   │   ├── suggest.rs
│   │   │   ├── synthetic_targets.rs
│   │   │   ├── test.rs
│   │   │   ├── tool.rs
│   │   │   └── toolstate.rs
│   │   ├── config
│   │   │   ├── config.rs
│   │   │   ├── flags.rs
│   │   │   └── mod.rs
│   │   ├── builder.rs
│   │   ├── download.rs
│   │   ├── metadata.rs
│   │   ├── mod.rs
│   │   └── sanity.rs
│   ├── tests
│   │   ├── builder.rs
│   │   ├── config.rs
│   │   └── setup.rs
│   ├── utils
│   │   ├── bin_helpers.rs
│   │   ├── cache.rs
│   │   ├── cc_detect.rs
│   │   ├── channel.rs
│   │   ├── dylib.rs
│   │   ├── helpers.rs
│   │   ├── job.rs
│   │   ├── metrics.rs
│   │   ├── mod.rs
│   │   ├── render_tests.rs
│   │   └── tarball.rs
│   └── lib.rs
├── Cargo.lock
├── Cargo.toml
├── README.md
├── bootstrap.py
├── bootstrap_test.py
├── build.rs
├── configure.py
└── download-ci-llvm-stamp

The next step involves:

• Adding more doc-comments to the bootstrap internals (although we already have a decent amount, there is space for improvement).
• Breaking large modules into smaller, more manageable modules.
• Significantly increasing our unit test coverage (which is currently lacking).

This PR should serve as an initial step to make the tasks above much more easier.

r? Mark-Simulacrum

[rustbot]

r? @albertlarsan68

(rustbot has picked a reviewer for you, use r? to override)

[onur-ozkan]

I am open to making improvements on this work if anyone has suggestions.

cc @rust-lang/bootstrap

[bors]

☔ The latest upstream changes (presumably #116514) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #116448) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #116407) made this pull request unmergeable. Please resolve the merge conflicts.

[onur-ozkan]

@bors r=Mark-Simulacrum

[bors]

📌 Commit 3ecff1b has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

⌛ Testing commit 3ecff1b with merge 347452e...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 347452e to master...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 347452e to master...

[rust-timer]

Finished benchmarking commit (347452e): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.8%, 0.8%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.3%	[-1.3%, -1.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.2%	[-1.3%, 0.8%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 626.612s -> 625.857s (-0.12%)
Artifact size: 305.59 MiB -> 305.58 MiB (-0.00%)