Introduce -Zsplit-metadata option #93945

bjorn3 · 2022-02-12T17:58:41Z

This will split the crate metadata out of library files. Instead only the svh is preserved to allow for loading the right rmeta file. This significicantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate.

Fixes #23366
Fixes #57076
Fixes #10786 (through 453d9a0bf778e385b45ecced681a71315fe8517e)
Should fix #82151 (through 453d9a0bf778e385b45ecced681a71315fe8517e, unconfirmed)

rust-highfive · 2022-02-12T17:58:44Z

r? @davidtwco

(rust-highfive has picked a reviewer for you, use r? to override)

bjorn3 · 2022-02-12T17:58:55Z

@bors try @rust-timer queue

rust-timer · 2022-02-12T17:58:56Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2022-02-12T17:59:03Z

⌛ Trying commit bf8a8f82a73414610384b4e36323c0966db8e908 with merge 86e45ec336f0ccde044af1ace8b3c425239d01f2...

bors · 2022-02-12T18:12:36Z

💔 Test failed - checks-actions

fbstj · 2022-02-12T18:16:16Z

compiler/rustc_metadata/src/locator.rs

+                        // The file was present and created by the same compiler version, but we
+                        // couldn't load it for some reason.  Give a hard error instead of silently
+                        // ignoring it, but only if we would have given an error anyway.


this comment doesn't make sense with the code below it

I'm not sure where it came from. Maybe it was correct in an old draft? I finished a draft from September last year just now.

Dylib metadata compression increases code complexity and will become less important once -Zsplit-metadata is introduced

I don't think it is necessary anymore. As I understand it from issue 39504 the original problem was that rustbuild changed a hardlink in the cargo build dir to point to copy in the sysroot while cargo may have hardlinked it to the original first. I don't think this happens anymore and as such this workaround is no longer necessary.

If a crate is both an rlib and cdylib, the metadata loader would previously incorrectly assume that it is possible to link against the cdylib. Cdylibs don't export all symbols required for linking as rust library. Possibly fixes issue number 82151

This will make it easier to report rustc versions for older metadata formats.

This will split the crate metadata out of library files. Instead only the svh is preserved to allow for loading the right rmeta file. This significicantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate.

bjorn3 · 2022-02-12T18:20:23Z

@bors try @rust-timer queue

rust-timer · 2022-02-12T18:20:24Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2022-02-12T18:20:30Z

⌛ Trying commit af25358 with merge 3e62ae8cce9be671b29a9715b3d390e4ad0c92d2...

jyn514 · 2022-02-13T07:48:33Z

Would this allow metadata to be reused between cargo build and cargo check? rust-lang/cargo#3501

bjorn3 · 2022-02-13T08:17:00Z

Unfortunately not. Among other things rustc skips MIR encoding of generic and inline functions in check mode I believe.

rust-log-analyzer · 2022-02-13T11:42:14Z

The job x86_64-gnu-llvm-12 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

   Compiling std_detect v0.1.5 (/checkout/library/stdarch/crates/std_detect)
   Compiling hashbrown v0.12.0
   Compiling miniz_oxide v0.4.0
   Compiling addr2line v0.16.0
thread 'rustc' panicked at 'not yet implemented: return error', compiler/rustc_metadata/src/locator.rs:544:37
stack backtrace:
   0:     0x7f7549eb86dc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h4c1ed0e5024ec2d0
   1:     0x7f7549f26e4e - core::fmt::write::h04f3cb9c5bd3369c
   2:     0x7f7549ea76e1 - std::io::Write::write_fmt::hec437177037c4304
   3:     0x7f7549eb850b - std::sys_common::backtrace::print::h9084352f72bce3d6
   4:     0x7f7549ebccd4 - std::panicking::default_hook::{{closure}}::h4e03c6131e9375ec
   5:     0x7f7549ebc8b6 - std::panicking::default_hook::h7c7d01494eae6d70
   6:     0x7f754a992e11 - rustc_driver[a93e051fcd12f9f8]::DEFAULT_HOOK::{closure#0}::{closure#0}
   7:     0x7f7549ebd3f3 - std::panicking::rust_panic_with_hook::hea69582e6c4a33d9
   8:     0x7f7549ebd207 - std::panicking::begin_panic_handler::{{closure}}::he6a21a353581f6cb
   9:     0x7f7549eb8bf4 - std::sys_common::backtrace::__rust_end_short_backtrace::h2d0a50649be03b70
  10:     0x7f7549ebcec9 - rust_begin_unwind
  11:     0x7f7549e739d3 - core::panicking::panic_fmt::h7802ba5043cb2ca5
  12:     0x7f754c72edd8 - <rustc_metadata[2c610e6225a46cf]::locator::CrateLocator>::extract_one
  13:     0x7f754c72c692 - <rustc_metadata[2c610e6225a46cf]::locator::CrateLocator>::extract_lib
  14:     0x7f754c72a311 - <rustc_metadata[2c610e6225a46cf]::locator::CrateLocator>::maybe_load_library_crate
  15:     0x7f754c6d203a - <rustc_metadata[2c610e6225a46cf]::creader::CrateLoader>::load
  16:     0x7f754c6ce55a - <rustc_metadata[2c610e6225a46cf]::creader::CrateLoader>::maybe_resolve_crate
  17:     0x7f754c6cd6d2 - <rustc_metadata[2c610e6225a46cf]::creader::CrateLoader>::resolve_crate
  18:     0x7f754c6d4b1c - <rustc_metadata[2c610e6225a46cf]::creader::CrateLoader>::process_extern_crate
  19:     0x7f754b488a43 - <rustc_resolve[e93c816d39f0de31]::build_reduced_graph::BuildReducedGraphVisitor as rustc_ast[fb718f914770ae88]::visit::Visitor>::visit_item
  20:     0x7f754b4ca29d - rustc_ast[fb718f914770ae88]::visit::walk_crate::<rustc_resolve[e93c816d39f0de31]::build_reduced_graph::BuildReducedGraphVisitor>
  21:     0x7f754b48b2fc - <rustc_resolve[e93c816d39f0de31]::build_reduced_graph::BuildReducedGraphVisitor as rustc_ast[fb718f914770ae88]::visit::Visitor>::visit_crate
  22:     0x7f754b4ab0cd - <rustc_resolve[e93c816d39f0de31]::Resolver as rustc_expand[3f6f1937b52b127f]::base::ResolverExpand>::visit_ast_fragment_with_placeholders
  23:     0x7f754c8bf357 - <rustc_expand[3f6f1937b52b127f]::expand::MacroExpander>::collect_invocations
  24:     0x7f754c8ba6df - <rustc_expand[3f6f1937b52b127f]::expand::MacroExpander>::fully_expand_fragment
  25:     0x7f754c8ba38b - <rustc_expand[3f6f1937b52b127f]::expand::MacroExpander>::expand_crate
  26:     0x7f754aaa5a63 - <rustc_session[22750a20d77a6d56]::session::Session>::time::<core[ba5cb6be30a93795]::result::Result<rustc_ast[fb718f914770ae88]::ast::Crate, rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_interface[768531affdc57d1c]::passes::configure_and_expand::{closure#1}>
  27:     0x7f754ab3c28d - rustc_interface[768531affdc57d1c]::passes::configure_and_expand
  28:     0x7f754aa89250 - <rustc_interface[768531affdc57d1c]::passes::boxed_resolver::BoxedResolver>::access::<<rustc_interface[768531affdc57d1c]::queries::Queries>::expansion::{closure#0}::{closure#0}, core[ba5cb6be30a93795]::result::Result<rustc_ast[fb718f914770ae88]::ast::Crate, rustc_errors[938271d98dcf7a]::ErrorReported>>
  29:     0x7f754ab1cc3d - <rustc_interface[768531affdc57d1c]::queries::Queries>::expansion
  30:     0x7f754a94412f - <rustc_interface[768531affdc57d1c]::interface::Compiler>::enter::<rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}::{closure#2}, core[ba5cb6be30a93795]::result::Result<core[ba5cb6be30a93795]::option::Option<rustc_interface[768531affdc57d1c]::queries::Linker>, rustc_errors[938271d98dcf7a]::ErrorReported>>
  31:     0x7f754a981ac9 - rustc_span[db79cfb26f8098d6]::with_source_map::<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_interface[768531affdc57d1c]::interface::create_compiler_and_run<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>::{closure#1}>
  32:     0x7f754a94242b - rustc_interface[768531affdc57d1c]::interface::create_compiler_and_run::<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>
  33:     0x7f754a91f57e - <scoped_tls[9ef8146b5f47b671]::ScopedKey<rustc_span[db79cfb26f8098d6]::SessionGlobals>>::set::<rustc_interface[768531affdc57d1c]::util::run_in_thread_pool_with_globals<rustc_interface[768531affdc57d1c]::interface::run_compiler<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#0}::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>
  34:     0x7f754a91cd05 - std[39bfa0482ff6d9eb]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[768531affdc57d1c]::util::run_in_thread_pool_with_globals<rustc_interface[768531affdc57d1c]::interface::run_compiler<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>
  35:     0x7f754a984e21 - std[39bfa0482ff6d9eb]::panicking::try::<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, core[ba5cb6be30a93795]::panic::unwind_safe::AssertUnwindSafe<<std[39bfa0482ff6d9eb]::thread::Builder>::spawn_unchecked_<rustc_interface[768531affdc57d1c]::util::run_in_thread_pool_with_globals<rustc_interface[768531affdc57d1c]::interface::run_compiler<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#1}::{closure#0}>>
  36:     0x7f754a91f1f2 - <<std[39bfa0482ff6d9eb]::thread::Builder>::spawn_unchecked_<rustc_interface[768531affdc57d1c]::util::run_in_thread_pool_with_globals<rustc_interface[768531affdc57d1c]::interface::run_compiler<core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>, rustc_driver[a93e051fcd12f9f8]::run_compiler::{closure#1}>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#0}, core[ba5cb6be30a93795]::result::Result<(), rustc_errors[938271d98dcf7a]::ErrorReported>>::{closure#1} as core[ba5cb6be30a93795]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  37:     0x7f7549ecc1c3 - std::sys::unix::thread::Thread::new::thread_start::h06eaa01755c40ab7
  38:     0x7f754423e609 - start_thread
  39:     0x7f7549d35293 - clone
  40:                0x0 - <unknown>
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.


note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.60.0-nightly (e6e90a610 2022-02-13) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type dylib --crate-type rlib -C prefer-dynamic -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=0 -C debug-assertions=on -C symbol-mangling-version=legacy -Z unstable-options -Z split-metadata -Z macro-backtrace -C link-args=-Wl,-z,origin -C link-args=-Wl,-rpath,$ORIGIN/../lib -C prefer-dynamic -C llvm-args=-import-instr-limit=10 -C embed-bitcode=yes -Z crate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/") -Z binary-dep-depinfo -Z force-unstable-if-unmarked
note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack

bjorn3 · 2022-02-13T11:51:09Z

I think what happens is that when --extern some_lib=deps/libsome_lib.rlib -Ldeps is passed, it will try to load the rlib first and because the rlib is loaded, the rmeta file in deps is skipped, which fails as now there is no metadata loaded anymore.

bors · 2022-02-18T02:48:15Z

☔ The latest upstream changes (presumably #94103) made this pull request unmergeable. Please resolve the merge conflicts.

Dylan-DPC · 2022-04-05T17:12:16Z

closing this due to inactivity

Introduce -Zsplit-metadata option This will split the crate metadata out of library files. Instead only the svh and a bit of extra metadata is preserved to allow for loading the right rmeta file. This significantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate. A fair amount of the complexity in this PR is to work around the fact that cargo doesn't directly support this option yet. Fixes rust-lang#23366 Fixes rust-lang#57076 Revives rust-lang#93945

bjorn3 · 2024-02-09T19:01:22Z

Opened #120855 to revive this.

bjorn3 added A-metadata Area: Crate metadata T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 12, 2022

rust-highfive assigned davidtwco Feb 12, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 12, 2022

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 12, 2022

This comment was marked as outdated.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 12, 2022

fbstj reviewed Feb 12, 2022

View reviewed changes

bjorn3 added 7 commits February 12, 2022 19:20

Don't compress dylib metadata

6786a09

Dylib metadata compression increases code complexity and will become less important once -Zsplit-metadata is introduced

Move metadata header and version checks together

99f92df

This will make it easier to report rustc versions for older metadata formats.

Use to_le_bytes and from_le_bytes instead of manual byte shuffling

6d5a457

Rustbuild: enable -Zsplit-metadata for stage != 0

af25358

bjorn3 force-pushed the split_metadata3 branch from bf8a8f8 to af25358 Compare February 12, 2022 18:20

This comment has been minimized.

Sign in to view

Fix encoder

7969fd3

petrochenkov self-assigned this Feb 13, 2022

Don't try to get crate root for reference only metadata

5f0a9f8

bjorn3 force-pushed the split_metadata3 branch from 8738e49 to 5f0a9f8 Compare February 13, 2022 11:19

This comment has been minimized.

Sign in to view

Ship rmeta files

c0d8f3e

bjorn3 force-pushed the split_metadata3 branch from 6a21ba1 to c0d8f3e Compare February 13, 2022 11:34

bjorn3 marked this pull request as draft February 13, 2022 11:50

davidtwco removed their assignment Feb 16, 2022

Dylan-DPC closed this Apr 5, 2022

Dylan-DPC added S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 5, 2022

This was referenced May 15, 2022

Move metadata out of dylibs #23366

Open

Metadata for DLLs should actually go in the LIB import library on Windows #29511

Open

bjorn3 mentioned this pull request Aug 9, 2022

Report codegen timings for binary crates rust-lang/cargo#10960

Closed

petrochenkov mentioned this pull request May 10, 2023

Load only the crate header for locator::crate_matches #111329

Merged

bjorn3 mentioned this pull request Feb 9, 2024

Introduce -Zsplit-metadata option #120855

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce -Zsplit-metadata option #93945

Introduce -Zsplit-metadata option #93945

bjorn3 commented Feb 12, 2022 •

edited

Loading

rust-highfive commented Feb 12, 2022

bjorn3 commented Feb 12, 2022

rust-timer commented Feb 12, 2022

bors commented Feb 12, 2022

This comment was marked as outdated.

This comment was marked as outdated.

bors commented Feb 12, 2022

fbstj Feb 12, 2022

bjorn3 Feb 12, 2022

bjorn3 commented Feb 12, 2022

rust-timer commented Feb 12, 2022

bors commented Feb 12, 2022

This comment has been minimized.

jyn514 commented Feb 13, 2022 •

edited

Loading

bjorn3 commented Feb 13, 2022

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Feb 13, 2022

bjorn3 commented Feb 13, 2022

bors commented Feb 18, 2022

Dylan-DPC commented Apr 5, 2022

bjorn3 commented Feb 9, 2024

Introduce -Zsplit-metadata option #93945

Introduce -Zsplit-metadata option #93945

Conversation

bjorn3 commented Feb 12, 2022 • edited Loading

rust-highfive commented Feb 12, 2022

bjorn3 commented Feb 12, 2022

rust-timer commented Feb 12, 2022

bors commented Feb 12, 2022

This comment was marked as outdated.

This comment was marked as outdated.

bors commented Feb 12, 2022

fbstj Feb 12, 2022

Choose a reason for hiding this comment

bjorn3 Feb 12, 2022

Choose a reason for hiding this comment

bjorn3 commented Feb 12, 2022

rust-timer commented Feb 12, 2022

bors commented Feb 12, 2022

This comment has been minimized.

jyn514 commented Feb 13, 2022 • edited Loading

bjorn3 commented Feb 13, 2022

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Feb 13, 2022

bjorn3 commented Feb 13, 2022

bors commented Feb 18, 2022

Dylan-DPC commented Apr 5, 2022

bjorn3 commented Feb 9, 2024

bjorn3 commented Feb 12, 2022 •

edited

Loading

jyn514 commented Feb 13, 2022 •

edited

Loading