Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linker: Never use whole-archive linking unless explicitly requested #85144

Closed
wants to merge 1 commit into from

Conversation

petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented May 10, 2021

Reminder: "whole-archive" linking for a static library means including all the object files from the library into the output binary instead only those that are needed ("needed" is determined by the current list of unresolved symbols).

We have two automatic uses of whole-archive right now.

  • Static library (kind=static, but not kind=static-nobundle) linked into the final executable or dynamic library directly (but not through a rlib dependency).
    The use of whole-archive in this case is pretty random and inconsistent, and also entirely unnecessary in regular cases.
    Moreover, when whole-archive was properly implemented for MSVC targets in linker: MSVC supports linking static libraries as a whole archive #72785 it caused regressions (see links to other issues from that PR).
  • Rust static library (rlib), possibly with bundled native libraries, linked into a Rust dylib.
    The comment tells that it's done because "dylib can be reused as an intermediate artifact", perhaps it was necessary in the past, but now it shouldn't be necessary because everything usable from a dylib should be already kept alive by export lists (fn export_symbols).

I wanted to land this sooner in the release cycle in case I'm missing something and there are regressions.
However, it may also make sense to land this together with stabilizing the +whole-archive linking modifier (#81490) to make sure there's a fix available on stable in cases where whole-archive was actually required in one these specific situations (which should be a rare case).

@rust-highfive
Copy link
Collaborator

r? @jackh726

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 10, 2021
@rust-log-analyzer

This comment has been minimized.

@petrochenkov
Copy link
Contributor Author

petrochenkov commented May 10, 2021

Interesting, I haven't observed any errors on a full x.py test --stage 2 run on windows-gnu.
I'll investigate (the CI error is about the second case).

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 10, 2021
@JohnCSimon JohnCSimon added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 31, 2021
@bors

This comment has been minimized.

@mati865
Copy link
Contributor

mati865 commented Jun 8, 2021

FWIW LLD shows even more missing symbols:

  = note: ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..alloc..layout..LayoutError$u20$as$u20$core..fmt..Debug$GT$::fmt::hd9bfd5c6c01ee268 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::white_space::lookup::hea0caa0bb45317fa [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to hashbrown::raw::sse2::Group::static_empty::ha2a8a717acd84536 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to hashbrown::raw::Fallibility::capacity_overflow::h3722fba48769c39e [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to hashbrown::raw::Fallibility::alloc_err::hf0376dc0c7a9cf5d [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to alloc::str::_$LT$impl$u20$str$GT$::to_lowercase::h77b000cf3481e807 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to alloc::str::_$LT$impl$u20$str$GT$::repeat::ha9d4f7e312aee989 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..sync..atomic..AtomicBool$u20$as$u20$core..fmt..Debug$GT$::fmt::h73c2137ead87af5f [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..sync..atomic..AtomicUsize$u20$as$u20$core..fmt..Debug$GT$::fmt::hbfe36297b50ef593 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$alloc..boxed..Box$LT$str$GT$$u20$as$u20$core..clone..Clone$GT$::clone::h9ab85fd63cae8683 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::lowercase::lookup::h1d31df8070d8b713 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::uppercase::lookup::h6440b9dcc5bc89aa [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::str::traits::str_index_overflow_fail::h303ad7ee33da751e [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::num::dec2flt::parse::parse_decimal::h76512508e280a14c [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::num::dec2flt::algorithm::make_ratio::h6fbfec93992c15b9 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::num::dec2flt::algorithm::power_of_ten::h1c5b7d25c71c1dde [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..core_arch..x86..__m128i$u20$as$u20$core..fmt..Debug$GT$::fmt::h9accebf1d6482d64 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..core_arch..x86..__m256i$u20$as$u20$core..fmt..Debug$GT$::fmt::ha5768a86ccf99907 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to std_detect::detect::cache::detect_and_initialize::h6c7c0a5590b6608d [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to std_detect::detect::cache::CACHE::hdad2f91b5c8ae2e5 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::alphabetic::lookup::h370e707f200d1468 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::n::lookup::h649376680efd1baa [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to alloc::str::_$LT$impl$u20$str$GT$::to_uppercase::h6ea3dda8edbee42d [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::conversions::to_lower::h1acd82e0168ef2fd [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to core::unicode::unicode_data::conversions::to_upper::h7834bd9157a9a53c [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..cmp..Ordering$u20$as$u20$core..fmt..Debug$GT$::fmt::hd4d8a195a7a93714 [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..num..nonzero..NonZeroUsize$u20$as$u20$core..str..traits..FromStr$GT$::from_str::h954512cb6147da7e [--no-allow-shlib-undefined]
          ld: error: /home/mateusz/Projects/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-6f3eccdfec750c19.so: undefined reference to _$LT$core..alloc..layout..Layout$u20$as$u20$core..fmt..Debug$GT$::fmt::h9e06c0156b0366ba [--no-allow-shlib-undefined]
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ ld --version
LLD 11.0.0 (compatible with GNU linkers)

@jackh726
Copy link
Member

@petrochenkov I'm not a good reviewer for this. Any idea who might be better?

@petrochenkov
Copy link
Contributor Author

r? @bjorn3 I guess

I obviously made some wrong assumptions about dylibs and I need to figure out what exactly I got wrong, but I wasn't able to get to this PR since May for prioritization reasons, so this PR is pretty far from needing a review right now.

@rust-highfive rust-highfive assigned bjorn3 and unassigned jackh726 Jun 24, 2021
@JohnCSimon JohnCSimon added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 12, 2021
@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-llvm-10 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
   Compiling rustc_privacy v0.0.0 (/checkout/compiler/rustc_privacy)
   Compiling rustc_interface v0.0.0 (/checkout/compiler/rustc_interface)
   Compiling rustc_driver v0.0.0 (/checkout/compiler/rustc_driver)
   Compiling rustc-main v0.0.0 (/checkout/compiler/rustc)
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-deddb2dd1fd284f9.rustc_main.04bc62e8-cgu.0.rcgu.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-deddb2dd1fd284f9.rustc_main.04bc62e8-cgu.1.rcgu.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-deddb2dd1fd284f9.rustc_main.04bc62e8-cgu.2.rcgu.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-deddb2dd1fd284f9.rustc_main.04bc62e8-cgu.3.rcgu.o" "-Wl,--as-needed" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/psm-fc72508f457ff2af/out" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-1b17de65bfd28043/out" "-L" "/usr/lib/llvm-10/lib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-lrustc_driver-94967fd0a7379a2d" "-Wl,--start-group" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-lstd-acc102086c90e281" "-Wl,--end-group" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b8b1093eb15f57a0.rlib" "-Wl,-Bdynamic" "-lLLVM-10" "-lstdc++" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-deddb2dd1fd284f9" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-Wl,-rpath,$ORIGIN/../lib"
  = note: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-94967fd0a7379a2d.so: undefined reference to `hashbrown::raw::Fallibility::alloc_err'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-94967fd0a7379a2d.so: undefined reference to `hashbrown::raw::Fallibility::capacity_overflow'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-94967fd0a7379a2d.so: undefined reference to `hashbrown::raw::sse2::Group::static_empty'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-94967fd0a7379a2d.so: undefined reference to `std_detect::detect::cache::detect_and_initialize'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-94967fd0a7379a2d.so: undefined reference to `std_detect::detect::cache::CACHE'
          collect2: error: ld returned 1 exit status
          
  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
error: aborting due to previous error

error: could not compile `rustc-main`

@JohnCSimon JohnCSimon added S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 31, 2021
@JohnCSimon JohnCSimon removed the S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. label Aug 15, 2021
@bors
Copy link
Contributor

bors commented Aug 27, 2021

☔ The latest upstream changes (presumably #88227) made this pull request unmergeable. Please resolve the merge conflicts.

@petrochenkov
Copy link
Contributor Author

I'm going to close this for now.

This PR contains two orthogonal changes:

  • The native static library change.
    Once Fix handling of +whole-archive native link modifier. #88161 lands I plan to audit the implementation of native linking modifiers once more and submit a PR stabilizing the modifier syntax in general and the whole-archive modifier specifically.
    I'll include native static library change in that PR, so it goes with a stable way to opt out in case of breakage.
  • The Rust dylib change.
    Not urgent, largely internal change. I'll leave it my queue and maybe investigate it some time in the next few years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants