trans: Internalize symbols without relying on LLVM #43183

michaelwoerister · 2017-07-12T15:44:16Z

This PR makes the compiler use the information gather by the trans collector in order to determine which symbols/trans-items can be made internal. This has the advantages:

of being LLVM independent,
of also working in incremental mode, and
of allowing to not keep all LLVM modules in memory at the same time.

This is in preparation for fixing issue #39280.

cc @rust-lang/compiler

…ges.

rust-highfive · 2017-07-12T15:44:20Z

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

Mark-Simulacrum · 2017-07-12T15:54:47Z

src/librustc_trans/back/symbol_export.rs

@@ -28,59 +27,87 @@ pub enum SymbolExportLevel {
 }

 /// The set of symbols exported from each crate in the crate graph.
+#[derive(Clone, Debug)]


Does this need to be Clone? It looks like it's a fairly large structure that ideally we wouldn't clone. It seems like it's in an Arc most of the time anyway so this feels unnecessary.

That is indeed not necessary anymore. Will remove.

Mark-Simulacrum · 2017-07-12T15:58:44Z

src/librustc_trans/collector.rs

    {
        assert!(!self.index.contains_key(&source));

        let start_index = self.targets.len();
-        self.targets.extend(targets);
+        let (targets_size_hint, targets_size_hint_max) = targets.size_hint();
+        debug_assert_eq!(targets_size_hint_max, Some(targets_size_hint));


It looks like what we really want here is an ExactSizeIterator?

I'll look into it.

Mark-Simulacrum · 2017-07-12T16:03:48Z

src/librustc_trans/partitioning.rs

+            cgu_name: cgu.name.clone()
+        };
+
+        'item:


This appears to no longer be necessary.

I don't understand. It's used in the nested continue below.

I'm probably just missing something, but I don't see a nested loop above it?

Oh right, the only place those nested loops still exist is in my head :) Sorry for the confusion. I'll remove the label.

Mark-Simulacrum · 2017-07-12T16:04:23Z

src/librustc_trans/partitioning.rs

+                                // Some accessors might not have been
+                                // instantiated. We can safely ignore those.
+                                trans_item_placements.get(accessor)
+                                                     .cloned()


Why do we need to clone the option? It looks like that shouldn't be necessary -- we only use it for PartialEq?

cloned() (as opposed to clone()) turns the Option<&T> into an Option<T>, which is what we need for filter_map.

I see what you meant now. Yes, cloning seems unnecessary.

eddyb · 2017-07-12T19:33:37Z

src/librustc_trans/collector.rs

-            hir::ItemFn(.., ref generics, _) => {
-                if !generics.is_type_parameterized() {
+            hir::ItemFn(_, _, constness, _, ref generics, _) => {
+                let is_const = match constness {


I recommend tcx.is_const_fn(def_id) - similarly for hir::Generics, ty::Generics should be preferred, although that one is preexisting.

Does is_const_fn do something differently than the code below? I'm not against changing, but the constness and generics from the HIR item are right there without the need for a hashtable lookup.

eddyb · 2017-07-12T19:34:04Z

src/librustc_trans/collector.rs

@@ -889,7 +956,14 @@ impl<'b, 'a, 'v> ItemLikeVisitor<'v> for RootCollector<'b, 'a, 'v> {
                    }
                };

-                if !generics.is_type_parameterized() && !is_impl_generic {
+                let is_const = match constness {


eddyb · 2017-07-12T19:35:01Z

r=me modulo nits and the build failure

…n LLVM.

michaelwoerister · 2017-07-13T15:59:09Z

@bors r=eddyb

All nits addressed, I think.

bors · 2017-07-13T15:59:09Z

📌 Commit 678d377 has been approved by eddyb

bors · 2017-07-15T01:59:07Z

⌛ Testing commit 678d377 with merge 685ccdf38707dc948ec9bbde82caa8c9e0ac5cdc...

bors · 2017-07-15T02:22:08Z

💔 Test failed - status-travis

Mark-Simulacrum · 2017-07-15T02:37:40Z

00:16:51] rustc: /checkout/src/llvm/include/llvm/IR/GlobalValue.h:227: void llvm::GlobalValue::setVisibility(llvm::GlobalValue::VisibilityTypes): Assertion `(!hasLocalLinkage() || V == DefaultVisibility) && "local linkage requires default visibility"' failed.
[00:16:51] �(B�[m�[31m�[1merror:�(B�[m Could not compile `core`.
[00:16:51] �(B�[m
[00:16:51] Caused by:
[00:16:51] �(B�[m�(B�[m  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name core /checkout/src/libcore/lib.rs --color always --error-format json --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=a6117b7e5e0db06a -C extra-filename=-a6117b7e5e0db06a --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/release/deps -Zincremental=/tmp/rust-incr-cache` (signal: 6, SIGABRT: process abort signal)
[00:16:51] --- stderr
[00:16:51] rustc: /checkout/src/llvm/include/llvm/IR/GlobalValue.h:227: void llvm::GlobalValue::setVisibility(llvm::GlobalValue::VisibilityTypes): Assertion `(!hasLocalLinkage() || V == DefaultVisibility) && "local linkage requires default visibility"' failed.
[00:16:51] 
[00:16:51] �(B�[mthread 'main' panicked at 'command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "-j" "4" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:16:51] expected success, got: exit code: 101', /checkout/src/bootstrap/compile.rs:586
[00:16:51] note: Run with `RUST_BACKTRACE=1` for a backtrace.

michaelwoerister · 2017-07-17T14:01:50Z

@bors r=eddyb

That assertion should be fixed, hopefully.

bors · 2017-07-17T14:01:51Z

📌 Commit 226bc92 has been approved by eddyb

bors · 2017-07-18T03:38:22Z

⌛ Testing commit 226bc92 with merge 82e2f9ddd4d0d32bb1ab96928e099c49975d932c...

bors · 2017-07-18T03:42:13Z

💔 Test failed - status-travis

est31 · 2017-07-18T06:51:01Z

@bors retry

bors · 2017-07-19T08:37:20Z

⌛ Testing commit 226bc92 with merge b45d8f0ebcf760610826ce2a43f5f3d437916086...

bors · 2017-07-19T08:53:41Z

💔 Test failed - status-appveyor

kennytm · 2017-07-19T09:21:22Z

Spurious. Could this be related to https://appveyor.statuspage.io/incidents/m2vdvw39kdk8?

  [35] SSL connect error (schannel: next InitializeSecurityContext failed: Unknown error (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline.)

michaelwoerister · 2017-07-19T11:55:21Z

@bors retry

(No harm in retrying while the queue is empty, I guess)

bors · 2017-07-19T11:55:30Z

⌛ Testing commit 226bc92 with merge 49765af5c6bcd1182b8d279b404d79fdd8a81c18...

bors · 2017-07-19T11:59:11Z

💔 Test failed - status-appveyor

nagbot-rs · 2017-07-19T14:12:09Z

@bors: retry

…

On Wed, Jul 19, 2017 at 6:59 AM, bors ***@***.***> wrote: 💔 Test failed - status-appveyor <https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3983> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#43183 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AaPN0DaQBIlhKb8psY9iNfX3uCfRj9k0ks5sPe-hgaJpZM4OV07r> . -- You received this message because you are subscribed to the Google Groups "rust-ops" group. To unsubscribe from this group and stop receiving emails from it, send an email to ***@***.*** To post to this group, send email to ***@***.*** To view this discussion on the web visit https://groups.google.com/d/ msgid/rust-ops/rust-lang/rust/pull/43183/c316363043%40git.luolix.top <https://groups.google.com/d/msgid/rust-ops/rust-lang/rust/pull/43183/c316363043%40git.luolix.top?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout.

bors · 2017-07-19T14:12:12Z

@nagbot-rs: 🔑 Insufficient privileges: and not in try users

alexcrichton · 2017-07-19T14:14:29Z

@bors: retry

Mark-Simulacrum · 2017-07-19T14:18:55Z

@bors retry -- appveyor revocation certs, appears to be fixed once #43333 lands.

bors · 2017-07-19T23:40:09Z

⌛ Testing commit 226bc92 with merge 528330520a5ad6953e4a0f25f79d4db1ad019e7f...

bors · 2017-07-20T00:11:33Z

💔 Test failed - status-appveyor

kennytm · 2017-07-20T00:22:11Z

x86_64-pc-windows-gnu failed to link stage1-std ("undefined reference to std::thread::local::os::destroy_value::hXXXXXX"). Seems legit.

error: linking with `gcc` failed: exit code: 1
  |
  = note: "gcc" "-Wl,--enable-long-section-names" "-fno-use-linker-plugin" "-Wl,--nxcompat" "-nostdlib" "-m64" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\dllcrt2.o" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\deps\\std-ac667c8bbd86e364.0.o" "-o" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\deps\\std-ac667c8bbd86e364.dll" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\deps\\std-ac667c8bbd86e364.crate.metadata.o" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\deps\\std-ac667c8bbd86e364.crate.allocator.o" "-nodefaultlibs" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\deps" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\release\\deps" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\native\\libbacktrace\\.libs" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\native\\jemalloc\\lib" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1-std\\x86_64-pc-windows-gnu\\release\\build\\compiler_builtins-3f52fffdb7374465\\out" "-L" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "-l" "backtrace" "-Wl,--no-whole-archive" "-Wl,-Bdynamic" "-l" "advapi32" "-l" "ws2_32" "-l" "userenv" "-l" "shell32" "-Wl,-Bstatic" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\libpanic_unwind-d7ba3ed13adbba78.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\libunwind-4a5bfaad61fe8668.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\liblibc-f12a6f077c568615.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\librand-fc4f55c30253478d.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\liballoc_system-fa945e0e44cf2c85.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\liballoc-7e25d9aea6733470.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\libstd_unicode-b2da6d85e26577df.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\libcore-133c3145709a073d.rlib" "-Wl,--no-whole-archive" "C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\rustc.HACeg3fOyh3C\\libcompiler_builtins-b49c9299458b621e.rlib" "-l" "gcc_eh" "-l" "pthread" "-shared" "-Wl,-Bdynamic" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-luser32" "-lkernel32" "C:\\projects\\rust\\build\\x86_64-pc-windows-gnu\\stage1\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsend.o"
  = note: C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0xd8): undefined reference to `std::thread::local::os::destroy_value::ha44314076d36f203'
          C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0xe8): undefined reference to `std::thread::local::os::destroy_value::hc6b9798d5a01e96f'
          C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0xf8): undefined reference to `std::thread::local::os::destroy_value::hfc4011e45796e013'
          C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0x108): undefined reference to `std::thread::local::os::destroy_value::hc6b9798d5a01e96f'
          C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0x118): undefined reference to `std::thread::local::os::destroy_value::h96e432fe261f7344'
          C:\projects\rust\build\x86_64-pc-windows-gnu\stage1-std\x86_64-pc-windows-gnu\release\deps\std-ac667c8bbd86e364.0.o:(.data+0x128): undefined reference to `std::thread::local::os::destroy_value::h10f11fb8acf74f1c'
          collect2.exe: error: ld returned 1 exit status
          
error: aborting due to previous error
error: Could not compile `std`.

michaelwoerister · 2017-07-20T09:21:30Z

Thanks, @kennytm!

aidanhs · 2017-07-20T14:31:09Z

[00:40:46] ---- [compile-fail] compile-fail/E0534.rs stdout ----
[00:40:46] 	
[00:40:46] error: compile-fail test compiled successfully!
[00:40:46] status: exit code: 0
[00:40:46] command: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc /checkout/src/test/compile-fail/E0534.rs -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail --target=x86_64-unknown-linux-gnu --error-format json -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/E0534.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux -C prefer-dynamic -o /checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/E0534.stage2-x86_64-unknown-linux-gnu -Crpath -O -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers
[00:40:46] stdout:

(and a few more)

michaelwoerister · 2017-07-20T20:14:41Z

@bors r=eddyb

@eddyb I pushed one more commit to fix the linker issue on Windows. It should be rather uncontroversial but feel free to take another look.

bors · 2017-07-20T20:14:41Z

📌 Commit f6e5416 has been approved by eddyb

bors · 2017-07-21T02:24:25Z

⌛ Testing commit f6e5416 with merge e3c8433...

…lvm, r=eddyb trans: Internalize symbols without relying on LLVM This PR makes the compiler use the information gather by the trans collector in order to determine which symbols/trans-items can be made internal. This has the advantages: + of being LLVM independent, + of also working in incremental mode, and + of allowing to not keep all LLVM modules in memory at the same time. This is in preparation for fixing issue #39280. cc @rust-lang/compiler

bors · 2017-07-21T04:48:56Z

☀️ Test successful - status-appveyor, status-travis
Approved by: eddyb
Pushing e3c8433 to master...

trans: Collect all accesses between trans-items, not just inlining ed…

1ebfab5

…ges.

rust-highfive assigned eddyb Jul 12, 2017

Mark-Simulacrum reviewed Jul 12, 2017

View reviewed changes

eddyb reviewed Jul 12, 2017

View reviewed changes

aidanhs added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 13, 2017

michaelwoerister force-pushed the internalize-symbols-without-llvm branch from ad74e77 to 2b9ab65 Compare July 13, 2017 09:49

trans: Internalize symbols at the trans-item level, without relying o…

2f07eb3

…n LLVM.

michaelwoerister force-pushed the internalize-symbols-without-llvm branch from 2b9ab65 to 2f07eb3 Compare July 13, 2017 11:28

Adapt cgu-partitioning tests to pre-trans symbol internalization.

c93e62b

michaelwoerister force-pushed the internalize-symbols-without-llvm branch from efdc922 to c93e62b Compare July 13, 2017 11:29

Address some nits in trans-collector and partitioner.

678d377

michaelwoerister mentioned this pull request Jul 14, 2017

incr.comp.: Clarify compatibility of red/green evaluation and recovering from cycle errors #42633

Closed

partitioning: Fix visibility of internalized symbols.

226bc92

michaelwoerister force-pushed the internalize-symbols-without-llvm branch from 42c4564 to e8ff75b Compare July 20, 2017 14:35

trans: Make the collector search const fn invocations.

f6e5416

michaelwoerister force-pushed the internalize-symbols-without-llvm branch from e8ff75b to f6e5416 Compare July 20, 2017 20:11

bors merged commit f6e5416 into rust-lang:master Jul 21, 2017

bors mentioned this pull request Jul 21, 2017

librustc_*: Use pub(crate) instead of pub and remove dead code #43192

Closed

2 tasks

michaelwoerister mentioned this pull request Jul 24, 2017

change internalize_symbols to work per-CGU #41707

Closed

nikomatsakis mentioned this pull request Jul 29, 2017

change how MIR inlining handles cycles #43542

Closed

alexcrichton mentioned this pull request Oct 31, 2017

cdylib artifact size boost #45613

Closed

trans: Internalize symbols without relying on LLVM #43183

trans: Internalize symbols without relying on LLVM #43183

Conversation

michaelwoerister commented Jul 12, 2017

rust-highfive commented Jul 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddyb commented Jul 12, 2017 • edited Loading

michaelwoerister commented Jul 13, 2017

bors commented Jul 13, 2017

bors commented Jul 15, 2017

bors commented Jul 15, 2017

Mark-Simulacrum commented Jul 15, 2017

michaelwoerister commented Jul 17, 2017

bors commented Jul 17, 2017

bors commented Jul 18, 2017

bors commented Jul 18, 2017

est31 commented Jul 18, 2017

bors commented Jul 19, 2017

bors commented Jul 19, 2017

kennytm commented Jul 19, 2017

michaelwoerister commented Jul 19, 2017

bors commented Jul 19, 2017

bors commented Jul 19, 2017

nagbot-rs commented Jul 19, 2017 via email

bors commented Jul 19, 2017

alexcrichton commented Jul 19, 2017

Mark-Simulacrum commented Jul 19, 2017

bors commented Jul 19, 2017

bors commented Jul 20, 2017

kennytm commented Jul 20, 2017

michaelwoerister commented Jul 20, 2017

aidanhs commented Jul 20, 2017

michaelwoerister commented Jul 20, 2017

bors commented Jul 20, 2017

bors commented Jul 21, 2017

bors commented Jul 21, 2017

eddyb commented Jul 12, 2017 •

edited

Loading