Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow writing metadata without llvm #44085

Merged
merged 17 commits into from
Sep 25, 2017
Merged

Conversation

bjorn3
Copy link
Member

@bjorn3 bjorn3 commented Aug 25, 2017

Todo:

  • Rebase
  • Fix eventual errors
  • Find some crate to write elf files (will do it later)

Cc #43842

@rust-highfive
Copy link
Collaborator

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

@shepmaster shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 25, 2017
Copy link
Member

@Mark-Simulacrum Mark-Simulacrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to see this as a possibility! Left a few minor comments about bootstrap.

builder.cargo(compiler, Mode::Libstd, target, "build")
}else{
builder.cargo(compiler, Mode::Libstd, target, "check")
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems wrong/unrelated? If not, reasoning would be appreciated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should make this no llvm only. When stage 1 rustc doesnt have llvm support it cant build regular rlibs, but only metadata only rlibs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. I suppose that makes sense.

@@ -161,6 +165,7 @@ pub fn std_cargo(build: &Build,
// missing
// We also only build the runtimes when --enable-sanitizers (or its
// config.toml equivalent) is used
//cargo.env("RUST_FLAGS", "-Zno-trans");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to go before we merge.

::rustc_trans_utils::find_exported_symbols(tcx, &analysis.reachable);
let (metadata, _hashes) =
cstore.encode_metadata(tcx, &link_meta, &exported_symbols);
let mut builder = Builder::new(File::create("abc.rlib").unwrap());
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to make this write to the file asked to, instead of abc.rlib.

@bjorn3
Copy link
Member Author

bjorn3 commented Aug 26, 2017

$ ./x.py build --stage 1
[...]
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling rustc_back v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_back)
   Compiling syntax v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/libsyntax)
   Compiling log v0.3.8
[...]
   Compiling rustc_save_analysis v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_save_analysis)
warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries

warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries

   Compiling rustc-main v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/rustc)
error: LLVM is not supported by this rustc

error: Could not compile `rustc-main`.

Caused by:
  process didn't exit successfully: `/Users/bjorn/Documents/rust_fork/no_llvm_build/build/bootstrap/debug/rustc --crate-name rustc /Users/bjorn/Documents/rust_fork/src/rustc/rustc.rs --error-format json --crate-type bin --emit=dep-info,link -C opt-level=2 --cfg feature="jemalloc" --cfg feature="rustc_back" -C metadata=9c436ad388a4a0ca -C extra-filename=-9c436ad388a4a0ca --out-dir /Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/release/deps --extern rustc_back=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_back-853a8b36deced665.dylib --extern rustc_driver=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_driver-cc8fef39b95e564b.dylib -L native=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/build/miniz-sys-e7bda6e0ddcea6f1/out` (exit code: 101)
thread 'main' panicked at 'command did not execute successfully: "/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "-j" "8" "--target" "x86_64-apple-darwin" "--release" "--features" " jemalloc" "--manifest-path" "/Users/bjorn/Documents/rust_fork/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', /Users/bjorn/Documents/rust_fork/src/bootstrap/compile.rs:883:8
[...]

@bjorn3
Copy link
Member Author

bjorn3 commented Aug 28, 2017

What is this error message of the "Fix tidy errors" commit:

[00:16:47] error: linking with `cc` failed: exit code: 1
[00:16:47]   |
[00:16:47]   = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.0.o" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.crate.allocator.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ea2e08566ed2ca9b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-117698016bc6c938.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-aa6c11e69908f4f9.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-d94cd37ff18e6e1c.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-388107ef329739bd.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-c45b3574292a7ea1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-c791e7351d8a272a.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-54feb014ef663465.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-81b84a407143b5e1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ba9896bad78dc777.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"
[00:16:47]   = note: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib: error adding symbols: Archive has no index; run ranlib to add one
[00:16:47]           collect2: error: ld returned 1 exit status
[00:16:47]           
[00:16:47] 
[00:16:47] error: aborting due to previous error
[00:16:47] 
[00:16:47] error: Could not compile `syntax`.

@arielb1
Copy link
Contributor

arielb1 commented Aug 29, 2017

[00:02:07] ======================================================================

[00:02:07] FAIL: program_config (bootstrap.RustBuild)

[00:02:07] Doctest: bootstrap.RustBuild.program_config

[00:02:07] ----------------------------------------------------------------------

[00:02:07] Traceback (most recent call last):

[00:02:07]   File "/usr/lib/python2.7/doctest.py", line 2226, in runTest

[00:02:07]     raise self.failureException(self.format_failure(new.getvalue()))

[00:02:07] AssertionError: Failed doctest test for bootstrap.RustBuild.program_config

[00:02:07]   File "/checkout/src/bootstrap/bootstrap.py", line 519, in program_config

[00:02:07] 

[00:02:07] ----------------------------------------------------------------------

[00:02:07] File "/checkout/src/bootstrap/bootstrap.py", line 527, in bootstrap.RustBuild.program_config

[00:02:07] Failed example:

[00:02:07]     cargo_path.rstrip(".exe") == os.path.join("/tmp/rust",

[00:02:07]     "bin", "cargo")

[00:02:07] Expected:

[00:02:07]     True

[00:02:07] Got:

[00:02:07]     False

@@ -159,6 +159,10 @@ fn main() {
cmd.arg("-C").arg("panic=abort");
}

if cfg!(not(feature="llvm")) && stage != "0" {
Copy link
Contributor

@arielb1 arielb1 Aug 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't you mean !self.build.config.llvm_enabled here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesnt have self.build.config but i dont think its compiled with the feature, so i think i will have to use an env var.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, an env var would be fine. I think there's already precedent for that (i.e. grep for a env var with LLVM in it) so there may not be a need to add a new one.

@arielb1
Copy link
Contributor

arielb1 commented Sep 5, 2017

[00:35:42] error[E0308]: mismatched types
[00:35:42]    --> /checkout/src/librustdoc/test.rs:260:31
[00:35:42]     |
[00:35:42] 260 |         driver::compile_input(&sess, &cstore, &input, &out, &None, None, &control)
[00:35:42]     |                               ^^^^^ types differ in mutability
[00:35:42]     |
[00:35:42]     = note: expected type `&mut rustc::session::Session`
[00:35:42]                found type `&rustc::session::Session`

@bors
Copy link
Contributor

bors commented Sep 8, 2017

☔ The latest upstream changes (presumably #44142) made this pull request unmergeable. Please resolve the merge conflicts.

@bjorn3 bjorn3 force-pushed the no_llvm_write_metadata branch 4 times, most recently from bec1f52 to ea497a2 Compare September 16, 2017 16:51
@bors
Copy link
Contributor

bors commented Sep 16, 2017

☔ The latest upstream changes (presumably #43017) made this pull request unmergeable. Please resolve the merge conflicts.

@bjorn3 bjorn3 force-pushed the no_llvm_write_metadata branch 3 times, most recently from b88fcf7 to 243d64f Compare September 17, 2017 08:26
@bjorn3 bjorn3 changed the title [WIP] Allow writing metadata without llvm Allow writing metadata without llvm Sep 17, 2017
@bjorn3
Copy link
Member Author

bjorn3 commented Sep 17, 2017

I added a generic TransCrate trait, which is to be implemented by compiler backends.

@@ -524,14 +524,20 @@ impl<'a> Builder<'a> {
// For other crates, however, we know that we've already got a standard
// library up and running, so we can use the normal compiler to compile
// build scripts in that situation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also modify the comment?

@@ -243,7 +248,48 @@ pub fn compile_input(sess: &Session,
tcx.print_debug_stats();
}

let trans = phase_4_translate_to_llvm(tcx, analysis, incremental_hashes_map,
#[cfg(not(feature="llvm"))]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving this outside of the main driver function?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should remove the #[cfg]

@@ -243,7 +248,48 @@ pub fn compile_input(sess: &Session,
tcx.print_debug_stats();
}

let trans = phase_4_translate_to_llvm(tcx, analysis, incremental_hashes_map,
#[cfg(not(feature="llvm"))]
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also not sure about the idea of creating dylibs that are actually rlibs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without llvm we cant create real dylibs, but many rustc crates are compiled to dylibs, so by emitting rlibs instead we can compile all rustc libs.


#[cfg(not(feature="llvm"))]
mod no_llvm_metadata_loader {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to an out-of-line module?

#[cfg(not(feature="llvm"))]
mod rustc_trans {
use syntax_pos::symbol::Symbol;
mod trans_metadata_only {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to an out-of-line module?

fn get_dylib_metadata(&self, _target: &Target, _filename: &Path) -> Result<ErasedBoxRef<[u8]>, String> {
bug!("DummyMetadataLoader::get_dylib_metadata");
}
}
Copy link
Contributor

@arielb1 arielb1 Sep 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also move the "metadata-only" TransCrate implementation here to rustc_trans_utils?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do it

@@ -0,0 +1,17 @@
[package]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this exist as a separate crate from rustc_trans_utils? They are both depended upon by exactly rustc_trans and rustc_driver, and they semantically perform a very similar task.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about utils to be used by backends only, while traits contains the glue between backends and rustc_driver. I can merge them though.


use self::back::write::OngoingCrateTranslation;
#[cfg(not(feature="llvm"))]
mod rustc_trans {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move all of these items to the TransCrate trait?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because most of them are llvm specific, eg they would still use llvm even if trans is non llvm. (enable_llvm_debug, print_passes)

@bjorn3
Copy link
Member Author

bjorn3 commented Sep 23, 2017

@arielb1 rebased and fixed most nits

@arielb1
Copy link
Contributor

arielb1 commented Sep 24, 2017

@bors r+

@bors
Copy link
Contributor

bors commented Sep 24, 2017

📌 Commit 3c32c6a has been approved by arielb1

@bors
Copy link
Contributor

bors commented Sep 25, 2017

⌛ Testing commit 3c32c6a with merge 0e7f2f843dce90acc9e8ec1f29665ba69ac1f844...

@bors
Copy link
Contributor

bors commented Sep 25, 2017

💔 Test failed - status-travis

@bjorn3
Copy link
Member Author

bjorn3 commented Sep 25, 2017

Error should be fixed now.

@arielb1
Copy link
Contributor

arielb1 commented Sep 25, 2017

@bors r+

@bors
Copy link
Contributor

bors commented Sep 25, 2017

📌 Commit 843cd5b has been approved by arielb1

@bors
Copy link
Contributor

bors commented Sep 25, 2017

⌛ Testing commit 843cd5b with merge 3df1f7b...

bors added a commit that referenced this pull request Sep 25, 2017
Allow writing metadata without llvm

# Todo:

* [x] Rebase
* [x] Fix eventual errors
* [x] <strike>Find some crate to write elf files</strike> (will do it later)

Cc #43842
@bors
Copy link
Contributor

bors commented Sep 25, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 3df1f7b to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants