Add implib to the stamp files #6875

chouquette · 2019-04-25T13:20:29Z

Hi,

This MR aims at including the import libraries in the stamp file, in order for them to be copied around when uplifting stages during rust-lang's compilation.

This was already done when targetting msvc, but it needs to be done for
all targets to allow rustc to link with shared libraries when using llvm.
-l is only supported by gcc, while llvm needs an import library
to link indirectly with a dll

This MR is related to rust-lang/rust#60260

I'm unsure what the best way is, when it comes to cross dependent MRs, so I'll gladly change things according to your reviews!

Thanks,

rust-highfive · 2019-04-25T13:20:34Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

ehuss · 2019-04-25T20:19:47Z

I'm not familiar with gnu linking on Windows. Whenever I try to build a dylib with x86_64-pc-windows-gnu, I get linker errors about onexitbegin/onexitend. Is there some way I can test this?

alexcrichton · 2019-04-26T14:43:34Z

I think we'll probably want to hold off merging this until rust-lang/rust#60260 reaches a conclusion as well, but we can of course test it in the meantime :)

I'm gonna try to comment more on rust-lang/rust#60260 today

chouquette · 2019-04-26T15:01:22Z

Do you have the exact error and/or linker line? I'd bet on a missing crtbegin.o/crtend.o

You might also have to ensure the linker is actually the mingw-w64 one, I've had cases where the linker was a native gcc, which didn't work so well (but I'll confess I haven't look into this at all)

ehuss · 2019-04-29T04:03:29Z

I'm using the regular mingw from mingw.org. I'm a little confused by mingw-w64, is that yet another clone? I find the whole mingw/msys stuff somewhat confusing. Searching for "rust" "onexitbegin" on google brings up a handful of people having the same problem, but it looks like people resorted to manually copying object files around? It seems pretty broken.

which gcc is /mingw64/bin/gcc

Here's the link output:

error: linking with gcc failed: exit code: 1
|
= note: "gcc" "-Wl,--enable-long-section-names" "-fno-use-linker-plugin" "-Wl,--nxcompat" "-nostdlib" "-m64" "D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib\dllcrt2.o" "D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib\rsbegin.o" "-L" "D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib" "D:\proj\rust\cargo\scratch\x\target\x86_64-pc-windows-gnu\debug\deps\x.3y8q2vk936zgttly.rcgu.o" "-o" "D:\proj\rust\cargo\scratch\x\target\x86_64-pc-windows-gnu\debug\deps\x.dll" "D:\proj\rust\cargo\scratch\x\target\x86_64-pc-windows-gnu\debug\deps\x.3xikuc9r6krp6r04.rcgu.o" "D:\proj\rust\cargo\scratch\x\target\x86_64-pc-windows-gnu\debug\deps\x.50blxss3ek1uzwfi.rcgu.o" "-nodefaultlibs" "-L" "D:\proj\rust\cargo\scratch\x\target\x86_64-pc-windows-gnu\debug\deps" "-L" "D:\proj\rust\cargo\scratch\x\target\debug\deps" "-L" "D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib" "-Wl,--start-group" "-Wl,-Bstatic" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libstd-9613caf92bf338df.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libpanic_unwind-e14e7db32928b206.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libbacktrace_sys-9cbbef916a77eb16.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\librustc_demangle-5a2ee4a924859898.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libhashbrown-b73edc3c5895801b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\librustc_std_workspace_alloc-23aa1c436e3798da.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libunwind-1bd4a2e47098ecbf.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\liblibc-f7efcf93314b1f3f.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\liballoc-0e5af186936c6975.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\librustc_std_workspace_core-d2692d38939d9cba.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libcore-01cec8cca5453d6f.rlib" "-Wl,--no-whole-archive" "-Wl,--end-group" "C:\Users\Eric\AppData\Local\Temp\rustczCJ0To\libcompiler_builtins-e880191e806bbc81.rlib" "-Wl,-Bdynamic" "-ladvapi32" "-lws2_32" "-luserenv" "-Wl,-Bstatic" "-lgcc_eh" "-lpthread" "-shared" "-Wl,-Bdynamic" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmsvcrt" "-luser32" "-lkernel32" "D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib\rsend.o"
= note: D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib\dllcrt2.o:crtdll.c:(.rdata$.refptr.__onexitend[.refptr.__onexitend]+0x0): undefined reference to `__onexitend'
D:\rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-gnu\lib\dllcrt2.o:crtdll.c:(.rdata$.refptr.__onexitbegin[.refptr.__onexitbegin]+0x0): undefined reference to `__onexitbegin'
collect2.exe: error: ld returned 1 exit status

chouquette · 2019-04-29T08:03:32Z

I agree that it easily confusing.

TL;DR: mingw is the original windows toolchain to cross compile from linux to windows
mingw-w64 is a fork that originated about 10 years ago, with among other things, a better 64bits support.

chouquette · 2019-04-29T08:11:32Z

Arg sorry, posted too early.

I have no experience with mingw, and have only used/contributed to mingw-w64.
Copying the various begin/end object files is supposedly what the compiler already does: https://github.com/rust-lang/rust/blob/master/src/bootstrap/compile.rs#L309

From a quick look I don't see any reference to onexitbegin/onexitend in either gcc/llvm not mingw-w64 so I wonder it it's not a mingw (non -w64) specificity.

This was already done when targetting msvc, but it needs to be done for all targets to allow rustc to link with shared libraries when using llvm -l<dllname> is only supported by gcc, while llvm needs an import library to link indirectly with a dll

alexcrichton · 2019-07-19T18:53:04Z

@chouquette to double-check, is this still necessary for the UWP targets?

mati865 · 2019-07-19T19:21:15Z

@alexcrichton it's not UWP specific but LLD specific (using windows-gnu with LLD also requires this PR). It came out with UWP because the best toolchain for this target is is based on LLVM and accompanying projects.

@ehuss it's known Rust bug: rust-lang/rust#47048 the best workaround known to me is replacing problematic libraries like here: https://github.com/rust-lang/libc/pull/1376/files#diff-479f71e48abedfd84790e1287381eb8dR49
and configuring linker in .cargo/config:

[target.x86_64-pc-windows-gnu]
linker = "D:/msys64/mingw64/bin/gcc.exe"

mati865 · 2020-02-09T14:54:49Z

@ehuss your issue is mostly fixed on nightly.
~~If you install MinGW via Chocolatey you will have to add C:/ProgramData/Chocolatey/lib/mingw/tools/install/mingw64/bin to the PATH though.~~
No longer necessary to modify PATH.

Could you test it again?

mati865

@ehuss I'd really appreciate if it could make it to the bootstrap compiler soon since it makes working on linking Rust with LLD much much easier.
Let me know if you need any assistance.

mati865 · 2020-04-02T09:02:30Z

src/cargo/core/compiler/build_context/target_info.rs

@@ -213,6 +213,17 @@ impl TargetInfo {
                should_replace_hyphens: false,
            })
        }
+        else if target_triple.ends_with("windows-gnu")


Suggested change

else if target_triple.ends_with("windows-gnu")

// LD can link DLL directly but LLD requires import library

if target_triple.ends_with("windows-gnu")

ehuss · 2020-04-19T18:32:46Z

I'm sorry for the long delay. This change seems reasonable to me.

@alexcrichton Would it be OK to add a x86_64-pc-windows-gnu job to Cargo's CI? I'd like to test this, and in general I think it would be good to have some test coverage for windows-gnu.

alexcrichton · 2020-04-19T19:08:34Z

Yeah I'd be fine adding that, it's been tricky historically but it's probably much easier now!

mati865 · 2020-04-20T12:41:03Z

I'll see if I can make tests pass. Due to recent changes to MinGW targets I suppose the tests will yield different results for stable, beta and nightly.

ehuss · 2020-04-20T16:23:27Z

@mati865 That would be great if you can help. Would you be up for opening a PR? I started, but I only got as far as:

diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 392ecb9ed..fdd3141b5 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -41,6 +41,9 @@ jobs:
       x86_64-msvc:
         TOOLCHAIN: stable-x86_64-pc-windows-msvc
         OTHER_TARGET: i686-pc-windows-msvc
+      x86_64-gnu:
+        TOOLCHAIN: nightly-x86_64-pc-windows-gnu
+        OTHER_TARGET: i686-pc-windows-gnu

 - job: rustfmt
   pool:
diff --git a/ci/azure-install-rust.yml b/ci/azure-install-rust.yml
index f37b4a6ab..f11171506 100644
--- a/ci/azure-install-rust.yml
+++ b/ci/azure-install-rust.yml
@@ -4,7 +4,7 @@ steps:
       rustup set profile minimal
       rustup component remove --toolchain=$TOOLCHAIN rust-docs || echo "already removed"
       rustup update --no-self-update $TOOLCHAIN
-      if [ "$TOOLCHAIN" = "nightly" ]; then
+      if [[ "$TOOLCHAIN" == "nightly"* ]]; then
         rustup component add --toolchain=$TOOLCHAIN rustc-dev
       fi
       rustup default $TOOLCHAIN

I think using nightly would be good just to mix things up (since we don't have any windows nightly tests). The tests that failed were:

build::cdylib_final_outputs
build::cdylib_not_lifted
build_script::build_script_with_dynamic_native_dependency
features::feature_off_dylib
out_dir::avoid_build_scripts
out_dir::binary_with_debug
out_dir::cargo_build_out_dir
out_dir::dynamic_library_with_debug
out_dir::include_only_the_binary_from_the_current_package
out_dir::static_library_with_debug
plugins::plugin_with_dynamic_native_dependency
standard_lib::basic
standard_lib::bench
standard_lib::simple_bin_std
standard_lib::test
tree::filters_target

The build-std tests should be disabled with a link to rust-lang/wg-cargo-std-aware#46. The out_dir tests look pretty straightforward (need to add the filenames for windows-gnu). I haven't looked at the others.

mati865 · 2020-04-20T17:04:09Z

@ehuss I'm waiting on my Azure Pipelines before opening WIP PR.
So far I have:

failures:
    features::feature_off_dylib
    init::ignores_failure_to_format_source
    standard_lib::basic
    standard_lib::bench
    standard_lib::simple_bin_std
    standard_lib::test

features::feature_off_dylib needs deeper investigation.
init::ignores_failure_to_format_source probably won't happen on CI since this test modifies PATH and when building locally one of crates pulls openssl from MSYS2:

$ ntldd /d/cargo/target/debug/cargo.exe
        KERNEL32.dll => C:\WINDOWS\SYSTEM32\KERNEL32.dll (0x000000000e880000)
        SHELL32.dll => C:\WINDOWS\SYSTEM32\SHELL32.dll (0x000000000e880000)
        libcrypto-1_1-x64.dll => D:\msys64\mingw64\bin\libcrypto-1_1-x64.dll (0x000000000e880000)
        zlib1.dll => D:\msys64\mingw64\bin\zlib1.dll (0x00000000001c0000)
        ADVAPI32.dll => C:\WINDOWS\SYSTEM32\ADVAPI32.dll (0x000000000f170000)
        CRYPT32.dll => C:\WINDOWS\SYSTEM32\CRYPT32.dll (0x000000000f310000)
        msvcrt.dll => C:\WINDOWS\SYSTEM32\msvcrt.dll (0x000000000f310000)
        ntdll.dll => C:\WINDOWS\SYSTEM32\ntdll.dll (0x000000000e940000)
        ole32.dll => C:\WINDOWS\SYSTEM32\ole32.dll (0x000000000f170000)
        RPCRT4.dll => C:\WINDOWS\SYSTEM32\RPCRT4.dll (0x000000000f310000)
        USER32.dll => C:\WINDOWS\SYSTEM32\USER32.dll (0x000000000f170000)
        USERENV.dll => C:\WINDOWS\SYSTEM32\USERENV.dll (0x00000000001c0000)
        WINHTTP.dll => C:\WINDOWS\SYSTEM32\WINHTTP.dll (0x000000000f660000)
        WS2_32.dll => C:\WINDOWS\SYSTEM32\WS2_32.dll (0x000000000f430000)

Add windows-gnu CI and fix tests One remaining failure: ``` ---- features::feature_off_dylib stdout ---- running `d:\a\1\s\target\debug\cargo.exe build --features f1` running `d:\a\1\s\target\debug\cargo.exe run -p bar` thread 'features::feature_off_dylib' panicked at ' Expected: execs but: exited with exit code: 101 --- stdout --- stderr Compiling foo v0.0.1 (D:\a\1\s\target\cit\t663\foo) Compiling bar v0.0.1 (D:\a\1\s\target\cit\t663\foo\bar) Finished dev [unoptimized + debuginfo] target(s) in 0.69s Running `target\debug\bar.exe` thread 'main' panicked at 'assertion failed: `(left == right)` left: `"f1"`, right: `"no f1"`', bar\src\main.rs:5:17 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace error: process didn't exit successfully: `target\debug\bar.exe` (exit code: 101) ', crates\cargo-test-support\src\lib.rs:833:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: features::feature_off_dylib ``` I disassembled the dylibs and `cargo run -p bar` correctly rebuilt it inside `target/debug/deps/` but did not copy it to `target/debug`. To further confirm, calling `cp target/debug/deps/foo.dll target/debug/` manually solved the issue. Any idea? ---- I left `FIXME` in places where import lib should be added with #6875. `TOOLCHAIN: nightly-x86_64-pc-windows-gnu` can be replaced with beta on Thursday.

Uplift windows gnu DLL import libraries. This is a repost of #6875, rebased with tests fixed.

ehuss · 2020-04-21T18:05:13Z

I have rebased this and fixed the tests in #8141.

rust-highfive assigned ehuss Apr 25, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 25, 2019

ehuss added the S-blocked label May 8, 2019

Add implib to the stamp files

7f1c3c7

This was already done when targetting msvc, but it needs to be done for all targets to allow rustc to link with shared libraries when using llvm -l<dllname> is only supported by gcc, while llvm needs an import library to link indirectly with a dll

chouquette force-pushed the uwp_support branch from 65173f6 to 7f1c3c7 Compare May 16, 2019 12:31

mati865 reviewed Apr 2, 2020

View reviewed changes

mati865 mentioned this pull request Apr 6, 2020

MinGW: improve compatibility with LLD rust-lang/rust#70852

Closed

ehuss removed the S-blocked label Apr 19, 2020

mati865 mentioned this pull request Apr 20, 2020

Add windows-gnu CI and fix tests #8139

Merged

ehuss mentioned this pull request Apr 21, 2020

Uplift windows gnu DLL import libraries. #8141

Merged

bors added a commit that referenced this pull request Apr 21, 2020

Auto merge of #8141 - ehuss:uwp_support, r=ehuss

8751eb3

Uplift windows gnu DLL import libraries. This is a repost of #6875, rebased with tests fixed.

ehuss closed this Apr 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add implib to the stamp files #6875

Add implib to the stamp files #6875

chouquette commented Apr 25, 2019

rust-highfive commented Apr 25, 2019

ehuss commented Apr 25, 2019

alexcrichton commented Apr 26, 2019

chouquette commented Apr 26, 2019

ehuss commented Apr 29, 2019

chouquette commented Apr 29, 2019

chouquette commented Apr 29, 2019

alexcrichton commented Jul 19, 2019

mati865 commented Jul 19, 2019 •

edited

Loading

mati865 commented Feb 9, 2020 •

edited

Loading

mati865 left a comment

mati865 Apr 2, 2020

ehuss commented Apr 19, 2020

alexcrichton commented Apr 19, 2020

mati865 commented Apr 20, 2020

ehuss commented Apr 20, 2020

mati865 commented Apr 20, 2020

ehuss commented Apr 21, 2020

Add implib to the stamp files #6875

Add implib to the stamp files #6875

Conversation

chouquette commented Apr 25, 2019

rust-highfive commented Apr 25, 2019

ehuss commented Apr 25, 2019

alexcrichton commented Apr 26, 2019

chouquette commented Apr 26, 2019

ehuss commented Apr 29, 2019

chouquette commented Apr 29, 2019

chouquette commented Apr 29, 2019

alexcrichton commented Jul 19, 2019

mati865 commented Jul 19, 2019 • edited Loading

mati865 commented Feb 9, 2020 • edited Loading

mati865 left a comment

Choose a reason for hiding this comment

mati865 Apr 2, 2020

Choose a reason for hiding this comment

ehuss commented Apr 19, 2020

alexcrichton commented Apr 19, 2020

mati865 commented Apr 20, 2020

ehuss commented Apr 20, 2020

mati865 commented Apr 20, 2020

ehuss commented Apr 21, 2020

mati865 commented Jul 19, 2019 •

edited

Loading

mati865 commented Feb 9, 2020 •

edited

Loading