Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provide a way of overriding the MinGW library path #68887

Closed
tanriol opened this issue Feb 6, 2020 · 41 comments · Fixed by #76167
Closed

provide a way of overriding the MinGW library path #68887

tanriol opened this issue Feb 6, 2020 · 41 comments · Fixed by #76167
Labels
A-linkage Area: linking into static, shared libraries and binaries C-feature-request Category: A feature request, i.e: not implemented / a PR. O-windows-gnu Toolchain: GNU, Operating system: Windows T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@tanriol
Copy link
Contributor

tanriol commented Feb 6, 2020

The recently landed #67429 introduced a heuristic to search for the system MinGW libraries relative to the cross-gcc (used as linker) itself. However, there are systems that do not follow this pattern. For example, on my Gentoo system:

  • /usr/bin/x86_64-w64-mingw32-gcc is a symlink to /usr/x86_64-pc-linux-gnu/x86_64-w64-mingw32/gcc-bin/9.2.0/x86_64-w64-mingw32-gcc
  • /usr/x86_64-w64-mingw32/usr/lib/ is the directory with the actual libraries in the /usr/x86_64-w64-mingw32 sysroot

It would be useful to have a way of overriding the heuristics and informing rustc about the correct library path manually. In the current configuration even RUSTFLAGS="-L /usr/x86_64-w64-mingw32/usr/lib/" does not help because the rustc-provided libraries have higher priority.

@jonas-schievink jonas-schievink added A-linkage Area: linking into static, shared libraries and binaries C-feature-request Category: A feature request, i.e: not implemented / a PR. O-windows-gnu Toolchain: GNU, Operating system: Windows T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 6, 2020
@mati865
Copy link
Contributor

mati865 commented Feb 7, 2020

cc @alexcrichton
My comment in the PR:

Maybe it'd be possible to reuse one of Cargo configuration fields or add new one to specify path of system libraries.

@alexcrichton
Copy link
Member

My hope is that this would be of far lesser urgency than #67429 because this only affects the startup object files which ideally change far less in the gcc toolchain than the rest of the toolchain. The main issue was that we'd prefer our own import libraries and such, which really was only effective if they came from the same toolchain as the host toolchain, which they rarely did.

@mati865
Copy link
Contributor

mati865 commented Feb 7, 2020

It'd be nice to have Rust's MinGW follow latest MinGW release but it's not backwards compatible: https://www.diffchecker.com/mjMudsm6
Although it's worth mentioning the last breaking change to the CRT happened 2 years ago. This means using newer MinGW would already increase amount of systems where it just works.

How does *-linux-gnu target work in regard of CRT? It doesn't ship startup objects unlike *-linux-musl and *-windows-gnu targets.

@alexcrichton
Copy link
Member

The toolchain used for MinGW has historically been "whatever is on the builders", so it's not strictly kept in sync with something or intentionally held back. It should be updateable at any time!

The non-MinGW targets all rely on the linker to pull in the CRT. None of them work without a preinstalled toolchain, though, whereas one of the purposes of the MinGW toolchain is to work without a preinstalled toolchain.

@mati865
Copy link
Contributor

mati865 commented Feb 8, 2020

@alexcrichton there is another case of nonstandard path. 😔
This time on Windows when MinGW is installed through Chocolatey #68872 (comment)

I'll check if it is using symlink which we follow but I'm at a loss.
Chocolatey install wrappers so there is really no way to get location of original executable.

@alexcrichton
Copy link
Member

Sorry I don't know of a great way to fix things, I know very little about MinGW myself.

@rivy
Copy link

rivy commented Feb 15, 2020

@mati865 , On AppVeyor CI (which uses a choco install of mingw), while testing, I used the relative path "..\lib\mingw\tools\install\mingw64" from the wrapper to the main directory of the install. I believe that should be a relatively stable path. It should be at least testable.

@rivy
Copy link

rivy commented Feb 15, 2020

@mati865 ,

After a bit of googling and stackoverflow search, I found a second, more robust, though more complicated/slower, method of finding the installation main directory (ref: SE). Using the --shimgen-noop option to gcc returns a reference to the true installation path (see path to executable).

C:>"c:\ProgramData\chocolatey\bin\gcc.exe" --shimgen-noop
[shim]: Set up Shim to run with the following parameters:
  path to executable: c:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin\gcc.exe
  working directory: C:\ProgramData\chocolatey\bin
  is gui? False
  wait for exit? True
  command (optional):
[shim]: Command line '"c:\ProgramData\chocolatey\bin\gcc.exe"  --shimgen-noop'
[shim]: Current process '"c:\ProgramData\chocolatey\bin\gcc.exe"'
[shim]: Command line after removing process '  --shimgen-noop'
[shim]: Arguments after removing shimgen args - ''
[shim]: Arguments are ''

I hope that's helpful.

@mati865
Copy link
Contributor

mati865 commented Feb 15, 2020

@rivy Rust could get library path from gcc itself but calling executables on Windows is slow.
I was about to add Chocolatey structure as one of the paths (and fix Cygwin as well) but there was a lot going on for me this week.

@rivy
Copy link

rivy commented Feb 17, 2020

@mati865 , yep, I think using the "usual" relative path (ie, "..\lib\mingw\tools\install\mingw64") would be the better choice. I guess you could fall back to parsing the gcc --shimgen-noop output, but that's a lot of effort for a low percentage occurrence.

I do want to say thanks for your efforts on this problem.

@tanriol
Copy link
Contributor Author

tanriol commented Feb 17, 2020

Still, how about a fallback to some config item or environment variable so that it can be made work on unsupported layouts too without rebuilding rustc?

@mati865
Copy link
Contributor

mati865 commented Feb 17, 2020

@rivy calling gcc -print-file-name=crt2.o would be better than gcc --shimgen-noop but it's still calling external executable which is slow on Windows. This was one of the concerns of regarding my PR.

@tanriol I'm not member of compiler team so I cannot tell but I think they'd accept unstable flag. I don't know that part of the compiler and would love if somebody else did it (if compiler team agrees).
Another option would be upgrading mingw-w64 which Rust CI is using but I don't know which team "controls" it.

@rivy
Copy link

rivy commented Mar 1, 2020

@mati865 , slow execution speed is an understandable concern, but "works slowly" is slightly better than "doesn't work". 😄

If gcc is on the PATH and requested as a compiler, I do think that searching via faster known-path methods with a fallback to the slower gcc -print-file-name=crt2.o would be very reasonable. That would also essentially solve the "unsupported layouts" issue raised by @tanriol .

@mati865
Copy link
Contributor

mati865 commented Mar 1, 2020

@rivy I'm not decisive here.

If gcc is on the PATH and requested as a compiler, I do think that searching via faster known-path methods with a fallback to the slower gcc -print-file-name=crt2.o would be very reasonable.

Maybe it'd be feasible to do it only when cross compiling but you would need to discuss it with the team.

@saurik
Copy link

saurik commented Apr 18, 2020

Hello! After a false start of confusing myself on another issue thread :(, I've now realized that this is actually what is going on and thereby the issue I should be commenting on ;P. In my case, this is my "linker" (which is a command, per rust-lang/cargo#8071):

/path/to/any/clang++ -D_WIN32_WINNT=0x0601 -target i686-pc-windows-gnu --sysroot /build/mingw32 -B/usr/local/bin/i686-w64-mingw32- "$@"

The main things to notice are that 1) I'm using clang and 2) I have a custom sysroot. I really need to be using my sysroot here for multiple reasons: 1) it is just more correct to have a consistent sysroot that I build myself; 2) the linker's default sysroot doesn't work due to #12859 (all of the MinGW toolchains I work with--macOS Homebrew and Ubuntu--default to SjLj); and 3) the ones that come with Rust don't work due to #49078 (the __onexit* symbols). So what I really need is to just get Rust to not break the usage of my sysroot somehow.

As I'm cross compiling with a sysroot, the path of my linker really has absolutely nothing at all to do with the path of the binary being run as my linker (not even my underlying linker program, much less the wrapper script I have to generate to run my linker and pass it the right flags, are going to be inside of my sysroot).

Of the options presented here, --shimgen-noop will not work as I'm not using gcc, but -print-file-name=dllcrt2.o does work as expected. FWIW, I would also be perfectly happy to provide the path manually if I had some way to do so: I don't really need this to be automated.

@rivy
Copy link

rivy commented Apr 18, 2020

@rivy I'm not decisive here.

If gcc is on the PATH and requested as a compiler, I do think that searching via faster known-path methods with a fallback to the slower gcc -print-file-name=crt2.o would be very reasonable.

Maybe it'd be feasible to do it only when cross compiling but you would need to discuss it with the team.

@alexcrichton , @mati865 , how would I go about asking about adding such a feature when cross-compiling (especially on windows)?

@saurik
Copy link

saurik commented Apr 18, 2020

FWIW, I would argue that the "known path" methods without an explicit override mechanism are "wrong", as the compiler--which you are accepting as a "linker"--is itself providing an override (in the form of --sysroot) in addition to defaulting that to some relative path (meaning I might in fact have a sysroot that came with the compiler that I am explicitly not using), and so if the user is using gcc/clang for a linker you can't use alternative approaches only as a "fallback": if you want it to work automatically and correctly the most often I would think you should thereby ask the compiler for the path even if that is slow and then allow people who want to link faster to specify it manually, or if you want it to always be fast but sometimes wrong then you definitely need to provide the manual override for those people who overrode their compiler (personally, I would vote for the former with some caching, as my experiencing is that the MinGW linker--even when not run on Windows--is so much slower than any other linker I use that if you run an external program once or even "every now and then" I can't imagine that having much of an impact on overall build time: if it really really actually matters to someone they can then provide an override as an optimization).

@saurik
Copy link

saurik commented May 22, 2020

So, I am now also running into this problem with libmsvcrt.a, which is manifesting as the __imp___acrt_iob_func issue from #47048 :(. Is there anything I can do to help get this solution to run -print-file-name=crt2.o worked on? My company is willing to sponsor this work, as frankly I'm now at the point of just regretting even trying to use any Rust libraries in my program :(. I'm over a month in and still just fighting with basic stuff like "compiling things at all" (not just for MinGW), when I'm pretty sure there are even deeper issues in the Rust build system with respect to secure reproducible builds that I thought I'd be spending my time diagnosing. FWIW, I really just need a way to set the sysroot here: it doesn't have to be automatic... if nothing else, I'd seriously suggest deleting all of the libraries that Rust is shipping with, as those are pretty much never going to be correct and seem to be leading to some kind of unsafe assumed coupling. (cc @mati865 @pnkfelix)

@mati865
Copy link
Contributor

mati865 commented May 22, 2020

@saurik if you simply delete them the linker won't be able to find them. I'll try to find the time to fix it next week.

@saurik
Copy link

saurik commented May 22, 2020

@mati865 Thanks!! FWIW, I really am willing to make sure you can get compensated for the time to stare at this; like, this is the kind of fix I'd normally try to do myself, but I honestly don't know where to begin with respect to "if I change this in the code, what do I have to rebuild to test the change", as I am only right now at the stage of "beginning to integrate Rust" as opposed to "coding in Rust". At it's simplest, if have just like a PayPal account or something, I could at least donate/contribute.

I also had written another comment last night that I thought I'd sent but am not finding anywhere, which is that I finally decided to try setting LINKER to "true" out of despair, and that actually "worked", in that the cruel irony here is that I don't even want this linker output: I'm linking these libraries into a larger application (hence why I need to use specific compilers and sysroots to be compatible with the rest of the object files) and so just want the staticlib, but Cargo has no way to override crate-types without editing the upstream Cargo.toml file, and so I end up getting a bunch of side outputs of cdylibs and the such that I have no real use for, but need to not fail ;P.

I had had a plan to maybe write a wrapper to set as LINKER that would attempt to figure out which arguments were using Rust toolchain paths and rewrite them using a sysroot passed from the environment, but the realization that I didn't even want the linker output anyway (while writing up a comment in defense of one of those Cargo issues that had stagnated years ago) I realized maybe the build system would "just work" if I passed "true" and then I could get my staticlib, which I end up linking myself anyway, and that did in fact work great ;P.

(In the end, though, getting this library to link is blocked on a bunch of other things; like, there is a simple-looking issue with bindgen not supporting iOS--it is passing the wrong --target to clang, as the iPhone uses arm64, not aarch64--and even if that were fixed, the dependency from bindgen needing access to llvm-config, which Apple doesn't ship, is also looking like it adds just enough operational complexity to what I'm doing here that I took a long and serious look last night at why I am using this library at all and finally decided I was stretching to need it in the first place.)

So like, long story short: I still think this should really be fixed, as I've now seen a bunch of people strewn across many comments where people are running into this problem on both 32-bit and 64-bit MinGW (manifesting in different ways), and it seems like the -print-file-name fix will work; so, I'd thereby love to help sponsor getting it fixed to help all those people, and would definitely help test anything you come up with; but, since I finally decided I should just go with a different protocol that wouldn't require the usage of anything that indirectly depends on the Rust nettle-sys package, and don't really need to support 32-bit Windows, I'm personally no longer blocked on this issue.

@petrochenkov
Copy link
Contributor

petrochenkov commented May 23, 2020

@mati865
This issue is related to #72274, so the solution should ideally cover all of the targets for which we provide "self-contained experience for simple programs", i.e. mingw, musl and wasi.

The common thing for these targets is that rustc need to support two modes - native toolchain mode, and self-contained mode.

For these two modes to work libraries and objects distributed with rustc for the self-contained mode should reside in a separate directory, like <rustc-sysroot>/self-contained, rather than just in <rustc-sysroot>, so rustc could easily include or exclude this directory from the library and object search paths.

Native toolchain mode:

Self-contained mode:

So the main question is "how to enable/disable the self-contained mode", the two alternatives are:

  • Some rules and heuristics based on already existing options and environment (probing specific paths, running processes (expensive), checking whether the linker path has "musl" in it, checking whether the linker was passed explicitly, checking whether the linker is a path foo/bar rather than just a name bar, comparing host with the target, etc).
  • An new explicit option to set the mode - -C link-self-contained=yes/no or something like that, probably still with defaults based on heuristics above.

EDIT: #71769 prepared some groundwork, so I planned to improve the heuristics (fn crt_objects_fallback) next.

@mati865
Copy link
Contributor

mati865 commented May 23, 2020

@petrochenkov

This issue is related to #72274, so the solution should ideally cover all of the targets for which we provide "self-contained experience for simple programs", i.e. mingw, musl and wasi.

I knew somebody will ask for that once I start working on MinGW side of that issue 😄

The common thing for these targets is that rustc need to support two modes - native toolchain mode, and self-contained mode.

Sounds great but I'd make it in separate PR so users could already play with (unstable) option to prefer (force?) own path for CRT.

So the main question is "how to enable/disable the self-contained mode", the two alternatives are:

Musl declares itself to be both backward and forward compatible so it's users are fine when mixing different CRT versions. That would need discussion.
Mingw-w64 tries to be backward compatible but things are not always easy. Preferring user installed CRT over the shipped one is the only way to make objects built different compilers (like Rust crate linking to C library) compatible.
I think I've never used wasi so I have no clue about it.

EDIT: #71769 prepared some groundwork, so I planned to improve the heuristics (fn crt_objects_fallback) next.

I think I saw all your linkage PR's and was hoping to see "even more magic".

RalfJung added a commit to RalfJung/rust that referenced this issue Jun 19, 2020
…, r=Mark-Simulacrum

Create self-contained directory and move there some of external binaries/libs

One of the steps to reach design described in rust-lang#68887 (comment)
This PR moves things around and allows link code to handle the new directory structure.
@mati865
Copy link
Contributor

mati865 commented Jun 19, 2020

Status update for anybody who follows this issue:
With #72999 merged #72738 can be moved forward. As described in #68887 (comment) Rust will rely on the linker to provide correct paths. That will make option to override MinGW library path unnecessary.

@lu-zero
Copy link
Contributor

lu-zero commented Jul 1, 2020

I just tried using the homebrew-provided mingw64 and it fails linking with this error:

  = note: /usr/local/Cellar/mingw-w64/7.0.0_2/toolchain-x86_64/bin/x86_64-w64-mingw32-ld: /Users/lu_zero/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-pc-windows-gnu/lib/dllcrt2.o:crtdll.c:(.text+0x16): undefined reference to `_encode_pointer'
          /usr/local/Cellar/mingw-w64/7.0.0_2/toolchain-x86_64/bin/x86_64-w64-mingw32-ld: /Users/lu_zero/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-pc-windows-gnu/lib/dllcrt2.o:crtdll.c:(.text+0x1ac): undefined reference to `_decode_pointer'
          /usr/local/Cellar/mingw-w64/7.0.0_2/toolchain-x86_64/bin/x86_64-w64-mingw32-ld: /Users/lu_zero/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-pc-windows-gnu/lib/dllcrt2.o:crtdll.c:(.text+0x1c4): undefined reference to `_decode_pointer'
          /usr/local/Cellar/mingw-w64/7.0.0_2/toolchain-x86_64/bin/x86_64-w64-mingw32-ld: /Users/lu_zero/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-pc-windows-gnu/lib/dllcrt2.o:crtdll.c:(.rdata$.refptr.__onexitend[.refptr.__onexitend]+0x0): undefined reference to `__onexitend'
          /usr/local/Cellar/mingw-w64/7.0.0_2/toolchain-x86_64/bin/x86_64-w64-mingw32-ld: /Users/lu_zero/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-pc-windows-gnu/lib/dllcrt2.o:crtdll.c:(.rdata$.refptr.__onexitbegin[.refptr.__onexitbegin]+0x0): undefined reference to `__onexitbegin'
          collect2: error: ld returned 1 exit status

@mati865
Copy link
Contributor

mati865 commented Jul 1, 2020

@lu-zero could you try with -Z link-self-contained=no?
I wanted to make an "announcement" here asking people to try nightly with -Z link-self-contained=yes/no right after the nightly with #72738 has been published but forgot about it.
Detection improvements will come in next PR but I wanted to gather some data first.

Short summary:

  • link-self-contained=yes is meant for native Windows only with rust-mingw component installed. It allows to build crates without C dependencies even with MinGW C toolchain not installed.
  • link-self-contained=no forces to use external MinGW C toolchain so you always want to use this option when external MinGW C toolchain is available.
  • no option right now behaves exactly the same as it did before so it's a mix of link-self-contained=no and link-self-contained=yes. In 2 weeks branching will happen and once it's done I want to land PR that will rework MinGW detection and stabilise link-self-contained option.

@lu-zero
Copy link
Contributor

lu-zero commented Jul 1, 2020

Let me undo my evil hack and I'll do, is it fine if I get you the tests results tomorrow?

@mati865
Copy link
Contributor

mati865 commented Jul 1, 2020

Let me undo my evil hack and I'll do, is it fine if I get you the tests results tomorrow?

Sure, there are 2 weeks left to the branching 😉.

@lu-zero
Copy link
Contributor

lu-zero commented Jul 1, 2020

❯ RUSTFLAGS="-Z link-self-contained=no" cargo +nightly build --target x86_64-pc-windows-gnu

Works as intended for a target with cdylib crate type.

@mati865
Copy link
Contributor

mati865 commented Jul 1, 2020

Thanks!

@tanriol @saurik could you test if Rust nighty with -Z link-self-contained=no fixes the issue for you?

@tanriol
Copy link
Contributor Author

tanriol commented Jul 1, 2020

@mati865 The code compiled successfully so I'd assume it's ok.

@saurik
Copy link

saurik commented Jul 2, 2020

-Z link-self-contained=no worked so well--for both 64-bit and 32-bit--that I was even able to use clang as a wrapper for the linker instead of having to directly specify mingw's ld! \o/ I guess the question I have now is: if the user is specifying LINKER, shouldn't link-self-contained default to no? (I'm assuming the idea of having this self-contained environment in the first place is to avoid people having to bring their own install of MinGW, and so one is being shipped with rustc?) If you have an overridden LINKER, it seems like the probability that the libraries that come with rustc will be exactly compatible is "slim" (and so I'd expect "user brought their own linker, but wants to roll the die with the rustc embedded libraries" to be the corner case scenario that should require a flag override, not "user brought their own linker and thereby probably the linker works"; I didn't bring this up before because I had assumed there was some reason all these flags had to be passed manually, and thereby the paths needed to be known, but now seeing that "just call the linker" was an option from the beginning it seems like it should be the default in cases when the user provided the linker).

@mati865
Copy link
Contributor

mati865 commented Jul 2, 2020

@saurik the plan is to always disable self contained mode when cross compiling and on Windows when the linker location is outside of Rust sysroot.

To do it properly we need to stabilise link-self-contained first but that requires MCP.
There is open MCP already for what #72999 did and if it's not approved all the changes made so far will be probably reverted: rust-lang/compiler-team#310

@mati865
Copy link
Contributor

mati865 commented Jul 16, 2020

Status update:
MCP was seconded and since the beta was cut I've opened #74410 to completely ignore shipped CRT objects when cross compiling.

@saurik
Copy link

saurik commented Jul 16, 2020

@mati865 To verify, by "cross compiling" here we mean "when the user is using a toolchain that isn't the system default toolchain (and thereby has provided their own LINKER)", not "when the user happens to be compiling for a host/target architecture that is different from the build/host architecture" (to use x/y as gnu/cargo naming)? (Like, if I am on Win32 compiling for Win32, many people would not call that "cross compiling"--though I can see an argument that it is, and it is obvious to see that when one considers one things like "I am on Linux and am compiling a binary for Linux but I am on a system with glibc 2.23 and need the resulting binary to run on a system with glibc 2.17 and so I have a sysroot for an old version of CentOS that is my host/target system while a very recent build of Ubuntu is my build/host system": in some sense I am "cross compiling" from Ubuntu bionic to CentOS 7--but if I am specifying the linker and thereby have a sysroot I would need to not have these shipped objects injected no matter the architectures involved. I mostly ask this question as I ran into a really annoying bug in Cargo rust-lang/cargo#8147 wherein it incorrectly makes assumptions about toolchains based on architecture.)

@mati865
Copy link
Contributor

mati865 commented Jul 16, 2020

@saurik that's a good point. Until link-self-contained option is stabilised and heuristic for automatic external linker detection is added, it's expected when user provides own linker self-contained mode makes no sense. Do you want to open PR yourself?

@saurik
Copy link

saurik commented Jul 16, 2020

@mati865 FWIW, "it's expected when user provides own linker self-contained mode makes no sense" is what I agree with / want, so if you are saying that will be the behavior I have no issue? My belief is just that "self contained mode" is the "narrow" (even if very common) case that only makes sense if the user is using the rust-provided linker (and that that has nothing to do with "cross compiling").

@mati865
Copy link
Contributor

mati865 commented Jul 16, 2020

My proposal for now:

  • -Z link-self-contained=yes/no takes precedence over anything else, it will become stable -C flag at some point
  • host != target: self-contained mode is disabled
  • user manually specified the linker: for now just disable self-contained mode, later this will change to "is external linker available in the PATH?"

Would you like to implement and open PR for "user manually specified the linker: for now just disable self-contained mode" (since it's your idea) or want me to do it?

@saurik
Copy link

saurik commented Jul 17, 2020

My proposal would be:

  • if LINKER is set, then use that linker and assume it isn't self-contained (as the user should not even know where that linker is located to specify it manually... and if they do, honestly, I bet they are doing something epic and want all the magic behavior to go away);

  • otherwise if target is a platform for which rust ships a self-contained toolchain (I know of Win32 and Linux+musl or whatever) then use self-contained toolchain;

  • otherwise (if the user didn't specify LINKER and is expecting rust to just guess at a linker, but rust also doesn't ship its own preferable linker) if the user has a useful-looking linker on their path then use that linker without self-contained and hope (really hard) that it works well enough to make a useful binary (and if the user wants this linker over the default preferable self-contained one they should specify it manually with LINKER, which would throw them into the earlier case);

  • otherwise, throw an error to the user telling them they need to provide a linker.

I don't see why host != target is relevant at all... :(. But, I will note that as I have said in passing a couple times, I am making an assumption about what this self-contained mode is even for and who is using it: my assumption is that this is essentially a compete toolchain with sysroot/libraries that ships with rust so the compiler can do builds for a handful of platforms that are common targets for which the user is unlikely to have a toolchain installed on their current computer, such as for non-standard libc on Linux, Win32, or maybe emscripten or something. If so, I would expect this to be purely a "fallback" option that would always be available for these targets regardless of the host and the tooling for which might cause "unwelcome" command line arguments if the user manually specified LINKER to use a linker they brought, and which would cause mix-and-march issues--like we are seeing in these various ones I have run into--were you to attempt to use the rust libraries with a linker that came from someone else, no matter where that linker is located... like, my confusion on the use case here is so strong I honestly don't see why you would have this as a -C flag: either you are using the self-container linker or you aren't, and there doesn't seem like there should be any grey area where we have to ask the user.

Is the issue that rust isn't shipping a linker, and is only shipping the libraries? If so, I feel like that just needs to be fixed, as otherwise that is going to just keep causing problems forever; like, there is simply no forward compatibility guarantee for libraries: even on macOS old linkers can't link against new libraries as the load commands at the head of the binary are extended with new features every year. A linker and a sysroot aren't exactly glued together, but there is at least a loose level of coupling between them and it isn't safe to just mix and match them: either don't ship libraries and expect the user to bring them or ship a full toolchain.

My confusion with PATH might be that I don't know if you are saying this is for linkers found on the path by rust without LINKER or linkers found on the path that were manually specified with LINKER. In the former case I see the relevance, but then "later" is strange as both rules are relevant, as custom linkers are often--but not always--not on your path and adding them to the path to be found might not work a seat the name might be strange. In the latter case, I am just really confused as most of my time on this issue I was actually using the system copy of MinGW that I installed from homebrew/Ubuntu: I only switched to an off-PATH linker at the very end when I "could" (as I happen to prefer using an off-path copy of clang from the Android SDK as a wrapper for the on-PATH copy of mingw-ld... but I could just as easily be using the distribution's default install of clang and maybe even lld instead of mingw and now everything is both on path and strange); but even to the extent to which my toolchain is off-PATH today doesn't mean I might not be setting my PATH differently tomorrow (and sometimes I do this) so that weird folder is on my path.

Taking a further step back, once I saw how the command line being passed to my linker was complete "vanilla" and it worked, I don't even understand what this option does: why not pass the vanilla options to your own self-contained linker? If the paths are internally wrong on that linker and it can't find it's sysroot, why not fix that in the self-contained linker? I would expect the self-contained linker to work just like my linker does, and my linker "just works". Why does the rust self contained linker fail to "just work" in a way that requires the crt objects to be manually passed in?

(As for a PR, I am not in a position to test this without some maybe-hopefully-minimal instruction. I would be willing to accept said instruction, and maybe that will mean you get more patches out of me in the future ;P, but I just don't know what all I need to recompile to make this kind of change or then how to use that alternative copy: I have been "using" the rust toolchain, but I have so far not developed on it myself. Note that I continue to be willing to provide some compensation for you doing changes--assuming you agree with them, of course--in addition to paying for your time on this already that I might have caused, as me doing my job to pay you to do these tasks probably is arguably efficient for the world than me doing this directly; but again: if there is some minimal-investment-to-you way to tell me what I need to rebuild to make a change and test it, given that I seem to be resourceful enough to understand how to use the tooling itself and have been a maintainer of compilers in the past, I am definitely willing to do that and maybe seeing what I am suggesting manifested concretely will make more sense, or the process of me editing the code will help me understand the use case and thereby come up with a more reasonable suggestion for what to change.)

@mati865
Copy link
Contributor

mati865 commented Jul 17, 2020

I'll get to that wall of text later 😉.
For now I'll address only

I don't see why host != target is relevant at all...

For MinGW the linker is shipped only when it's the host compiler and is preferred over external linkers in the PATH. So we assume that if user overridden the linker they don't want self contained mode.
However when cross compiling users don't need to specify the linker because it will be taken from the PATH. Hence self contained mode is disabled entirely.

@mati865
Copy link
Contributor

mati865 commented Jul 30, 2020

@saurik Apologies, I was quite busy recently.

The change itself is trivial and comes to adding condition that check if linker option is none, similarly to:
#74410

There is a (big) catch however, testing this change is complicated. You have to modify https://github.com/mati865/rust/blob/08990e5c7554d4d6a0440debc2edc99c8e9565c8/src/librustc_codegen_ssa/back/link.rs#L1017 to always return None, then copy and paste few lines in CI config to create dist archive for windows-gnu.
Finally you need to run it on Windows machine with MinGW compiler added to the PATH and test that:

  • if -C linker=gcc was not specified rustc picks CRT objects from the sysroot
  • if -C linker=gcc was specified rustc picks CRT objects from the MinGW in the system

bors added a commit to rust-lang-ci/rust that referenced this issue Aug 10, 2020
…cross-compiling, r=petrochenkov

MinGW: disable self-contained mode when cross compiling

When cross compiling users have to provide own linker and libraries anyway.
Using rust provided MinGW crt objects is harmful here and has no benefits.

cc rust-lang#68887
@bors bors closed this as completed in a4e30a6 Sep 3, 2020
@mati865
Copy link
Contributor

mati865 commented Sep 3, 2020

-C link-self-contained has been stabilised in nightly and MinGW detection was reworked in the next nightly.

If somebody still encounters the issue with tomorrow's nightly, please open new report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-linkage Area: linking into static, shared libraries and binaries C-feature-request Category: A feature request, i.e: not implemented / a PR. O-windows-gnu Toolchain: GNU, Operating system: Windows T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants