-
Notifications
You must be signed in to change notification settings - Fork 12.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide a way of overriding the MinGW library path #68887
Comments
cc @alexcrichton
|
My hope is that this would be of far lesser urgency than #67429 because this only affects the startup object files which ideally change far less in the gcc toolchain than the rest of the toolchain. The main issue was that we'd prefer our own import libraries and such, which really was only effective if they came from the same toolchain as the host toolchain, which they rarely did. |
It'd be nice to have Rust's MinGW follow latest MinGW release but it's not backwards compatible: https://www.diffchecker.com/mjMudsm6 How does |
The toolchain used for MinGW has historically been "whatever is on the builders", so it's not strictly kept in sync with something or intentionally held back. It should be updateable at any time! The non-MinGW targets all rely on the linker to pull in the CRT. None of them work without a preinstalled toolchain, though, whereas one of the purposes of the MinGW toolchain is to work without a preinstalled toolchain. |
@alexcrichton there is another case of nonstandard path. 😔
|
Sorry I don't know of a great way to fix things, I know very little about MinGW myself. |
@mati865 , On AppVeyor CI (which uses a choco install of mingw), while testing, I used the relative path "..\lib\mingw\tools\install\mingw64" from the wrapper to the main directory of the install. I believe that should be a relatively stable path. It should be at least testable. |
@mati865 , After a bit of googling and stackoverflow search, I found a second, more robust, though more complicated/slower, method of finding the installation main directory (ref: SE). Using the C:>"c:\ProgramData\chocolatey\bin\gcc.exe" --shimgen-noop
[shim]: Set up Shim to run with the following parameters:
path to executable: c:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin\gcc.exe
working directory: C:\ProgramData\chocolatey\bin
is gui? False
wait for exit? True
command (optional):
[shim]: Command line '"c:\ProgramData\chocolatey\bin\gcc.exe" --shimgen-noop'
[shim]: Current process '"c:\ProgramData\chocolatey\bin\gcc.exe"'
[shim]: Command line after removing process ' --shimgen-noop'
[shim]: Arguments after removing shimgen args - ''
[shim]: Arguments are '' I hope that's helpful. |
@rivy Rust could get library path from |
@mati865 , yep, I think using the "usual" relative path (ie, "..\lib\mingw\tools\install\mingw64") would be the better choice. I guess you could fall back to parsing the I do want to say thanks for your efforts on this problem. |
Still, how about a fallback to some config item or environment variable so that it can be made work on unsupported layouts too without rebuilding rustc? |
@rivy calling @tanriol I'm not member of compiler team so I cannot tell but I think they'd accept unstable flag. I don't know that part of the compiler and would love if somebody else did it (if compiler team agrees). |
@mati865 , slow execution speed is an understandable concern, but "works slowly" is slightly better than "doesn't work". 😄 If |
@rivy I'm not decisive here.
Maybe it'd be feasible to do it only when cross compiling but you would need to discuss it with the team. |
Hello! After a false start of confusing myself on another issue thread :(, I've now realized that this is actually what is going on and thereby the issue I should be commenting on ;P. In my case, this is my "linker" (which is a command, per rust-lang/cargo#8071):
The main things to notice are that 1) I'm using clang and 2) I have a custom sysroot. I really need to be using my sysroot here for multiple reasons: 1) it is just more correct to have a consistent sysroot that I build myself; 2) the linker's default sysroot doesn't work due to #12859 (all of the MinGW toolchains I work with--macOS Homebrew and Ubuntu--default to SjLj); and 3) the ones that come with Rust don't work due to #49078 (the __onexit* symbols). So what I really need is to just get Rust to not break the usage of my sysroot somehow. As I'm cross compiling with a sysroot, the path of my linker really has absolutely nothing at all to do with the path of the binary being run as my linker (not even my underlying linker program, much less the wrapper script I have to generate to run my linker and pass it the right flags, are going to be inside of my sysroot). Of the options presented here, --shimgen-noop will not work as I'm not using gcc, but -print-file-name=dllcrt2.o does work as expected. FWIW, I would also be perfectly happy to provide the path manually if I had some way to do so: I don't really need this to be automated. |
@alexcrichton , @mati865 , how would I go about asking about adding such a feature when cross-compiling (especially on windows)? |
FWIW, I would argue that the "known path" methods without an explicit override mechanism are "wrong", as the compiler--which you are accepting as a "linker"--is itself providing an override (in the form of --sysroot) in addition to defaulting that to some relative path (meaning I might in fact have a sysroot that came with the compiler that I am explicitly not using), and so if the user is using gcc/clang for a linker you can't use alternative approaches only as a "fallback": if you want it to work automatically and correctly the most often I would think you should thereby ask the compiler for the path even if that is slow and then allow people who want to link faster to specify it manually, or if you want it to always be fast but sometimes wrong then you definitely need to provide the manual override for those people who overrode their compiler (personally, I would vote for the former with some caching, as my experiencing is that the MinGW linker--even when not run on Windows--is so much slower than any other linker I use that if you run an external program once or even "every now and then" I can't imagine that having much of an impact on overall build time: if it really really actually matters to someone they can then provide an override as an optimization). |
So, I am now also running into this problem with libmsvcrt.a, which is manifesting as the |
@saurik if you simply delete them the linker won't be able to find them. I'll try to find the time to fix it next week. |
@mati865 Thanks!! FWIW, I really am willing to make sure you can get compensated for the time to stare at this; like, this is the kind of fix I'd normally try to do myself, but I honestly don't know where to begin with respect to "if I change this in the code, what do I have to rebuild to test the change", as I am only right now at the stage of "beginning to integrate Rust" as opposed to "coding in Rust". At it's simplest, if have just like a PayPal account or something, I could at least donate/contribute. I also had written another comment last night that I thought I'd sent but am not finding anywhere, which is that I finally decided to try setting LINKER to "true" out of despair, and that actually "worked", in that the cruel irony here is that I don't even want this linker output: I'm linking these libraries into a larger application (hence why I need to use specific compilers and sysroots to be compatible with the rest of the object files) and so just want the staticlib, but Cargo has no way to override crate-types without editing the upstream Cargo.toml file, and so I end up getting a bunch of side outputs of cdylibs and the such that I have no real use for, but need to not fail ;P. I had had a plan to maybe write a wrapper to set as LINKER that would attempt to figure out which arguments were using Rust toolchain paths and rewrite them using a sysroot passed from the environment, but the realization that I didn't even want the linker output anyway (while writing up a comment in defense of one of those Cargo issues that had stagnated years ago) I realized maybe the build system would "just work" if I passed "true" and then I could get my staticlib, which I end up linking myself anyway, and that did in fact work great ;P. (In the end, though, getting this library to link is blocked on a bunch of other things; like, there is a simple-looking issue with bindgen not supporting iOS--it is passing the wrong --target to clang, as the iPhone uses arm64, not aarch64--and even if that were fixed, the dependency from bindgen needing access to llvm-config, which Apple doesn't ship, is also looking like it adds just enough operational complexity to what I'm doing here that I took a long and serious look last night at why I am using this library at all and finally decided I was stretching to need it in the first place.) So like, long story short: I still think this should really be fixed, as I've now seen a bunch of people strewn across many comments where people are running into this problem on both 32-bit and 64-bit MinGW (manifesting in different ways), and it seems like the -print-file-name fix will work; so, I'd thereby love to help sponsor getting it fixed to help all those people, and would definitely help test anything you come up with; but, since I finally decided I should just go with a different protocol that wouldn't require the usage of anything that indirectly depends on the Rust nettle-sys package, and don't really need to support 32-bit Windows, I'm personally no longer blocked on this issue. |
@mati865 The common thing for these targets is that rustc need to support two modes - native toolchain mode, and self-contained mode. For these two modes to work libraries and objects distributed with rustc for the self-contained mode should reside in a separate directory, like Native toolchain mode:
Self-contained mode:
So the main question is "how to enable/disable the self-contained mode", the two alternatives are:
EDIT: #71769 prepared some groundwork, so I planned to improve the heuristics ( |
I knew somebody will ask for that once I start working on MinGW side of that issue 😄
Sounds great but I'd make it in separate PR so users could already play with (unstable) option to prefer (force?) own path for CRT.
Musl declares itself to be both backward and forward compatible so it's users are fine when mixing different CRT versions. That would need discussion.
I think I saw all your linkage PR's and was hoping to see "even more magic". |
…r errors - `undefined reference to `_imp____acrt_iob_func'` - ref: <rust-lang/rust#68887 (comment)> - ref: <https://users.rust-lang.org/t/linking-with-gcc-failed-exit-code-1/34124> - ref: <https://www.gitmemory.com/issue/rust-lang/rust/47048/530376978> - ref: <https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/?limit=250&page=7>
…r errors - `undefined reference to `_imp____acrt_iob_func'` - ref: <rust-lang/rust#68887 (comment)> - ref: <https://users.rust-lang.org/t/linking-with-gcc-failed-exit-code-1/34124> - ref: <https://www.gitmemory.com/issue/rust-lang/rust/47048/530376978> - ref: <https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/?limit=250&page=7>
…, r=Mark-Simulacrum Create self-contained directory and move there some of external binaries/libs One of the steps to reach design described in rust-lang#68887 (comment) This PR moves things around and allows link code to handle the new directory structure.
Status update for anybody who follows this issue: |
I just tried using the homebrew-provided mingw64 and it fails linking with this error:
|
@lu-zero could you try with Short summary:
|
Let me undo my evil hack and I'll do, is it fine if I get you the tests results tomorrow? |
Sure, there are 2 weeks left to the branching 😉. |
Works as intended for a target with |
@mati865 The code compiled successfully so I'd assume it's ok. |
|
@saurik the plan is to always disable self contained mode when cross compiling and on Windows when the linker location is outside of Rust sysroot. To do it properly we need to stabilise |
Status update: |
@mati865 To verify, by "cross compiling" here we mean "when the user is using a toolchain that isn't the system default toolchain (and thereby has provided their own LINKER)", not "when the user happens to be compiling for a host/target architecture that is different from the build/host architecture" (to use x/y as gnu/cargo naming)? (Like, if I am on Win32 compiling for Win32, many people would not call that "cross compiling"--though I can see an argument that it is, and it is obvious to see that when one considers one things like "I am on Linux and am compiling a binary for Linux but I am on a system with glibc 2.23 and need the resulting binary to run on a system with glibc 2.17 and so I have a sysroot for an old version of CentOS that is my host/target system while a very recent build of Ubuntu is my build/host system": in some sense I am "cross compiling" from Ubuntu bionic to CentOS 7--but if I am specifying the linker and thereby have a sysroot I would need to not have these shipped objects injected no matter the architectures involved. I mostly ask this question as I ran into a really annoying bug in Cargo rust-lang/cargo#8147 wherein it incorrectly makes assumptions about toolchains based on architecture.) |
@saurik that's a good point. Until |
@mati865 FWIW, "it's expected when user provides own linker self-contained mode makes no sense" is what I agree with / want, so if you are saying that will be the behavior I have no issue? My belief is just that "self contained mode" is the "narrow" (even if very common) case that only makes sense if the user is using the rust-provided linker (and that that has nothing to do with "cross compiling"). |
My proposal for now:
Would you like to implement and open PR for "user manually specified the linker: for now just disable self-contained mode" (since it's your idea) or want me to do it? |
My proposal would be:
I don't see why host != target is relevant at all... :(. But, I will note that as I have said in passing a couple times, I am making an assumption about what this self-contained mode is even for and who is using it: my assumption is that this is essentially a compete toolchain with sysroot/libraries that ships with rust so the compiler can do builds for a handful of platforms that are common targets for which the user is unlikely to have a toolchain installed on their current computer, such as for non-standard libc on Linux, Win32, or maybe emscripten or something. If so, I would expect this to be purely a "fallback" option that would always be available for these targets regardless of the host and the tooling for which might cause "unwelcome" command line arguments if the user manually specified LINKER to use a linker they brought, and which would cause mix-and-march issues--like we are seeing in these various ones I have run into--were you to attempt to use the rust libraries with a linker that came from someone else, no matter where that linker is located... like, my confusion on the use case here is so strong I honestly don't see why you would have this as a -C flag: either you are using the self-container linker or you aren't, and there doesn't seem like there should be any grey area where we have to ask the user. Is the issue that rust isn't shipping a linker, and is only shipping the libraries? If so, I feel like that just needs to be fixed, as otherwise that is going to just keep causing problems forever; like, there is simply no forward compatibility guarantee for libraries: even on macOS old linkers can't link against new libraries as the load commands at the head of the binary are extended with new features every year. A linker and a sysroot aren't exactly glued together, but there is at least a loose level of coupling between them and it isn't safe to just mix and match them: either don't ship libraries and expect the user to bring them or ship a full toolchain. My confusion with PATH might be that I don't know if you are saying this is for linkers found on the path by rust without LINKER or linkers found on the path that were manually specified with LINKER. In the former case I see the relevance, but then "later" is strange as both rules are relevant, as custom linkers are often--but not always--not on your path and adding them to the path to be found might not work a seat the name might be strange. In the latter case, I am just really confused as most of my time on this issue I was actually using the system copy of MinGW that I installed from homebrew/Ubuntu: I only switched to an off-PATH linker at the very end when I "could" (as I happen to prefer using an off-path copy of clang from the Android SDK as a wrapper for the on-PATH copy of mingw-ld... but I could just as easily be using the distribution's default install of clang and maybe even lld instead of mingw and now everything is both on path and strange); but even to the extent to which my toolchain is off-PATH today doesn't mean I might not be setting my PATH differently tomorrow (and sometimes I do this) so that weird folder is on my path. Taking a further step back, once I saw how the command line being passed to my linker was complete "vanilla" and it worked, I don't even understand what this option does: why not pass the vanilla options to your own self-contained linker? If the paths are internally wrong on that linker and it can't find it's sysroot, why not fix that in the self-contained linker? I would expect the self-contained linker to work just like my linker does, and my linker "just works". Why does the rust self contained linker fail to "just work" in a way that requires the crt objects to be manually passed in? (As for a PR, I am not in a position to test this without some maybe-hopefully-minimal instruction. I would be willing to accept said instruction, and maybe that will mean you get more patches out of me in the future ;P, but I just don't know what all I need to recompile to make this kind of change or then how to use that alternative copy: I have been "using" the rust toolchain, but I have so far not developed on it myself. Note that I continue to be willing to provide some compensation for you doing changes--assuming you agree with them, of course--in addition to paying for your time on this already that I might have caused, as me doing my job to pay you to do these tasks probably is arguably efficient for the world than me doing this directly; but again: if there is some minimal-investment-to-you way to tell me what I need to rebuild to make a change and test it, given that I seem to be resourceful enough to understand how to use the tooling itself and have been a maintainer of compilers in the past, I am definitely willing to do that and maybe seeing what I am suggesting manifested concretely will make more sense, or the process of me editing the code will help me understand the use case and thereby come up with a more reasonable suggestion for what to change.) |
I'll get to that wall of text later 😉.
For MinGW the linker is shipped only when it's the host compiler and is preferred over external linkers in the PATH. So we assume that if user overridden the linker they don't want self contained mode. |
@saurik Apologies, I was quite busy recently. The change itself is trivial and comes to adding condition that check if linker option is none, similarly to: There is a (big) catch however, testing this change is complicated. You have to modify https://github.com/mati865/rust/blob/08990e5c7554d4d6a0440debc2edc99c8e9565c8/src/librustc_codegen_ssa/back/link.rs#L1017 to always return
|
…cross-compiling, r=petrochenkov MinGW: disable self-contained mode when cross compiling When cross compiling users have to provide own linker and libraries anyway. Using rust provided MinGW crt objects is harmful here and has no benefits. cc rust-lang#68887
If somebody still encounters the issue with tomorrow's nightly, please open new report. |
The recently landed #67429 introduced a heuristic to search for the system MinGW libraries relative to the cross-gcc (used as linker) itself. However, there are systems that do not follow this pattern. For example, on my Gentoo system:
/usr/bin/x86_64-w64-mingw32-gcc
is a symlink to/usr/x86_64-pc-linux-gnu/x86_64-w64-mingw32/gcc-bin/9.2.0/x86_64-w64-mingw32-gcc
/usr/x86_64-w64-mingw32/usr/lib/
is the directory with the actual libraries in the/usr/x86_64-w64-mingw32
sysrootIt would be useful to have a way of overriding the heuristics and informing
rustc
about the correct library path manually. In the current configuration evenRUSTFLAGS="-L /usr/x86_64-w64-mingw32/usr/lib/"
does not help because the rustc-provided libraries have higher priority.The text was updated successfully, but these errors were encountered: