Emscripten: Drop -lc the emcc driver will add it when it is needed

[hoodmane]

According to @sbc100, when linking with emcc we shouldn't pass -lc:

your best bet is to remove any/all explicit additions of -lc, since
the compiler driver (emcc) will add it if/when its needed.
(I don't think this is an emscripten bug BTW, but a rustc driver bug)

emscripten-core/emscripten#17191 (comment)

Resolves #98155.

[rust-highfive]

r? @cjgillot

(rust-highfive has picked a reviewer for you, use r? to override)

[hoodmane]

r? @compiler-errors

[compiler-errors]

@bors r=compiler-errors,hoodmane

[bors]

📌 Commit d565b20 has been approved by compiler-errors,hoodmane

[petrochenkov]

I haven't investigated the details here, but what this PR does looks really hacky.

Why does rustc pass -lc in the first place, if it's always eliminated later?
Are there cases when -lc is indeed needed ? (non SIDE_MODULE?)
If linking of "fundamental" libraries is determined entirely by emcc, then no_default_libraries: false should be used in the target spec and libc crate shouldn't link anything.

[compiler-errors]

@bors r-

@petrochenkov do you have concerns about landing this? happy to let this sit for discussion instead of just r+'ing it.

[bjorn3]

@bors r-

[sbc100]

I haven't investigated the details here, but what this PR does looks really hacky.

Why does rustc pass -lc in the first place, if it's always eliminated later? Are there cases when -lc is indeed needed ? (non SIDE_MODULE?) If linking of "fundamental" libraries is determined entirely by emcc, then no_default_libraries: false should be used in the target spec and libc crate shouldn't link anything.

That does sounds like a more ideal solution. Otherwise the rust driver will be forever playing catchup with the emcc logic as to which default libraries it should link.

[Mark-Simulacrum]

Closing and reopening to bump bors (hopefully).

[hoodmane]

Setting no_default_libraries: false doesn't fix this problem though. As far as I can tell, setting that removes -s DEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[] but it still passes -lc to Emscripten. I personally think that Emscripten should discard -lc but @sbc100 doesn't seem to be in support of that.

[hoodmane]

@sbc100 based on your comment, could Emscripten just always ignore -lc as an argument?

[bors]

😪 I'm awake I'm awake

[sbc100]

@sbc100 based on your comment, could Emscripten just always ignore -lc as an argument?

It could, but fixing this on the rust side rather makes more sense to me.

[hoodmane]

Could you explain why it makes more sense? I honestly have no clue. I lean in favor of modify-Rust over modify-Emscripten because we are building several hundred dynamic Emscripten libraries with ~5 different build systems and Rust is the only one that is passing -lc to the Emscripten linker. This makes me think that Rust is doing something weird rather than Emscripten.

What does native code do? I guess if I am using symbols from libc, the linker would recognize them as libc symbols and include them automatically? But surely the normal linker wouldn't cause an error if I explicitly pass -lc?

[sbc100]

So to be clear, passing -lc to emcc when linking a main module is a no op and when linking a side module causes an error?

I think #17191 is failing because -lc is mapped to -lc-debug and libc-debug.a has not yet been built, and building as side module I guess doesn't trigger emscirpten's install "build any system library that is missing" and instead ends up just reporting -lc-debug as missing.

However, even if we fix it to build libc-debug.a (you can do this yourself by running ./embuilder build libc-debug --pic BTW) it would still be a semantic error to link libc in your side module. This is because libc (under emscripten) is always a static library and linking it into a side module would result in multiple copied of libc be included at runtime which can be really bad (think multiple copies of malloc and free).

So, there maybe things we can do to improve the ergonomics when somebody tried to link as static system library (such as libc) into a side module (e.g. make it hard error), but its remains the wrong thing to do (I think).

[hoodmane]

I tried fixing it up so emcc would find the actual libc once and there were a lot of errors about relocations.

So if we agree that rustc shouldn't pass -lc to emcc, then the question is how best to make it stop doing that. This PR is pretty hacky but it was easy and it does fix the problem. I think the current situation that we have to remove -lc in a linker wrapper is pretty unsatisfactory.

I would be okay with adding more complete support for emscripten side modules, but I would need advice on how to do that. Would people be okay with adding a comment here saying: "TODO: this is a hack. remove when we add support for Emscripten side modules" and merging this?

@RReverser

[sbc100]

What about that suggested solution from #98303 (comment)?

[hoodmane]

With no_default_libraries: true it passes the setting -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[]. If you set no_default_libraries: false, it won't do this. -lc is passed either way.

rust/compiler/rustc_codegen_ssa/src/back/linker.rs

Line 1143 in 5750a6a

self.cmd.args(&["-s", "DEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[]"]);

[sbc100]

So who makes the decision to add -lc in the first place? Is that something you can change?

[hoodmane]

Passing -Zlink-native-libraries=no to rustc seems to turn it off.

[sbc100]

Passing -Zlink-native-libraries=no to rustc seems to turn it off.

Is that enough to solve your side module linking issue?

[hoodmane]

Yup.

[bjorn3]

I believe rustc currently assumes that if dynamic libraries are supported, libc itself is dynamically linked. This is true for every other platform afaik. Why is it not true for emscripten?

[petrochenkov]

With no_default_libraries: true it passes the setting -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[].

That's something strange, emcc is similar to gcc/clang, so no_default_libraries: true is supposed to translate to something like -nodefaultlibs and remove things like libc/libgcc/libunwind from linking.

So it looks like emscripten targets are already effectively no_default_libraries: false now.
That means the libc crate (https://crates.io/crates/libc) shouldn't link anything on those targets, but it currently does link libc.

[petrochenkov]

fn set_output_kind with EmLinker does nothing, so apparently the actual output type setting bypasses rustc and is done using raw linker arguments like -sSIDE_MODULE=2.
That's pretty suspicious too.

How do output kinds with emscripten targets map on rustc's crate types CrateType/LinkOutputKind (executable, C-style dynamic library, etc)?
It would be preferable to set them through fn set_output_kind if possible.

no_default_libraries: false is rarely used by rustc targets, typically if the linker has some "secret knowledge" that cannot be easily reproduced in rustc and libc/libunwind crates.
But if emscripten output kinds do not map on rustc's output kinds well, then it might be the case for no_default_libraries: false.

[petrochenkov]

Another possibility would be to turn link_dynlib into a no-op when linking a side module:

This looks like something reasonable.
Even if the issue with -lc specifically is resolved with no_default_libraries: false, it may resurface with some other crate attempts to link some different library dynamically.
It will require rustc gaining knowledge about emscripten's side modules though, apparently that's something unique without equivalents in other targets that we currently have.

What exactly should not be linked to side modules, and how can we recognize it automatically?

• dynamic libraries (can be addressed by fn link_dylib)
• static "fundamental" libraries like libc (can be addressed by #[link(name = "c", cfg(not(some_predicate_that_is_true_for_side_modules)))] or by no_default_libraries: false)
• some other static libraries???

[hoodmane]

I think I will close this PR and open an issue. Passing -Zlink-native-libraries=no makes linking side modules much more ergonomic and can be used right away while we work on a real fix.

[hoodmane]

I will make a PR that uses set_output_kind to make Emscripten dynamic libraries when building a cdylib. I'm not sure if this is a good change, but I think it will at least help with the discussion.

[sbc100]

With no_default_libraries: true it passes the setting -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[].

That's something strange, emcc is similar to gcc/clang, so no_default_libraries: true is supposed to translate to something like -nodefaultlibs and remove things like libc/libgcc/libunwind from linking.

So it looks like emscripten targets are already effectively no_default_libraries: false now. That means the libc crate (https://crates.io/crates/libc) shouldn't link anything on those targets, but it currently does link libc.

-sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE=[] is almost certainly wrong/out-of-date. This setting defaults to the empty list anyway these days so its redundant. Also we now support -nodefaultlibs so you can use that, as with other toolchains.

[sbc100]

I believe rustc currently assumes that if dynamic libraries are supported, libc itself is dynamically linked. This is true for every other platform afaik. Why is it not true for emscripten?

Its partly technical, partly historical, and partly based on the peculiarities of the web, but emscrpiten's dynamic linking support is based on the MAIN_MODULE/SIDE_MODULE split with all the system libraries being linked into the main module. Its not ideal but that is the way it works today.

[petrochenkov]

Passing -Zlink-native-libraries=no makes linking side modules much more ergonomic and can be used right away while we work on a real fix.

-Zlink-native-libraries=no will also skip all static libraries though.