Implement set_output_kind for Emscripten linker

[hoodmane]

This is on top of #98149. See also the earlier discussion on #98303. With this PR, a crate that specifies that it is a cdylib will compile to a working Emscripten dynamic library without adding any extra cargo, rustc, or linker flags and without fiddling with environment variables.

@sbc100

r? @petrochenkov

[rust-highfive]

⚠️ Warning ⚠️

• These commits modify compiler targets. (See the Target Tier Policy.)

[petrochenkov]

@hoodmane
Could you answer the question from #98303 (comment) in some more detail?
What output kinds does Emscripten have?
How do they map to Rust crate and output kinds?
What does -sMAIN_MODULE=0/1/2/? and -sSIDE_MODULE=0/1/2/? mean?
@rustbot author

[hoodmane]

Copying over the most relevant comments so we can have the context all in one place.

fn set_output_kind with EmLinker does nothing, so apparently the actual output type setting bypasses rustc and is done using raw linker arguments like -sSIDE_MODULE=2.
That's pretty suspicious too.

How do output kinds with emscripten targets map on rustc's crate types CrateType/LinkOutputKind (executable, C-style dynamic library, etc)?
It would be preferable to set them through fn set_output_kind if possible.

no_default_libraries: false is rarely used by rustc targets, typically if the linker has some "secret knowledge" that cannot be easily reproduced in rustc and libc/libunwind crates.
But if emscripten output kinds do not map on rustc's output kinds well, then it might be the case for no_default_libraries: false.

Originally posted by @petrochenkov in #98303 (comment)

It will require rustc gaining knowledge about emscripten's side modules though, apparently that's something unique without equivalents in other targets that we currently have.

What exactly should not be linked to side modules, and how can we recognize it automatically?

• dynamic libraries (can be addressed by fn link_dylib)
• static "fundamental" libraries like libc (can be addressed by #[link(name = "c", cfg(not(some_predicate_that_is_true_for_side_modules)))] or by no_default_libraries: false)
• some other static libraries???

Originally posted by @petrochenkov in #98303 (comment)

[hoodmane]

How do output kinds with emscripten targets map on rustc's crate types CrateType/LinkOutputKind (executable, C-style dynamic library, etc)?

I think the output kinds map perfectly reasonably. Emscripten has the following possible outputs:

• an executable: a pair of a .js file and a .wasm file and possibly a .data file, can be run with node
• a library libwhatever.a: works like a normal static library
• a dynamic library libwhatever.so: works similar to a normal dynamic library but with some additional restrictions on when/how you can load them since if the library is larger than some threshold loading is asynchronous.

The thing that is causing the trouble is that Emscripten has decided that in many cases when native programs would use dynamic linking it is preferable to use static linking. In interest of porting these dynamic builds to Emscripten static libs without changing their build system, if you pass -shared Emscripten prints a warning and emits a static library. Emscripten has the -sSIDE_MODULE flag to indicate that you actually want to make a dynamic library.

The problem right now is that both staticlib and cdylib crates emit static libraries and so there is no remaining crate type for "for real I want a dynamic library". In my opinion the most reasonable fix is for staticlib to make a static library and for cdylib to make a dynamic library. If people actually want an Emscripten static library, they can switch the crate type easily enough.

[hoodmane]

What exactly should not be linked to side modules, and how can we recognize it automatically?

I think the problem here is libraries that have both a static and a dynamic version. Emscripten doesn't have the same logic that normal compilers do for picking between them. In particular, it is not a problem if you pass libwhatever.so explicitly. I think the problem here isn't as bad as it seems and the biggest issue may really be libc. Of course it would be better to link a static library into the main executable once rather than several times into different shared libraries, but that isn't the end of the world.

It would be helpful to have more data on this though. I should try to build more crates that explicitly link non-rust libraries.

[hoodmane]

@sbc100 anything more you have to add here? It would simplify things on the Rust side a lot if linking issues could be fixed on the Emscripten side...

[sbc100]

I would say that executable comes in two flavors, normal and MAIN_MODULE. You can think of them like -fno-pie and -fpie. Only the MAIN_MODULE / -fpie form can link against dynamic library. Normal executables are fully static.

[hoodmane]

Well there are four executable LinkOutputKinds: DynamicNoPicExe, DynamicPicExe, StaticNoPicExe, and StaticPicExe. So it seems like we could pass -sMAIN_MODULE=1 (or 2?) for DynamicPicExe and StaticPicExe.

[petrochenkov]

Well there are four executable LinkOutputKinds

Technically it's 5, there's also LinkOutputKind::WasiReactorExe, if there are any analogues in Emscripten, then it can also be potentially reused.

[hoodmane]

I think linking executables with dynamic linking enabled cannot work right now without fixes from Emscripten. We can't use -sMAIN_MODULE=1 because that passes -Wl,--whole-archive which breaks on Rust libraries because they include the lib.rmeta file. And we can't use -sMAIN_MODULE=2 because that requires us to calculate what exports are needed but Rust has no way to calculate this without extra help.

[bjorn3]

So it seems like we could pass -sMAIN_MODULE=1 (or 2?) for DynamicPicExe and StaticPicExe.

I don't think passing it for the *Pic* variants makes sense. A non-pic executable can still be dynamically linked and use dlopen. I think it should rather be done for the Dynamic* variants. Then toggling -sMAIN_MODULE=1 would be a matter of -Ctarget-feature=+/-crt-static which is also how non-emscripten targets effectively determine if dynamic linking and dlopen are supported.

[hoodmane]

Yes that sounds right.

[hoodmane]

@sbc100 how does it look to you?

[hoodmane]

@petrochenkov @compiler-errors could someone review this? Or if you don't have time, do you have any recommendations for other people I might ask?

[JohnTitor]

Failed in rollup: #99305 (comment)
@bors r-

[hoodmane]

@petrochenkov I cherry-picked the commit from #99243 to this branch because tests won't pass on this branch without it. Should I squash this too?

[petrochenkov]

No, it's fine as is.
@bors r+

[bors]

📌 Commit b0da01f has been approved by petrochenkov

It is now in the queue for this repository.

[ehuss]

@bors r-

Failed in rollup: #99776 (comment)

[JohnCSimon]

ping from triage:
@hoodmane can you post your the status of this PR?
Thanks

FYI: when a PR is ready for review, send a message containing
@rustbot ready to switch to S-waiting-on-review so the PR is in the reviewer's backlog.

[hoodmane]

This problem is fixed now:
emscripten-core/emscripten#17328
so I should remove it.

[hoodmane]

I think there's still a test failure that I don't understand. I will try to look into it again soon.

[JohnCSimon]

@hoodmane
Ping from triage: I'm closing this due to inactivity, Please reopen when you are ready to continue with this.
Note: if you do please open the PR BEFORE you push to it, else you won't be able to reopen - this is a quirk of github.
Thanks for your contribution.

@rustbot label: +S-inactive