-
Notifications
You must be signed in to change notification settings - Fork 331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compilation to WASM? #722
Comments
I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata. That way we should be able to build a rustc which doesn't depend on llvm or other C code. |
I had created one in the past but it bitrotted and it was unused, so I removed it in rust-lang/rust#58847. If you copy https://github.com/bjorn3/rustc_codegen_cranelift/blob/11d816c/src/lib.rs#L182-L190 into Another thing necessary is replacing the |
I want to do the same for https://github.com/bjorn3/rustc_codegen_cranelift/, but I want it to pass the rustc test suite first and |
Intriguing. :) I should add one warning though: Miri isn't a fast interpreter. It's really slow. So I don't think it is actually a good environment to use to run code, I see it as more useful for debugging and testing. But, don't let me stop you! I just felt I should give you a fair warning. And if ideas like this leak to people making Miri lightning fast while maintaining all the UB checking, I'll be even more happier. :D |
That's understandable, but I think it's ought to be good enough for typical playground snippets :) |
I guess that's one way, although I was wondering if Miri actually needs the main |
The codegen backend is necessary for |
Fair enough. |
I am currently trying to compile rustc for wasm (https://github.com/bjorn3/rust/tree/compile_rustc_for_wasm), but I am hitting a compiler bug: rust-lang/rust#60540. |
@bjorn3 I've rebased your branch onto master, updated deps and fixed cfg's from Eventually it compiled successfully, but then ran into the same runtime validation issue with invalid code generated by Rust. However, I recompiled in release mode and then it passed validation! That got me thinking it should work now, but running the generated file with |
@bjorn3 Oh... maybe it's just been taking so long (especially the compilation part). I've tried
|
Yes, it takes several minutes to compile it using
🎉 🎉 🎉
I tried actually compiling something, but it errors with:
I am currently trying to figure out were it errors. |
Places needing patching:
Edit: pushed bjorn3/rust@15f980f (based on @RReverser's branch). Now it errors at
Which is expected, as I had to remove the codegen backend dynamic loader. Will try to get https://github.com/bjorn3/rustc_codegen_cranelift to work with it. |
This needs a rustc compiled for wasi (see rust-lang/miri#722) It also needs bytecodealliance/target-lexicon#14
FWIW previously (before even filing this issue) I tried compiling rustc with Emscripten instead, which should, in theory, reduce number of these places to patch, as it supports a bit more than WASI does. Haven't gotten too far though, because I tried to build completely unpatched rustc and there were few things that still didn't compile and probably needed similar fixes as in your branch.
I thought the plan was to build it without any codegen, just with miri? Or do you want to build an actual full rustc? |
I want them both. :) I currently have |
Yeah for that I think you'll need to do the proper build (via |
Seems like it doesn't even reach the rustc version check for the libraries. I added |
Switching from wasmer to wasmtime fixed it. It even got to the beginning of codegen. Edit: filled wasmerio/wasmer#434. |
I am currently working on making miri compile for wasi, which this issue was actually about. |
It seems to trap while calling the
I pushed the wip stuff to my branch. |
@bjorn3 Left a comment on your MIRI commit on your branch. |
@bjorn3 But also, I'm not sure why rustc is now depending on miri... shouldn't it be the other way around? (like in non-WASI version) |
I did that to be able to prevent having to recompile every rustc crate, which is slow and to prevent having to copy all files in the dir layout rustc wants a sysroot to be. |
I'm not sure I understand what you're saying... neither should be affected by which crate you compile as an entry point. I've changed my local copy of Rust & MIRI to do just that, and got |
I meant that I had already compiled all crates in |
For those who don't want to build it themself here is the version I compiled. Concatenate both files together to form a |
When I do that, I think it gets corrupted. I get this error trying to compile it:
This is how I concatenate it:
|
If you concatenate them you get an xzip compressed tarball. You need to extract this tarball to get a directory containing both rustc.wasm and the precompiled standard library that will be used by rustc. In case of browser_wasi_shim's rustc.html example you have to rename the dist directory to examples/wasm-rustc. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
@bjorn3 can you please help to write blog for this and achieve this step by step for new devs also? |
I don't have the time for that right now, but I will consider writing a blog post in the future. First I want to clean it up and fix a panic that happens after it already produced the object files. Also it can't produce webassembly modules right now. Only x86_64, aarch64 and s390x object files. But reporting errors and warnings does work just fine. Is that enough for you? |
I know its easy to forget, so from one open source dev to another, just a reminder that nothing was promised and anything you give will/should be appreciated. I'm probably just as excited/eager as @beingminimal to eventually read a blog post from you @bjorn3 (I also don't think I can get it running just from the info in this thread). But you should do it when you have time/interest. |
Hey bro, so once you complete this work we will be able to write, compile rust in browsers also and in vscode.dev also, what are your thiughts? |
Compiling to working executables that can run in the browser doesn't work with this without a Webassembly backend for Cranelift (or a port of LLVM to run on wasm), but the equivalent of |
Hey thanks a lot for the reply. I am not that much experienced otherwise i could have implemented a demo with these available things and have written blog also. But now it seems with your help it would be possible to achieve this. Awaiting your reply. Thanks |
when I have time I might make a poc vscode-wasm extension |
I have, its not hard cause its the same as the browser (unless VS Code has recently made changes to the API, which I doubt they would). |
I did another build of rustc + cg_clif for wasm32-wasi with the latest rustc version and a couple of bugs fixed. The source code can be found at https://github.com/bjorn3/rust/tree/compile_rustc_for_wasm11 and the pre-compiled binaries at https://github.com/bjorn3/wasm-rustc (see comments.txt in the source tree for how to build it yourself) I have also updated the browser_wasi_shim example in bjorn3/browser_wasi_shim@9289c4c. It is now able to reach the exact point where the linker would be invoked again, rather than panicking during linking preparation. Compiling to wasm is tracked by bytecodealliance/wasmtime#2566. |
Opened rust-lang/rust#120348 to upstream most of the build system changes, pushed rust-lang/rustc_codegen_cranelift@7d3b293 to upstream a cg_clif change and managed to somewhat reduce the diff on the compile_rustc_for_wasm13 branch. The second commit on that branch uses the wasm32-wasi-preview1-threads target to further reduce the diff, but requires wasi-threads, which browser_wasi_shim currently doesn't support. |
rust-lang/rust#120348 has been merged. Opened rust-lang/rust#121392 to make patching away dlopen usage easier. I rebased the compile_rustc_for_wasm13 branch and removed a couple of unnecessary changes. Edit: rust-lang/rust#121392 has been merged too. Rebased. |
I took https://github.com/bjorn3/browser_wasi_shim/blob/main/examples/rustc.html and made it invoke the miri I built from the second to last commit of the compile_rustc_for_wasm13 branch instead. |
For others trying to run it, thats
|
Somehow no one pasted this here: https://garriga.dev/rubri/
cc @LyonSyonII |
Miri maintainer note: this is a fun project, but not something we currently intend to support officially. To keep maintenance manageable, Miri only supports running on platforms that rustc supports running on.
Compiling the whole Rustc to WASM is a pretty big undertaking for many reasons.
However, Miri doesn't need an actual codegen and many other parts of the whole Rustc, so I wonder how realistic it would be to compile it and the pieces it depends on to WASM instead? Are there any obvious blockers?
Mostly opening this to gauge interest and estimate complexity, as I believe there is an interest in running Rust directly in the browser on playground-like websites.
P.S. Despite what I said in the first sentence, this was actually done for Clang a while ago - https://tbfleming.github.io/cib/ - which includes LLVM compiled to WASM that, in turn, generates more WASM dynamically during runtime. In theory, it should be possible to do the same for Rust, especially since they share LLVM, but for now having just an interpreter could already be an interesting starting goal.
The text was updated successfully, but these errors were encountered: