Plans about WebAssembly

[sergei-kucher]

Do you have any plans about WebAssembly support?

[vendethiel]

LLVM already has a branch for that, so Crystal should be covered already ;)

[asterite]

Maybe when there's a WebAssembly specification we can start discussing this, but right now Crystal is more geared towards server and desktop applications, not the web. I'll close this until then.

[Svenskunganka]

A year later, now that a spec is in the works I figured that we could open discussion for this again? Chrome Canary and Firefox Nightly has experimental support for WebAssembly.

@asterite I understand that this is probably not a priority right now - but is this something you guys might be interested in the future?

[lbguilherme]

I believe there is not much to be done from crystal side until both browsers and the llvm target get mature enough. Clang is not ready for this yet.

We can for sure make a proof of concept that works, but not maintain it or port the whole stdlib to work on it yet. I suggest creating a milestone "post-1.0" and leaving this issue open there.

[vendethiel]

Clang is not ready for this yet.

Not sure how Clang is related to Crystal? Once it gets down to LLVM IR, does it matter?

[asterite]

@Svenskunganka I guess so, if you could write Crystal to target the browser it would be awesome. I know nothing about WebAseembly, and just a bit about asm.js, but I don't know what kind of things you can do... dom manipulation? Or just games? In any case, I think we should wait a bit.

[lbguilherme]

@vendethiel It is not in any way related or needed for crystal to work. But it is the one compiler that is working most towards targeting wasm. In fact, this is the goal of the first release of wasm: Have clang targeting wasm and running it in multiple browsers. It is safer and less work for us to let clang test the ground, solve most bugs, and get there before us.

@asterite It let's you access all standard web apis thought some kind of "LibJS", that includes DOM manipulation and everything else. All the stdlib (libc included, I think) would have to be implemented on top of that.

[jwaldrip]

@waj It may be a time to revisit/reopen this issue.

[RX14]

@jwaldrip why?

[pannous]

because the standards are now fixed and the toolchain is working nicely.
full support in most browsers and node.js

There is still a big opportunity to become the language of the binary Internet, racing against c++, rust and typescript.

[Svenskunganka]

Today WebAssembly is available in all the major stable browser releases, however its features are limited. It currently has no way to access the DOM or the Web APIs except if you're using JavaScript as the middle man. The WebAssembly Roadmap looks very promising however. Planned features are to allow wasm to directly access and modify the DOM, and call Web APIs without needing any JS at all.

Wasm has an (experimental) LLVM backend, and since Crystal uses LLVM I can imagine it would ease the work required to compile to wasm. Wasm post-MVP roadmap shows plans to implement access to low-level GC, however I'm not sure how this would interact with the Boehm GC that Crystal currently uses. I imagine that a new GC would have to be built specifically for wasm, or that the Boehm GC is adjusted to support the wasm "runtime". But this is only speculation, I'm not knowledgeable enough in that area to draw a conclusion.

Looking at these in progress features, the possibilities are very promising. Being able to compile to wasm would allow the community to create "isomorphic, fullstack" Meteor-like frameworks that shares the same code on both server & client (browser). To simplify, it would allow Crystal to do everything that JS currently does, like having a React-like library with full server-side rendering support. Other examples would be a Slack alternative that compiles to both the browser and regular platforms like Linux, Windows and Mac. I would imagine that Wasm would also be available on smartphones like iOS and Android through WKWebView and the like, so that Slack client could perhaps also be able to run on those platforms using the Wasm code.

It's worth mentioning though that Wasm will not be able to do things that JS currently cannot, like opening a raw TCP socket. We would have to use the alternatives the Web APIs provide like WebSocket and WebRTC.

[drhuffman12]

:+1

[akzhan]

@ysbaddaden what about wasm LLVM target? looks like it's a time to do.

[rhysd]

As long as I know, LLVM 4.0.0 does not support wasm. LLVM HEAD already supports it. So it may be available in 4.0.1 but I'm not sure. I remember LLVM is now in semantic versioning. So I guess it comes with 5.0.0 for stable release.

[ysbaddaden]

Same as other arch's: LLVM does the underlying work, that is generate the binary code (it's awesome), but there is far more to Crystal than just the language: it's library.

Without an official libc (the musl port is unofficial and experimental, advised not to use), we're left with the same state than embedded systems: most of the core and stdlib won't be available, the GC will likely not work, ... not even thinking of the fiber context switches ASM (cold sweat).

WASM may be fun, but it's far off the Crystal roadmap. If someone is courageous (crazy?) enough to work on this, please try, you'll have fun (maybe) :-)

[pannous]

isn't libc provided by emscripten? how else would it run hello_world.c ?

Quote:

Emscripten lets you "link" with JavaScript libraries to do useful things, like render WebGL, run a browser main loop, handle a filesystem, provide bindings to JS so it's easy to call into your compiled code, etc., as well as a full set of compiled libraries like libc and so forth.'

GC can be added later.

[pannous]

Here is how rust does it https://github.com/brson/mir2wasm/

[RX14]

@pannous rust doesn't need a gc.

If someone can demonstrate libgc compiled and working on wasm then this endeavour becomes actually credible.

[pannous]

can't we start without GC? should be fine for demo code which doesn't run for extended periods of time?

[akzhan]

BDWGC on Emscripten (LLVM to WebAssembly backend) now fails on MacOS hosts (but should be OK on others - ivmai/bdwgc#163 (comment)).

I suppose it will be fixed with ivmai/bdwgc#163.

[RX14]

So it looks like wasm doesn't have a normal stack at all. I have no idea how we would implement fibers in such an architecture. The note on the GC issue saying that the GC can't walk the stack to find roots is basically a show stopper too. Implementing crystal on wasm looks like it'd require a great deal of thought.

[pannous]

yep, no GC and fibres/threads in first version:
http://webassembly.org/docs/future-features/

I'm fine with those crystal features missing in the first adaptation.

[akzhan]

Looks like Crystal cannot be provided until EH, GC and coroutines landed to WebAssembly.

Just wait for these features (they will be added to browsers fast, I suppose).

[RX14]

@pannous I doubt most people would be OK with that or call it a "complete" crystal port so I don't think it's worth the effort at this point in crystal's development. If you or anyone else wishes to work on it i'm sure the core team would be happy to help or merge PRs.

[RX14]

@akzhan I doubt fibers are impossible in webassembly as-is or would get any kind of special support in the future. It's just a matter of finding out how to do it. GC and direct DOM access are things we should wait for.

[catmando]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know. I think it would have to be a subset without fibers initially. My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler. I am guessing the major problem is that you have to map crystal class structures, method calls, etc -> JS equivilents, rather than trying to do some lower level emulation. This is what opal does.

[vendethiel]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know.

wouldn't it be easier to find a way to get a GC/wait for GC?

My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler.

maybe parts of the parser can be reused, but i don't see anything else

[badlee]

👍 +1

[sam0x17]

This provides some very useful and up-to-date high-level information on the web assembly feature set and roadmap:

https://hacks.mozilla.org/2018/10/webassemblys-post-mvp-future/

@straight-shoota @RX14

[bew]

Interesting read about threading implementation for wasm: https://rustwasm.github.io/2018/10/24/multithreading-rust-and-wasm.html
There might be some useful things for us ;)

[parruda]

I just stumbled upon this issue. I’ve been thinking about adding Opal support to FaaStRuby, and if we could compile Crystal to Wasm... Full stack Crystal development in a serverless platform. Think about the possibilities!

I wish I could contribute to this effort, but I don’t know enough about Wasm or LLVM. But you guys can be sure I would contribute to make it popular by pushing the support to the platform as soon as you guys release it!

Thanks for the hard work.

[sam0x17]

Yeah, I imagine this is going to be one of the first big pushes after 1.0, and by then other languages will have figured out some of the do's and dont's of web assembly support, so it will be that much easier

[catmando]

FYI: I just put a discussion topic around this general topic: https://forum.crystal-lang.org/t/crystal-js-transpiler/903

[theganyo]

WASM will soon be very important on the server as Envoy is testing support for dynamic filters via WASM and Istio will be fully supporting that soon. That should surely add some heat to this discussion.

[jan-zajic]

LLVM WebAssembly target is production ready since LLVM 8.0.0. See http://releases.llvm.org/8.0.0/docs/ReleaseNotes.html#changes-to-the-webassembly-target .

[jwoertink]

Just keeping the talk going, this article was just released https://www.w3.org/blog/news/archives/8123. I think why this is important is because just like HTML, CSS, and JavaScript, WASM will be a web standard with the W3C backing it. I know with all the stuff this language still needs, this may be a little lower priority, but I think getting a "rough" start merged, that would allow us to really hash out how things should work. We could start with a preview or something like how the MT stuff is going, or just get some initial things like the windows support is doing.

Edit: Here's a spec link https://webassembly.github.io/spec/core/index.html

[RX14]

A "Hello world" for this feature would be 90% of the work :)

Keep your expectations low, this is about as hard as asking for the compiler to compile to JS directly.

[richardanaya]

Any updates on this? A lot of these comments sound like people are hoping for too much to be done at once. Just figure out how to talk with imported and exported functions, don't worry about supporting a standard library, or figuring out magical ways to talk to the dom (libraries can figure this out).

[jwoertink]

I was able to get a "hello" world from crystal done. Though, not in the way you'd want it. I basically just hand wrote bytes 😂 I did come across this gist showing how to compile LLVM with wasm support. I tried to see if I could get crystal compiled this way, but my eyes glossed over. It's a bit beyond my knowledge. Maybe someone else here can point out some stuff? My assumption is that if you can compile crystal with the latest LLVM, and add the right flags, maybe it'll just work™

[jwoertink]

Here's what I did https://gist.github.com/jwoertink/b8fb8430ed819dd4041a56ff6ed3ef64 if anyone is interested in expanding on that, or whatever.

[epipheus]

@jwaldrip why?

Well heres one example, Crystal should be the language of choice for blockchain but isn't. The fastest way to build secure blockchain base anything right now is via Substrate/Polkadot and pretty much any language that can compile to wasm can easily leverage Substrate/Polkadot to get started. Among many things it's Wasm that allows Polkadot to have forkless upgrades -- that's significant. Instead of using Crystal, folks are just deciding to pick up Rust. Crystal would be far better.

That's one example.

[pdpark]

WASM will soon be very important on the server as Envoy is testing support for dynamic filters via WASM and Istio will be fully supporting that soon. That should surely add some heat to this discussion.

Another example of a great use-case - transforming data on streaming platforms: https://thenewstack.io/webassembly-brings-easy-inline-data-transformations-to-redpanda-kafka-streaming-platform/

<< Adding functionality to an existing system is exactly the kind of place where WebAssembly is going to be useful, the Cloud Native Computing Foundation‘s Chief Technology Officer Chris Aniszczyk agreed. “I think we’ll see any project that has an extension-type mechanism will probably take advantage of WASM to go about doing that.” >>

[lbguilherme]

Here is a summary of what needs to happen for a WebAssembly target:

• Garbage collection: bdwgc can be compiled to WebAssembly with Emscripten, but it needs special care with the stack. In WebAssembly the stack doesn't live in linear memory and cannot be scanned by a GC as we do everywhere else. The only plausible solution would be to keep a live copy of the stack in linear memory. Or maybe emulate the stack using a heap array. This is tricky and requires lots of codegen changes.
• Exceptions: Again, there is no primitive to walk the stack or unwind it. One way to produce a useful stack trace for debugging would be to call into JavaScript and use new Error().stack there, which will include the WebAssembly stack (this isn't an option from WASI outside a browser). Another solution involves modifying every function to push its own name in a heap stack and pop it on exit. Again, involves codegen changes and will cost performance. For unwind, it needs to be implemented "by hand", having every function return a special value then it raises an exception, and having each caller check it. Pretty much like Go exceptions.
• Fibers: There is just no way to do context switching with multiple stacks. The solution Go uses for its Goroutines is a huge hack: Every function begins with a big jump list for all statements on that function inside a while. Control flow (ifs, loops, etc) is made by modifying the jump control variable. Those jump positions are stored in a heap stack. When Go needs to switch to another Goroutine it makes the current function return a special value, that trigger ALL functions to return as well. Then the initial function of the other Goroutine is called and it jumps to the exact position by repeating the same steps. This requires the value stack to be on the heap as well. (ref. proposal: wasm: use Asyncify for switching goroutines golang/go#43033)

There are some WebAssembly proposals that would help a lot here:

• Proposal: Stack Walking WebAssembly/design#1340
• Please Support Arbitrary Labels and Gotos. WebAssembly/design#796
• Typed continuations to model stacks WebAssembly/design#1359

What Go managed to do is just impressive, but it has quite a few performance issues. I don't think it is viable for us to go the same route just yet.

[jwoertink]

Wow! That's some great information @lbguilherme. Thanks for putting that together. It's much easier to see where we're at and where we need to get to.

[straight-shoota]

There are plans for including a GC into WebAssembly. It would probably be better to use such a native GC instead of hacking bdwgc into WASM.

[RX14]

There are plans for including a GC into WebAssembly. It would probably be better to use such a native GC instead of hacking bdwgc into WASM.

Whichever of stack walking or GC is available first would be good targets.

There are some WebAssembly proposals that would help a lot here

The "arbitrary labels and gotos" isn't important here, since LLVM takes care of relooping. Stack walking and stack switching are important to support fibers, but it'd be neat to support basic crystal without the stdlib before then.

[straight-shoota]

Basic basic Crystal (no stdlib, no exceptions) should already be feasible, right?

[lbguilherme]

Basic Crystal is indeed feasible.

#10870

[lbguilherme]

After some research, I concluded there are viable ways to implement the missing features with the current WebAssembly spec thanks to Binaryen. In particular, it has a pass called "Asyncify" that will take WebAssembly as input and apply a series of static transformations to it, generating WebAssembly out. The transformations modify the control flow of each function so that they can unwind the stack by pushing every local to a heap array, together with some information about where exactly execution was. And it adds a "rewind" strategy as well, consuming data from this heap array and refilling the stack to go back to the same place as before. This method is similar to what Go implemented for itself.

Sources with a documentation comment: https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp
A nice blog post about it: https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html

• Fibers: This is exactly what we need for stack switching. Unwind the current stack and then rewind a previous stack, with each paused fiber keeping its own stack saved for later. This has a performance penalty since the switch isn't constant time like every other platform, instead, it depends on the stack size (that should be small anyway).
• GC: We can use bdwgc's GC_set_on_collection_event to notify us before each GC collect (GC_EVENT_MARK_START..GC_EVENT_MARK_END). We can unwind the current stack and let the GC run without anything at all on the stack currently. After it finishes marking we can rewind the same Fiber back in place.
• Stack trace (for exceptions): We can unwind the current stack and then rewind it again. Then read and interpret data from the heap array generated in this process. The exact format needs to be well understood for that, but it should provide enough information. Debug information about function names, files, and lines can be included as auxiliary data later on.
• Exceptions: Maybe we can unwind until a suitable begin..rescue..ensure..end is found? This part is still unclear.

[maxfierke]

After some research, I concluded there are viable ways to implement the missing features with the current WebAssembly spec thanks to Binaryen. In particular, it has a pass called "Asyncify" that will take WebAssembly as input and apply a series of static transformations to it, generating WebAssembly out. The transformations modify the control flow of each function so that they can unwind the stack by pushing every local to a heap array, together with some information about where exactly execution was. And it adds a "rewind" strategy as well, consuming data from this heap array and refilling the stack to go back to the same place as before. This method is similar to what Go implemented for itself.

Sources with a documentation comment: https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp
A nice blog post about it: https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html

* Fibers: This is exactly what we need for stack switching. Unwind the current stack and then rewind a previous stack, with each paused fiber keeping its own stack saved for later. This has a performance penalty since the switch isn't constant time like every other platform, instead, it depends on the stack size (that should be small anyway).

* GC: We can use bdwgc's `GC_set_on_collection_event` to notify us before each GC collect (`GC_EVENT_MARK_START`..`GC_EVENT_MARK_END`). We can unwind the current stack and let the GC run without anything at all on the stack currently. After it finishes marking we can rewind the same Fiber back in place.

* Stack trace (for exceptions): We can unwind the current stack and then rewind it again. Then read and interpret data from the heap array generated in this process. The exact format needs to be well understood for that, but it should provide enough information. Debug information about function names, files, and lines can be included as auxiliary data later on.

* Exceptions: Maybe we can unwind until a suitable begin..rescue..ensure..end is found? This part is still unclear.

I have a few WIP branches in various stages from my previous WASM porting attempts that may be of interest:

• Asyncify: maxfierke@96bf6bd (I think this might be close)
• Module imports: maxfierke@a9cefc9 (This totally works, would probably apply pretty cleanly. Needed for the asyncify import)
• Exception handling: https://github.com/maxfierke/crystal/tree/mf-spike_wasm_eh (LLVM bits have shifted since, definitely broken but might point in the right direction)

Most recent attempt: https://github.com/maxfierke/crystal/commits/mf-spike_wasm_take_two w/ full wasi-libc bindings for further diving, if anyone is interested.

Various notes here: https://github.com/maxfierke/crystal-wasm-tools

These have drifted from upstream a bit, but I think there might be some things still useful there for your work @lbguilherme. If #10870 gets merged, I would be happy to jump back in on WASM support given the good basis (much more organized than my spike branches 😅 )

[pannous]

Looks like #10870 was merged, maybe WASM integration can proceed?

[lbguilherme]

I believe this issue is superseded by #12002. Keep in mind that progress on this will most likely move slowly until there is more community adoption/interest.