New webpage: How to add cross-platform plugin support to a Go program using WebAssembly.

[codefromthecrypt]

Is your feature request related to a problem? Please describe.
Many are interested in wazero as a way to add cross-platform plugin support to their applications. This topic covers too much ground for a one-liner or even a paragraph. We should have a web page on https://wazero.io for this.

Describe the solution you'd like
This should overview the general opportunities and constraints. For example, what's portable by default (eg can also run in something besides wazero ex WASI), and what you can define and choose to do. How wazero helps in general by being testable in go, etc.

It would mention that the program being extended can communicate with plugins in wasm via sandboxed memory, host callbacks, as well channels allowed in WASI (ex files).

It can overview the common stages done in plugin systems, such as an "init" which can setup reference data in memory for the plugin to use, as well some common vocabulary used in wasm plugin architectures (such as ABI, host, guest).

It would also have some section describing the impact of constrained languages, such as no_std in Rust and TinyGo, and some libraries like gjson and karmem, pointing to our allocation examples for common issues.

Finally, this would point to a new "baby plugin" example, which must be carefully done to be small, but outline the common things above.

Note: we'd have to be careful this doesn't turn into a book (at least immediately) as completing this is more important than covering all topics. We can link to some existing plugin systems at the end for reference.

Describe alternatives you've considered
We can defer entirely to documentation on other sites, like https://github.com/wapc/wapc-go or an example like https://github.com/tetratelabs/proxy-wasm-go-sdk. However, the basic ideas should be introduced first as plugins particularly in wasm have some nuance to them, which could be explained.

We could also make a presentation, and probably will, but a website page would be helpful anyway, and also could be translated as necessary.

Additional context
suggestion of mosn integration by @tanjianfeng
request in hackernews

[codefromthecrypt]

If anyone currently doing plugin stuff want to brain dump into here some thoughts, please do so! cc @pkedy @pims @anuraaga @mathetake @knqyf263 @inkeliz

I think one thing a lot of people are surprised about is you can't really program plugins via wasm exactly how you would do it in normal go. Ex it may look different, both in code and code arch, vs if it were pure go due to realities of language support and what wasm allows. It is also unclear how someone will know whether a language supports a 2.0 feature like SIMD or not. We probably would have some need to inventory a few FAQ maybe on another page or at least the bottom of this new one.

[skybrian]

A simple example might be CGI support. Originally, web servers supported dynamically generated web pages by launching a new process each time, sending parameters as environment variables, and sending stdout as the web page. Maybe that would be a way to get started, using API bindings that work in any language because they're based on very basic things like stdin, stdout, and environment variables?

But maybe people wanting to add plugin support to their programs don't particularly care about supporting multiple programming languages? They'll want to define a good plugin API for one language, at least at first, and maybe add more languages later if at all. Traditionally, this might be done using Tcl or Lua.

After all, you need to write documentation and examples about how to write plugins for your program, write tests to make sure they continue to work, and so on. Plugins are a big feature to support, and multiple-language plugins even more so, unless the plugin API is tiny.

For them, an advantage of WebAssembly would be runtime performance due to compiling plugins to native code, and a disadvantage is that you're on your own figuring out all the stuff that's specific to providing a good environment for each plugin language. It's a lot to take on.

But that might be beyond the scope of this project, too. Perhaps some direct consumers of a WebAssembly engine will be plugin language authors who generate higher-level API bindings for one language?

[pims]

@skybrian you've basically described what Wagi does: https://github.com/deislabs/wagi

I have an incomplete port of it to Go running on wazero, maybe one day I’ll get time to polish it and publish it.

It’s a great use case for Wasm/Wasi.

[gedw99]

and also spin
https://github.com/fermyon/spin
https://spin.fermyon.dev/

[codefromthecrypt]

ps I totally agree that an SDK approach seems to be the better plan, at least to start, with wasm plugin arch, I spoke about this last week in my talk literally only about experiences I had in wazero.

I feel like this issue may be hard to wrangle into a commit, but now that I have some focus time, I'll give it a think and look through how various folks are doing things.

[gedw99]

@skybrian you've basically described what Wagi does: https://github.com/deislabs/wagi

I have an incomplete port of it to Go running on wazero, maybe one day I’ll get time to polish it and publish it.

It’s a great use case for Wasm/Wasi.

@pims i would love to see this happen . I am writing wasm plugins and also live the old school std in / out approach .

i have been using the rust based wasi and spin , and would prefer a pure golang stack

[codefromthecrypt]

fwi due to various things, I expect to be able to focus again on this within 2 weeks unless someone else starts!

[codefromthecrypt]

I meant the doc, no promises on a cgi framework thing beyond an example.

[syke99]

I meant the doc, no promises on a cgi framework thing beyond an example.

Fwiw, I've recently released a library that wraps cgi.Serve() and allows you to write either individual route handlers, or an entire router that you also use the library to write. It was intended to simplify compiling modules to use with WAGI and provide you the option of handling your routing either through a modules.toml file, or through Go code. Idk if it would be of any help, (I've only very recently given Wazero a cursory look-over), but I'd be happy to help contribute by either merging my library into Wazero as a sub-package, or working along the Wazero team to use the two together.

[codefromthecrypt]

Thanks for the note @syke99. I'm pretty sure some have been at least curious about WAGI, so nice to hear about your project. Since the time this issue was created the ecosystem has blown up a bit, so I think merging plugin systems into the main repo is no longer a viable idea. However, a webpage makes sense, and probably more sense than before.

So, maybe what we can do is make a page that has a high level description of plugins, then links to a specific "where-to-next" minimally the GitHub repo. In the page somewhere we can say how to edit this (ex pull request) and ground rules for addition. For example,

• plugin models must have some kind of spec that's implementable
• host side must be open source and implemented at least once with wazero

We might also link our #wazero gophers channel in case some projects don't have a chat channel.

We can seed this with a few host side impls including below. In the table there could be a column for "spec" or closest to it.

• HTTP plugins
  • waggy - CGI or maybe WAGI?
  • http-wasm-host-go - http-wasm's website
• General purpose plugins
  • go-plugin - protobuf services
  • wapc-go - wapc's website
  • apex-go - apex's website

There are likely one or two I've forgotten, anyway main thing is structuring the page so that it is easy to update and see where to plug in info.

cc @pkedy @anuraaga @jcchavezs @knqyf263 for thoughts

[syke99]

No problem, thanks for the reply @codefromthecrypt!! I honestly wasn't expecting a reply and didn't even know if it would be helpful given the age of the issue, like you mentioned lol. But sure!! That would work for me as well. I have no qualms about maintaining Waggy as a stand-alone library. It's just had its v0.3.1 release a few minutes ago, but right now, it's WAGI-specific. However, there would only need to be a few small changes and updates made to support writing plug-ins in both CGI and WAGI simultaneously. If you'd like, I can comment back here updating whenever those changes have been released, or another issue could be opened. Both work for me on my end!

[codefromthecrypt]

@syke99 thanks for the feedback, and note I wasn't necessarily suggesting to do a CGI implementation, just I wasn't sure the canonical spec for WAGI as it seems to also refer to CGI ;)

[syke99]

@codefromthecrypt haha no worries!! It's something I've been thinking about doing anyways. It would make waggy more flexible and not only restricted to a WAGI environment lol 😄 but WAGI has some restrictions on, and also some additions to, CGI. But it wouldn't take hardly anything at all to adapt it to handle both!!

[gedw99]

speaking of community plugins…

Vugu compiles using tinygo. The wasm has a struct for binding and creating Vue html. It can do it in a browser and possibly one the server too ( gotta check ).

https://www.vugu.org/

https://github.com/vugu/vugu

https://github.com/orgs/vugu-examples/repositories

would be interesting if the same golang func can be hosted with Wazero.

[syke99]

@codefromthecrypt the pkg.go.dev documentation is taking a bit to catch up (though, I did verify the new release is deployed by attempting to go get it on a separate machine), but Waggy has just been bumped to v0.4.0 with support for a full CGI environment or running inside of a WAGI environment. So it should be ready to go!

[codefromthecrypt]

ps I think we won't be maintaining a sort of "awesome wasm" page, so I think some of the projects mentioned here, especially those that don't use wazero, could be better placed on one of them like https://github.com/mcuking/Awesome-WebAssembly-Applications

Things like frameworks, I think they should have their own lists which can handle guest sides more appropriately than wazero, which is host-side. I opened this for WAGI on that point deislabs/wagi#190

For here, I opened #1151 which will list tech that uses wazero to implement the host side of webassembly. Thanks for all the feedback, and even if not everything ends up in this repo, I hope the sharing has helped!