Init support for local WASM plugins #3349

emcfarlane · 2024-09-27T22:18:29Z

This PR adds support for running WASM based plugins. Declare a plugin with the suffix .wasm to invoke the plugin using a WASM runtime. For example the plugin buf-plugin-field-option-safe-for-ml can be compiled with GOOS=wasip1 GOARCH=wasm and added to your buf.yaml config:

version: v2
plugins:
  - plugin: buf-plugin-field-option-safe-for-ml.wasm

github-actions · 2024-09-27T22:18:48Z

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Oct 2, 2024, 4:07 AM

emcfarlane · 2024-10-01T14:10:53Z

private/usage/usage_unix.go

@@ -12,8 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-//go:build aix || darwin || dragonfly || freebsd || (js && wasm) || linux || netbsd || openbsd || solaris
-// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris
+//go:build unix || wasip1 || js


Use the unix build constraint and drop support for +build as that is required for go versions 1.16 and earlier. Fixes builds for GOOS=wasip1

bufdev · 2024-10-01T19:45:41Z

private/buf/bufcli/cache.go

@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er
 	)
 }

+// CreatePluginCacheDir creates the cache directory for plugins.
+//
+// This is used by the [bufwasm.WithLocalCacheDir] option.


Is the [] something that Godoc picks up? If not, we don't use it anywhere else. If so, good to know!

bufdev · 2024-10-01T19:49:07Z

private/buf/bufcli/cache.go

@@ -103,6 +103,10 @@ var (
 	//
 	// Normalized.
 	v3CacheModuleLockRelDirPath = normalpath.Join("v3", "modulelocks")
+	// v3CachePluginsRelDirPath is the relative path to the plugins cache directory in its newest iteration.


This documentation isn't really accurate. If you dive down into the code, it appears you are passing this to the bufwasm.Runtime, which then happens to use this as a cache for some WASM compilation that is happening. Given the naming, and the documentation here, I would expect this to be a cache of downloaded plugins from the BSR, which it is not. This is effectively an implementation-specific cache.

This should be v3/wasmruntime or the like, whatever makes sense, and the documentation should reflect what this is specifically used for.

bufdev · 2024-10-01T19:49:34Z

private/buf/bufcli/cache.go

@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er
 	)
 }

+// CreatePluginCacheDir creates the cache directory for plugins.


This doesn't seem correct. See above.

bufdev · 2024-10-01T19:51:17Z

private/bufpkg/bufcheck/multi_client.go

@@ -94,6 +95,7 @@ func (c *multiClient) Check(ctx context.Context, request check.Request) ([]*anno
 		jobs = append(
 			jobs,
 			func(ctx context.Context) error {
+				start := time.Now()


Such timings are the responsibilities of tracing.Tracers

bufdev · 2024-10-01T19:51:57Z

private/bufpkg/bufpluginrunner/bufpluginrunner.go

+	"pluginrpc.com/pluginrpc"
+)
+
+// RunnerProvider provides pluginrpc.Runners for a given plugin config.


PluginConfig

bufdev · 2024-10-01T19:54:43Z

private/bufpkg/bufpluginrunner/provider.go

+}
+
+// wasmRunner is a runner that loads a Wasm plugin.
+type wasmRunner struct {


Why is this in this file?

bufdev · 2024-10-01T19:56:32Z

private/bufpkg/bufwasm/bufwasm.go

@@ -0,0 +1,81 @@
+// Copyright 2020-2024 Buf Technologies, Inc.


Why is this in bufpkg? It uses nothing buf-specific, and therefore should be in pkg/wasm.

bufdev · 2024-10-01T19:56:55Z

private/bufpkg/bufwasm/bufwasm.go

+	"pluginrpc.com/pluginrpc"
+)
+
+// Plugin is a Wasm module.


If Plugin is a WASM module, why is it named Plugin, especially given that plugins have a specific meaning in buf?

bufdev · 2024-10-01T19:57:22Z

private/bufpkg/bufwasm/bufwasm.go

+}
+
+// NewUnimplementedRuntime returns a new unimplemented Runtime.
+func NewUnimplementedRuntime() Runtime {


This can be var UnimplementedRuntime = unimplementedRuntime{}

bufdev · 2024-10-01T19:58:12Z

private/bufpkg/bufpluginrunner/provider.go

+		if r.wasmRuntime == nil {
+			return nil, syserror.Newf("wasm runtime is required for local wasm plugins")
+		}
+		return newWasmRunner(


It's weird that this is implemented in bufpluginrunner while the command.Runner-based Runner is implemented in pluginrpcutil. Why the split? I don't see any reason for one, both Runners should be implemented in pluginrpcutil.

They can't be implemented in pluginrpcutil as the pkg now depends on bufconfig for the bufconfig.PluginConfig. Could move pluginrpcutil under bufpkg?

the wasmRunner in no way depends on bufconfig.PluginConfig, I'm not sure what you are talking about.

Referring to the pluginrpcutil. All good on moving the wasm runner.

Init support for WASM plugins

6cd7178

emcfarlane requested a review from doriable September 27, 2024 22:18

emcfarlane added 5 commits September 30, 2024 14:12

Remove custom look path

a17f1c1

Fix test

58c9e6f

Fix test

ccdb801

Fix test

1fb09fe

Use Wasm abbr

4084811

emcfarlane commented Oct 1, 2024

View reviewed changes

emcfarlane added 2 commits October 1, 2024 10:14

Fix description

cc42495

Move RunnerProvider to bufpluginrunner pkg

5cd84ca

bufdev reviewed Oct 1, 2024

View reviewed changes

Rename Plugin to CompiledModule

0b4321b

emcfarlane marked this pull request as ready for review October 1, 2024 22:43

emcfarlane marked this pull request as draft October 2, 2024 00:25

emcfarlane added 15 commits October 1, 2024 20:40

Use tracer

ce2c42a

Unimplemented as var

8ef1070

Move wasm pkg

fa26987

move to bufpluginrpcutil

9429c87

Fixes

9634e9d

use wasmruntime cache dir

3498241

doc RunnerProvider config

767897e

fix wasm args

a69da63

cleanup wasm module docs

87676c0

fix bufconfig

e42f48d

fix naming

88ffcb7

fix banimports

85c7fbc

Merge branch 'main' into ed/pluginWASM

6a45022

fix pluginrpcutil

41f8127

fix doc

2be37aa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Init support for local WASM plugins #3349

Init support for local WASM plugins #3349

emcfarlane commented Sep 27, 2024

github-actions bot commented Sep 27, 2024 •

edited

Loading

emcfarlane Oct 1, 2024 •

edited

Loading

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

bufdev Oct 1, 2024

emcfarlane Oct 1, 2024

bufdev Oct 2, 2024

emcfarlane Oct 2, 2024

		@@ -0,0 +1,81 @@
		// Copyright 2020-2024 Buf Technologies, Inc.

Init support for local WASM plugins #3349

Are you sure you want to change the base?

Init support for local WASM plugins #3349

Conversation

emcfarlane commented Sep 27, 2024

github-actions bot commented Sep 27, 2024 • edited Loading

emcfarlane Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 27, 2024 •

edited

Loading

emcfarlane Oct 1, 2024 •

edited

Loading