wasm: duplicate import in wasip1 compiled binary #60525

johanbrandhorst · 2023-05-30T22:29:16Z

What version of Go are you using (`go version`)?

$ go version
go version devel go1.21-993707a9d6 Tue May 30 20:19:54 2023 +0000 linux/arm64

What did you do?

package main

import (
	"crypto/rand"
	"encoding/hex"
	"fmt"
)

func main() {
	b := [16]byte{}
	rand.Read(b[:])
	fmt.Println(hex.EncodeToString(b[:]))
}

GOARCH=wasm GOOS=wasip1 go build -o main.wasm main.go

wasm2wat main.wasm

The imports in the generated .wat file are:

  (import "wasi_snapshot_preview1" "sched_yield" (func (;0;) (type 1)))
  (import "wasi_snapshot_preview1" "proc_exit" (func (;1;) (type 2)))
  (import "wasi_snapshot_preview1" "args_get" (func (;2;) (type 3)))
  (import "wasi_snapshot_preview1" "args_sizes_get" (func (;3;) (type 3)))
  (import "wasi_snapshot_preview1" "clock_time_get" (func (;4;) (type 4)))
  (import "wasi_snapshot_preview1" "environ_get" (func (;5;) (type 3)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;6;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;7;) (type 5)))
  (import "wasi_snapshot_preview1" "random_get" (func (;8;) (type 3)))
  (import "wasi_snapshot_preview1" "poll_oneoff" (func (;9;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_close" (func (;10;) (type 0)))
  (import "wasi_snapshot_preview1" "fd_read" (func (;11;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;12;) (type 5)))
  (import "wasi_snapshot_preview1" "random_get" (func (;13;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_fdstat_get" (func (;14;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_fdstat_set_flags" (func (;15;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_get" (func (;16;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_dir_name" (func (;17;) (type 6)))

Both fd_write and random_get appear twice.

What did you expect to see?

Each import appears only once

This isn't an invalid wasm file and doesn't affect the end result much, but there is clearly some part of the compiler that isn't weeding out duplicate imports for wasm.

The text was updated successfully, but these errors were encountered:

HarikrishnanBalagopal · 2023-08-15T09:08:32Z

Also happening for me with this code

The duplicate imports for fd_write in the compiled wasm (disassembled with wasm2wat)

(module
  (type (;0;) (func (param i32) (result i32)))
  (type (;1;) (func (result i32)))
  (type (;2;) (func (param i32)))
  (type (;3;) (func (param i32 i32) (result i32)))
  (type (;4;) (func (param i32 i64 i32) (result i32)))
  (type (;5;) (func (param i32 i32 i32 i32) (result i32)))
  (type (;6;) (func (param i32 i32 i32 i64 i32) (result i32)))
  (type (;7;) (func (param i32 i32 i32 i32 i32) (result i32)))
  (type (;8;) (func (param i32 i32 i32 i32 i32 i64 i64 i32 i32) (result i32)))
  (type (;9;) (func (param i32 i32 i32) (result i32)))
  (type (;10;) (func (param i64 i64 i64 i64) (result i64)))
  (type (;11;) (func (param i64 i64 i64) (result i64)))
  (type (;12;) (func (param i64) (result i64)))
  (type (;13;) (func (result i64)))
  (type (;14;) (func))
  (type (;15;) (func (param i64 i64) (result i64)))
  (type (;16;) (func (param f64) (result i64)))
  (import "wasi_snapshot_preview1" "sched_yield" (func (;0;) (type 1)))
  (import "wasi_snapshot_preview1" "proc_exit" (func (;1;) (type 2)))
  (import "wasi_snapshot_preview1" "args_get" (func (;2;) (type 3)))
  (import "wasi_snapshot_preview1" "args_sizes_get" (func (;3;) (type 3)))
  (import "wasi_snapshot_preview1" "clock_time_get" (func (;4;) (type 4)))
  (import "wasi_snapshot_preview1" "environ_get" (func (;5;) (type 3)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;6;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;7;) (type 5)))
  (import "wasi_snapshot_preview1" "random_get" (func (;8;) (type 3)))
  (import "wasi_snapshot_preview1" "poll_oneoff" (func (;9;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_close" (func (;10;) (type 0)))
  (import "wasi_snapshot_preview1" "fd_read" (func (;11;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_readdir" (func (;12;) (type 6)))
  (import "wasi_snapshot_preview1" "fd_filestat_get" (func (;13;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;14;) (type 5)))
  (import "wasi_snapshot_preview1" "path_filestat_get" (func (;15;) (type 7)))
  (import "wasi_snapshot_preview1" "path_open" (func (;16;) (type 8)))
  (import "wasi_snapshot_preview1" "fd_fdstat_get" (func (;17;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_fdstat_set_flags" (func (;18;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_get" (func (;19;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_dir_name" (func (;20;) (type 9)))
  (func $go_buildid (type 0) (param i32) (result i32)

Compiled with

$ go version
go version go1.21.0 darwin/amd64
$ CGO_ENABLED=0 GOOS=wasip1 GOARCH=wasm go build -o bin/test.wasm .
$ echo $?
0

The Golang code

package main

import (
	"os"

	"github.com/sirupsen/logrus"
)

func must(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	logrus.Infof("start")
	pwd, err := os.Getwd()
	logrus.Infof("pwd: '%s'", pwd)
	must(err)
	fs, err := os.ReadDir(".")
	must(err)
	for _, f := range fs {
		logrus.Infof("f: %+v", f)
	}
	b, err := os.ReadFile("./dep.json")
	must(err)
	logrus.Infof("The contents are:\n%s", string(b))
	logrus.Infof("done")
}

The go.mod file

module foo.com/b

go 1.21

require github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af

require golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect

The go.sum file

github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af h1:Sp5TG9f7K39yfB+If0vjp97vuT74F72r8hfRpP8jLU0=
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

elewis787 · 2024-11-19T19:55:01Z

This is likely known, but it appears to be related to the double import definition between os_wasip1.go and fs_wasip1.go.

go/src/runtime/os_wasip1.go

Line 74 in d90ce58

    
           func fd_write(fd int32, iovs unsafe.Pointer, iovsLen size, nwritten *size) errno

go/src/syscall/fs_wasip1.go

Line 225 in d90ce58

func fd_write(fd int32, iovs *iovec, iovsLen size, nwritten *size) Errno

go/src/runtime/os_wasip1.go

Line 78 in d90ce58

func random_get(buf *byte, bufLen size) errno

go/src/syscall/fs_wasip1.go

Line 273 in d90ce58

func random_get(buf *byte, bufLen size) Errno

This surfaced for me when trying to adapt a wasip1 go module to a waspi2 component. There are likely other challenges to get to a p2 component but this was the first error.

johanbrandhorst · 2024-11-19T20:04:11Z

Good spot, we might be able to fix that by having the runtime call the syscall package import instead.

elewis787 · 2024-11-19T22:29:03Z

I have a fix for this by using go linkname and having syscall calling the runtime fs write.

This felt backwards but was a pattern I saw in exec_libc.go

I can push a PR for reference.

johanbrandhorst · 2024-11-19T22:30:29Z

Using linkname between runtime and syscall is a time honored tradition. Please do submit a PR (or better, a CL to gerrit). Although shouldn't the dependency be the other way around? 🤔

elewis787 · 2024-11-19T22:38:12Z

Using linkname between runtime and syscall is a time honored tradition. Please do submit a PR (or better, a CL to gerrit). Although shouldn't the dependency be the other way around? 🤔

Will do!

That is what I thought as well - but I found a similar pattern in

go/src/syscall/exec_libc.go

Line 60 in 5fdadff

func write1(fd uintptr, buf uintptr, nbyte uintptr) (n uintptr, err Errno)

It's a small change if it makes sense to update!

working on the CL now.

johanbrandhorst · 2024-11-19T22:43:56Z

That link just points to the write1 function, I don't see a go:linkname directive? Conversely, I searched for linkname syscall in the runtime package and found 86 results. I think the runtime package function should linkname to the syscall function, which should implement the import. Does that make sense?

elewis787 · 2024-11-19T22:54:37Z

Yeah, that makes sense.

The exec_libc.go appeared to be pointing to https://github.com/golang/go/blob/master/src/runtime/syscall_aix.go#L232

my naive thought from the above was to say the runtime os ( wasip1 ) had an import fd_write that was linked through write1 or something similar and that the syscall pkg would leverage the runtime impl for it - this does feel backward tho!

gopherbot · 2024-11-19T23:16:10Z

Change https://go.dev/cl/629857 mentions this issue: syscall: point fd_write and random_get to runtime wasip1 implementations

elewis787 · 2024-11-19T23:16:12Z

https://go-review.googlesource.com/c/go/+/629857 - happy to update this to reflect the above - but pushed what I have working to help outline the path so far.

ydnar · 2024-11-21T18:58:05Z

I think it’s important to point out that a duplicate import is not an error, as there are valid cases for a Go program having duplicate go:wasmimport entries that import the same function from the host. In the spirit of minimizing use of go:linkname, I’d argue that duplicate go:wasmimport entries might not only be acceptable, but preferable.

(The contrary would require coordination between all WebAssembly programs and the standard library, which would overly restrict the language from adding new capabilities to the standard library that are not exposed as exported symbols, without breaking backwards compatibility.)

That said:

It should probably be an error if the function signature for duplicate imports differ at the ABI level.
The compiler could resolve duplicate imports (which would address (1)).

elewis787 · 2024-11-21T21:21:40Z

Thanks, @ydnar - makes sense.

If I am following, instead of the above solution using go:linkname, would we instead be looking at the compile steps and attempting to consolidate the duplicates there?

Is it valid for multiple imports to show in the wasm binary that has the same function signature?

i.e

  (import "example" "one" (func (;10;) (type 6)))
  (import "example" "one" (func (;11;) (type 6)))

Somewhat related is a PR in the wasm-tools repo bytecodealliance/wasm-tools#1787 and a related issue bytecodealliance/wasm-tools#1786

This PR leads me to believe that duplicates are acceptable but should be handled by the compiler and resolved to a single declaration.

If that is the case, I will need to dig deeper but I can start tracking through the go:wasmimport pragma. This may help identify if we have a clean way to identify the duplicates and ultimately use a single import.

My near-term goal is to get past the above issue to continue attempting to adapt a wasip1 module to a wasip2 component using an adapter. Transparently, I am approaching this very narrowly and learning along the way.

I'm happy to continue iterating on the above - feel free to assign the issue to me. I'll be following regardless.

I threw together an example of this here https://github.com/elewis787/go-wasm-imports.

The gist from my understanding using go:linkname solved for the internal fd_write and random_get but does not solve the overall issue.

ydnar · 2024-11-21T22:47:21Z

My near-term goal is to get past the above error to continue attempting to adapt a wasip1 module to a wasip2 component using an adapter. Transparently, I am approaching this very narrowly and learning along the way.

I'm sorry, but what's the actual error? The original issue describes a duplicate import but not an error that this causes.

If the wasip1 to wasip2 adapter step is failing, then the issue is likely with wasm-tools and not Go.

elewis787 · 2024-11-21T23:08:45Z

I'm sorry, but what's the actual error? The original issue describes a duplicate import but not an error that this causes.

If the wasip1 to wasip2 adapter step is failing, then the issue is likely with wasm-tools and not Go.

@ydnar here is a quick write-up from my end.

I agree, calling this an error is a stretch. I updated my verbiage above to call out the issue described vs error. The wasm module produced is validating correctly (albeit, IMO, technically malformed).

Initial Findings

Here are the steps I look that lead me to this issue

package main

import "fmt"

func main() {
	fmt.Println("hello world")
}

Compiled to wasip1/wasm
GOOS=wasip1 GOARCH=wasm go build -o main.wasm main.go

We can observe the duplicate fd_write imports by converting to wat

  (import "wasi_snapshot_preview1" "sched_yield" (func (;0;) (type 1)))
  (import "wasi_snapshot_preview1" "proc_exit" (func (;1;) (type 2)))
  (import "wasi_snapshot_preview1" "args_get" (func (;2;) (type 3)))
  (import "wasi_snapshot_preview1" "args_sizes_get" (func (;3;) (type 3)))
  (import "wasi_snapshot_preview1" "clock_time_get" (func (;4;) (type 4)))
  (import "wasi_snapshot_preview1" "environ_get" (func (;5;) (type 3)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;6;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;7;) (type 5)))
  (import "wasi_snapshot_preview1" "random_get" (func (;8;) (type 3)))
  (import "wasi_snapshot_preview1" "poll_oneoff" (func (;9;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_close" (func (;10;) (type 0)))
  (import "wasi_snapshot_preview1" "fd_write" (func (;11;) (type 5)))
  (import "wasi_snapshot_preview1" "fd_fdstat_get" (func (;12;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_fdstat_set_flags" (func (;13;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_get" (func (;14;) (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_dir_name" (func (;15;) (type 6)))

Ignoring them, using the wasmtime adapter pattern, I attempt to make a component
wasm-tools component new --verbose --adapt=wasi_snapshot_preview1.command.wasm -o component.wasm main.wasm

This results in:

error: failed to encode a component from module

Caused by:
    0: failed to decode world from module
    1: module was not valid
    2: module has duplicate import for `wasi_snapshot_preview1::fd_write`

The above error is how I found this issue and the related wasm-tools pr/issue.

Removing Duplicate Imports

When using the go:linkname fix for fd_write and random_get allows for a component.wasm to be created.

However, using wasmtime to execute the component fails
WASMTIME_LOG=wasmtime_wasi=trace wasmtime run --dir=$PWD --env PWD="$PWD" --env PATH="$PATH" component.wasm

This results in:

Error: failed to run main module `component.wasm`

Caused by:
    0: failed to invoke `run` function
    1: error while executing at wasm backtrace:
           0: 0x252ba8 - wit-component:adapter:wasi_snapshot_preview1!wasi_snapshot_preview1::macros::assert_fail::hfc880f03db2630d4
           1: 0x2532da - wit-component:adapter:wasi_snapshot_preview1!args_sizes_get
           2: 0x2577e0 - wit-component:shim!adapt-wasi_snapshot_preview1-args_sizes_get
           3: 0x78501 - <unknown>!runtime.args_sizes_get
           4: 0x78a87 - <unknown>!runtime.goenvs
           5: 0x87038 - <unknown>!runtime.schedinit
           6: 0x100bb6 - <unknown>!runtime.rt0_go
           7: 0x1037f4 - <unknown>!_rt0_wasm_wasip1
           8: 0x252ab3 - wit-component:adapter:wasi_snapshot_preview1!wasi:cli/run@0.2.2#run
    2: wasm trap: wasm `unreachable` instruction executed

Looking in Go

I started looking at how the imports were defined after reading the comment on the wasm-tools PR about removing duplicates: bytecodealliance/wasm-tools#1787 (comment)

johanbrandhorst added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-wasm WebAssembly issues labels May 30, 2023

seankhliao added this to the Unplanned milestone Jul 13, 2024

gabyhelp mentioned this issue Aug 21, 2024

cmd/compile: inaccurate compiler error on duplicate wasmexport symbol #68982

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasm: duplicate import in wasip1 compiled binary #60525

wasm: duplicate import in wasip1 compiled binary #60525

johanbrandhorst commented May 30, 2023

HarikrishnanBalagopal commented Aug 15, 2023 •

edited

Loading

elewis787 commented Nov 19, 2024

johanbrandhorst commented Nov 19, 2024

elewis787 commented Nov 19, 2024

johanbrandhorst commented Nov 19, 2024 •

edited

Loading

elewis787 commented Nov 19, 2024 •

edited

Loading

johanbrandhorst commented Nov 19, 2024

elewis787 commented Nov 19, 2024

gopherbot commented Nov 19, 2024

elewis787 commented Nov 19, 2024 •

edited

Loading

ydnar commented Nov 21, 2024

elewis787 commented Nov 21, 2024 •

edited

Loading

ydnar commented Nov 21, 2024

elewis787 commented Nov 21, 2024 •

edited

Loading

wasm: duplicate import in wasip1 compiled binary #60525

wasm: duplicate import in wasip1 compiled binary #60525

Comments

johanbrandhorst commented May 30, 2023

What version of Go are you using (go version)?

What did you do?

What did you expect to see?

HarikrishnanBalagopal commented Aug 15, 2023 • edited Loading

elewis787 commented Nov 19, 2024

johanbrandhorst commented Nov 19, 2024

elewis787 commented Nov 19, 2024

johanbrandhorst commented Nov 19, 2024 • edited Loading

elewis787 commented Nov 19, 2024 • edited Loading

johanbrandhorst commented Nov 19, 2024

elewis787 commented Nov 19, 2024

gopherbot commented Nov 19, 2024

elewis787 commented Nov 19, 2024 • edited Loading

ydnar commented Nov 21, 2024

elewis787 commented Nov 21, 2024 • edited Loading

ydnar commented Nov 21, 2024

elewis787 commented Nov 21, 2024 • edited Loading

Initial Findings

Removing Duplicate Imports

Looking in Go

What version of Go are you using (`go version`)?

HarikrishnanBalagopal commented Aug 15, 2023 •

edited

Loading

johanbrandhorst commented Nov 19, 2024 •

edited

Loading

elewis787 commented Nov 19, 2024 •

edited

Loading

elewis787 commented Nov 19, 2024 •

edited

Loading

elewis787 commented Nov 21, 2024 •

edited

Loading

elewis787 commented Nov 21, 2024 •

edited

Loading