Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust v1.82 enables reference type transformations by default #4211

Closed
ctm opened this issue Oct 18, 2024 · 27 comments
Closed

Rust v1.82 enables reference type transformations by default #4211

ctm opened this issue Oct 18, 2024 · 27 comments
Labels

Comments

@ctm
Copy link

ctm commented Oct 18, 2024

Describe the Bug

The Hello World! example fails to run or even compile with Rust 1.82.0, presumably due to reference types. That example uses webpack with a configuration that will invoke @webassemblyjs/wasm-parser which doesn't support reference types. This other bug report has a backtrace that is similar to what happens now.

Steps to Reproduce (when using Rust 1.82.0)

  1. cd examples/hello_world
  2. install the needed node packages (I used yarn install, I'm sure npm has something similar, possibly npm install)
  3. Try to build (e.g. npm run build or yarn build)
  4. See backtrace

Everything you need is in this repository other than installing the npm modules.

Expected Behavior

[main]% rustup default 1.81.0
rustup default 1.81.0
info: using existing install for '1.81.0-x86_64-apple-darwin'
info: default toolchain set to '1.81.0-x86_64-apple-darwin'

  1.81.0-x86_64-apple-darwin unchanged - rustc 1.81.0 (eeb90cda1 2024-09-04)

[main]% npm run build
npm run build

> build
> webpack

🧐  Checking for wasm-pack...

✅  wasm-pack is installed at /Users/ctm/.cargo/bin/wasm-pack. 

ℹ️  Compiling your crate in development mode...

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling hello_world v0.0.0 (/Users/ctm/git_others/wasm-bindgen/examples/hello_world)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.43s
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: ✨   Done in 0.87s
[INFO]: 📦   Your wasm pkg is ready to publish at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/pkg.
✅  Your crate has been correctly compiled

assets by status 54 KiB [cached] 1 asset
asset index.js 16.3 KiB [emitted] (name: main)
asset index.html 232 bytes [compared for emit]
runtime modules 4.87 KiB 7 modules
cacheable modules 3.61 KiB (javascript) 54 KiB (webassembly)
  modules by path ./pkg/*.js 3.5 KiB
    ./pkg/index.js 141 bytes [built] [code generated]
    ./pkg/index_bg.js 3.36 KiB [built] [code generated]
  ./index.js 48 bytes [built] [code generated]
  ./pkg/index_bg.wasm 70 bytes (javascript) 54 KiB (webassembly) [built] [code generated]
webpack 5.95.0 compiled successfully in 1640 ms

Actual Behavior

[main]% rustup default stable
rustup default stable
info: using existing install for 'stable-x86_64-apple-darwin'
info: default toolchain set to 'stable-x86_64-apple-darwin'

  stable-x86_64-apple-darwin unchanged - rustc 1.82.0 (f6e511eec 2024-10-15)

[main]% npm run build
npm run build

> build
> webpack

🧐  Checking for wasm-pack...

✅  wasm-pack is installed at /Users/ctm/.cargo/bin/wasm-pack. 

ℹ️  Compiling your crate in development mode...

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling hello_world v0.0.0 (/Users/ctm/git_others/wasm-bindgen/examples/hello_world)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.37s
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: ✨   Done in 0.80s
[INFO]: 📦   Your wasm pkg is ready to publish at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/pkg.
✅  Your crate has been correctly compiled

assets by status 1.58 KiB [cached] 1 asset
asset index.js 13.2 KiB [emitted] (name: main)
asset index.html 232 bytes [compared for emit]
runtime modules 1.77 KiB 4 modules
cacheable modules 3.91 KiB (javascript) 56.1 KiB (webassembly)
  modules by path ./pkg/*.js 3.82 KiB
    ./pkg/index.js 167 bytes [built] [code generated]
    ./pkg/index_bg.js 3.66 KiB [built] [code generated]
  ./index.js 48 bytes [built] [code generated]
  ./pkg/index_bg.wasm 40 bytes (javascript) 56.1 KiB (webassembly) [built] [code generated] [1 error]

ERROR in ./pkg/index_bg.wasm
Module parse failed: Unknown element type in table: 0xNaN
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
Error: Unknown element type in table: 0xNaN
    at parseTableType (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1018:13)
    at parseTableSection (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1274:24)
    at parseSection (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1406:24)
    at Object.decode (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1740:25)
    at decode (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/index.js:253:21)
    at WebAssemblyParser.parse (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/wasm-async/AsyncWebAssemblyParser.js:61:19)
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:1303:19
    at processResult (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:937:11)
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:1030:5
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/loader-runner/lib/LoaderRunner.js:407:3
 @ ./pkg/index.js 1:0-40 4:15-19 5:0-21
 @ ./index.js 1:0-30 3:0-5

webpack 5.95.0 compiled with 1 error in 1532 ms

Additional Context

I believe this can be fixed by disabling the reference-types feature for wasm32-unknown-unknown, however even after reading the Enabled WebAssembly Features section of the rustc book (and trying various things), I have not been successful at disabling that feature.

@widavies
Copy link

widavies commented Oct 18, 2024

I second this, cannot get this working on latest versions

Rust: 1.82
Webpack: 5.95.0
wasm-bindgen: 0.2.95

Trying to add a wasm rule in my Webpack config just leads to different errors:

  module: {
    rules: [
      {
        test: /\.wasm$/,
        type: "asset/resource"
      }
    ]
  },
ERROR in ../pkg/index.js 5:0-21
Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)
 @ ./index.js 1:0-43 4:17-29
 @ ./bootstrap.js 4:0-20

@ctm
Copy link
Author

ctm commented Oct 19, 2024

Upon further reading and experimenting, I suspect that for now it will simply impossible to use 1.82.0 with anything that uses wasm-pack, because disabling reference-types requires unstable features and 1.82.0 itself is stable. If so, this demo won't be able to use a stable 1.82 or later rust release until they either make it so that reference-types is not enabled by default or provide a stable way (that works through the wasm-pack stack) to disable that default.

@nickbabcock
Copy link
Contributor

Similarly, Rust 1.82 and Node 18 don't play well together when running wasm-bindgen-test, as node will crash with a bit of a nasty stacktrace.

@CryZe
Copy link
Contributor

CryZe commented Oct 19, 2024

So there's multiple things at play here. LLVM activated the reference-types feature by default, which got pulled into Rust 1.82. The reference-types feature generally affects almost nothing in how LLVM generates code, except it now supports some extended table index encoding.

The problem is that wasm-bindgen looks at the features section of the module and assumes that you want FULL reference-types support. It does so here:

// Enable reference type transformations if the module is already using it.
if let Ok(true) = wasm_bindgen_wasm_conventions::target_feature(&module, "reference-types")
{
self.externref = true;
}

Usually you are meant to use the wasm-bindgen CLI flag --reference-types to achieve the same effect. But this automatic detection above activates it anyway now.

This by itself isn't really a problem, but webpack, or rather its webassemblyjs dependency is mostly unmaintained and does not currently support the reference-types proposal.

@daxpedda I propose that wasm-bindgen for the time being should be patched to not automatically turn on the externref support if it detects it from the features section. I can do that PR if you think this is the right course of action.

@Kofituo
Copy link

Kofituo commented Oct 23, 2024

I literally created this exact issue last 2 weeks and it was quickly dismissed. I am glad it's finally being worked on as this is a major issue.

@mplanchard
Copy link

mplanchard commented Oct 25, 2024

Running into the same issue here:

❯ webpack  --version 

  System:
    OS: Linux 6.6 NixOS 24.05 (Uakari) 24.05 (Uakari)
    CPU: (20) x64 13th Gen Intel(R) Core(TM) i7-1370P
    Memory: 21.40 GB / 62.52 GB
  Binaries:
    Node: 20.9.0 - /nix/store/igr9rd5rbpm9bagf49q20b9vhcag9w08-nodejs-20.9.0/bin/node
    Yarn: 3.2.0 - /nix/store/60izdbxbkzm0fkknwi357xi5lcszcwwi-yarn-1.22.19/bin/yarn
    npm: 10.1.0 - /nix/store/igr9rd5rbpm9bagf49q20b9vhcag9w08-nodejs-20.9.0/bin/npm
  Browsers:
    Firefox: 131.0.2
  Monorepos:
    Yarn Workspaces: 3.2.0

❯ rustc --version
rustc 1.82.0 (f6e511eec 2024-10-15)

❯ wasm-bindgen --version
wasm-bindgen 0.2.92

Using react 18 with CRA-generated config, applying the following overrides to get wasm to work:

    if (!config.resolve.extensions) {
      config.resolve.extensions = [];
    }
    if (!config.resolve.extensions.includes(".wasm")) {
      config.resolve.extensions.push(".wasm");
    }

    const wasmExtensionRegExp = /\.wasm$/;

    if (!config.module.rules) {
      config.module.rules = [];
    }

    // Ensure that file-loader ignores WASM files.
    // Sourced from https://prestonrichey.com/blog/react-rust-wasm/
    config.module.rules.forEach((rule) => {
      (rule.oneOf || []).forEach((oneOf) => {
        if (oneOf.loader && oneOf.loader.indexOf("file-loader") >= 0) {
          // Make file-loader ignore WASM files
          oneOf.exclude.push(wasmExtensionRegExp);
        }
      });
    });

    // Use wasm-loader for WASM files
    config.module.rules.push({
      test: wasmExtensionRegExp,
      // I'm going to level with you: I copied this in from the example, but I
      // have no idea why it's necessary. If it's not here, it breaks, though.
      include: path.resolve(__dirname, "src"),
      use: [{ loader: require.resolve("wasm-loader"), options: {} }],
    });

Definitely seems limited to webpack for the bundler target specifically. We use the same package compiled for the nodejs target in Node without issues, as well as in svelte via vite.

bidzyyys pushed a commit to OpenZeppelin/rust-contracts-stylus that referenced this issue Oct 28, 2024
It seems that reference types from WASM are not supported by
`wasm-parser` crate
(rustwasm/wasm-bindgen#4211).
It makes us pin to the recent nightly toolchain version that works, and
contribution experience gets quite convoluted.
Hopefully there is an option to disable `reference-types` with a flag:
`-C target-cpu=mvp` for `wasm32-unknown-unknown` target.

Resolves #374

Single source of rust toolchain from `rust-toolchain.toml` added for ci
pipelines (actions-rust-lang/setup-rust-toolchain@v1 gh action used for
that).

Resolves #294
@daxpedda
Copy link
Collaborator

@CryZe

This by itself isn't really a problem, but webpack, or rather its webassemblyjs dependency is mostly unmaintained and does not currently support the reference-types proposal.

@daxpedda I propose that wasm-bindgen for the time being should be patched to not automatically turn on the externref support if it detects it from the features section. I can do that PR if you think this is the right course of action.

My first gut feeling is to continue as-is. Unfortunately, I'm a really clueless about webpack and the ecosystem it sits in. But it seems to me, that if its unmaintained, users should upgrade to something maintained instead of the rest of the ecosystem having to stay behind for the purpose of compatibility.

Inherently, this is a problem that should be solved by Rustc and not by wasm-bindgen. The reality however is that this is a bigger problem in the current toolchain (WebAssembly/tool-conventions#233) and wasm-bindgen is not the part playing its role here. However, it seems to me that applying this fix in wasm-bindgen doesn't really fix the issue, because LLVM already emits reference-type proposal specific things (e.g. table encodings) that could affect any part of the toolchain not supporting this already.

In that light, I will go ahead and pin this issue so we can get as much input as possible before settling on a solution!

@daxpedda daxpedda changed the title Hello World! example breaks with Rust 1.82.0 Rust v1.82 enables reference type transformations by default Oct 29, 2024
@daxpedda daxpedda pinned this issue Oct 29, 2024
@CryZe
Copy link
Contributor

CryZe commented Oct 29, 2024

My PR as per my testing fully unblocks the webpack ecosystem. It would only be a short term bugfix to unblock people who have been affected by it. I'm slowing working on PR on supporting reference types fully in webassemblyjs / webpack, but it probably is still taking a while (also depending on how responsive the maintainer is). So I would recommend going with my PR until webpack is fixed, which hopefully shouldn't take too long (but probably in the range of multiple months).

@daxpedda
Copy link
Collaborator

While I wouldn't mind holding off until then, I would like to know how this is going to play out in the future. What do we do when LLVM enables other proposals by default which break webpack? It seems unsustainable to me.

ls it possible to move wasm-pack away to something more sustainable?
If the issue only affects the bundler target in practice, could we change the behavior only for that target?

@mplanchard
Copy link

Just speaking for us as an example, we have a large, legacy application react using webpack. We would love to be able to move off of it, and we are in the process of doing so, but it is going to take some time. It would be great to not have to pin our Rust compiler to an old version until that happens, so a temporary fix would be helpful, whether that be here, in a webpack plugin, or wherever.

@Kofituo
Copy link

Kofituo commented Oct 30, 2024

Just speaking for us as an example, we have a large, legacy application react using webpack. We would love to be able to move off of it, and we are in the process of doing so, but it is going to take some time. It would be great to not have to pin our Rust compiler to an old version until that happens, so a temporary fix would be helpful, whether that be here, in a webpack plugin, or wherever.

This seem to work for now
webpack/webpack#15566 (comment)

@mplanchard
Copy link

mplanchard commented Oct 30, 2024 via email

@mplanchard
Copy link

FYI the strip = true solution doesn't seem to work for me.

Added the following in the package's Cargo.toml

[build]
strip = true

[profile.dev]
strip = true

[profile.release]
lto = true
opt-level = 'z'
strip = true

but still get the following when trying to import the WASM file in the built application:

Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)

@Kofituo
Copy link

Kofituo commented Oct 30, 2024

FYI the strip = true solution doesn't seem to work for me.

Added the following in the package's Cargo.toml

[build]
strip = true

[profile.dev]
strip = true

[profile.release]
lto = true
opt-level = 'z'
strip = true

but still get the following when trying to import the WASM file in the built application:

Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)

This sample repository works
https://github.com/Kofituo/hello-world/tree/master

@mplanchard
Copy link

Yeah, I believe this could be due to some settings in create-react-app? Our current override is to include wasm files and load them with the wasm-loader plugin, which still results in this error. I haven't found another way to override yet to get it working with the stripped WASM binary.

@daxpedda
Copy link
Collaborator

That is a different error entirely.
My first guess is a different CLI and wasm-bindgen version, but that should be caught by the hash check.
Could you try using Rust v1.81 just to make sure this is unrelated?

@ctm
Copy link
Author

ctm commented Oct 31, 2024

Thanks! Aha. Yes. I was inadvertently running an old wasm-bindgen:

[master]% which wasm-bindgen
/Users/ctm/.cargo/bin/wasm-bindgen

All I had done to get the wasm-bindgen crate was to have added

[patch.crates-io]
wasm-bindgen = { git = "https://github.com/rustwasm/wasm-bindgen" }

to my top-level Cargo.toml, but that doesn't bring in a new wasm-bindgen CLI. I didn't think about that because the stack I use is fairly opaque to me and I never run wasm-bindgen by hand; it's just done by magic when I say yarn dev or yarn build.

So now that I've also done a cargo install wasm-bindgen-cli --git "https://github.com/rustwasm/wasm-bindgen", my development and release builds work and build usable artifacts.

@daxpedda
Copy link
Collaborator

daxpedda commented Nov 7, 2024

webpack (more precisely webassemblyjs) seems to now support reference-types, but I'm unsure why wasm-pack is still failing.

So these are all the use-cases that have surfaced so far:

  • wasm-pack (failing because of webpack and webassemblyjs). Fixed in Update CI to Rust v1.82 #4235.
  • wasm-opt failing because reference type transformations use table.fill, an instruction that requires the bulk memory proposal. Fixed in Deprecate --reference-types #4237.
  • Safari <= v15 doesn't support reference types.
  • wasm2js doesn't support reference types.

@vr-devil
Copy link

Hi, everyone.

It seems that the wasm files packaged with wasm-pack still can’t be processed by webpack. I have confirmed that both webpack and webassemblyjs are updated to the latest versions, yet I’m still encountering the following error:

Module parse failed: Unknown element type in table: 0xNaN
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
Error: Unknown element type in table: 0xNaN

Currently, it only works correctly if I set strip = true in the Cargo.toml file.

I’d like to know if this issue has not yet been fully resolved, or if using strip = true is the final solution.

@daxpedda
Copy link
Collaborator

daxpedda commented Nov 11, 2024

The next release is going to not apply reference type transformation when using the bundler output, which should fix this.

However, I'm not sure what that strip usage is about, which is related to debug information and not the reference type target feature.

@vr-devil
Copy link

@daxpedda Thanks for the response. Alright, I’ll use strip = true as a temporary solution and wait for the next release.

alexjg added a commit to automerge/automerge that referenced this issue Nov 19, 2024
Problem: due to an intersection of changes in LLVM and wasm-bindgen
described here[1], compiling with rustc1 1.82.0 breaks the WebPack
webassembly integration. It looks like wasm-bindgen plan to push a fix
for this (despite it not really being their responsibility) but that's
not in yet.

Solution: pin the compiler to 1.81 until the wasm-bindgen fix is int.

[1]: rustwasm/wasm-bindgen#4211
alexjg added a commit to automerge/automerge that referenced this issue Nov 19, 2024
Problem: due to an intersection of changes in LLVM and wasm-bindgen
described here[1], compiling with rustc1 1.82.0 breaks the WebPack
webassembly integration. It looks like wasm-bindgen plan to push a fix
for this (despite it not really being their responsibility) but that's
not in yet.

Solution: pin the compiler to 1.81 until the wasm-bindgen fix is int.

[1]: rustwasm/wasm-bindgen#4211
alexjg added a commit to automerge/automerge that referenced this issue Nov 19, 2024
Problem: due to an intersection of changes in LLVM and wasm-bindgen
described here[1], compiling with rustc1 1.82.0 breaks the WebPack
webassembly integration. It looks like wasm-bindgen plan to push a fix
for this (despite it not really being their responsibility) but that's
not in yet.

Solution: pin the compiler to 1.81 until the wasm-bindgen fix is int.

[1]: rustwasm/wasm-bindgen#4211
alexjg added a commit to automerge/automerge that referenced this issue Nov 19, 2024
Problem: due to an intersection of changes in LLVM and wasm-bindgen
described here[1], compiling with rustc1 1.82.0 breaks the WebPack
webassembly integration. It looks like wasm-bindgen plan to push a fix
for this (despite it not really being their responsibility) but that's
not in yet.

Solution: pin the compiler to 1.81 until the wasm-bindgen fix is int.

[1]: rustwasm/wasm-bindgen#4211
alexjg added a commit to automerge/automerge that referenced this issue Nov 19, 2024
Problem: due to an intersection of changes in LLVM and wasm-bindgen
described here[1], compiling with rustc1 1.82.0 breaks the WebPack
webassembly integration. It looks like wasm-bindgen plan to push a fix
for this (despite it not really being their responsibility) but that's
not in yet.

Solution: pin the compiler to 1.81 until the wasm-bindgen fix is int.

[1]: rustwasm/wasm-bindgen#4211
@daxpedda
Copy link
Collaborator

With #4253 I consider this issue resolved.

To summarize: this is not a wasm-bindgen problem but a Rustc one, which enables reference-types by default. While it is true that the output of Rustc can be sanitized to not include any reference-type instructions by using removing the debugging table, this is a workaround that could break at any time.

In practice, if you are relying on a toolchain that does not support the reference type proposal, please update it! webpack has been patched to support reference types now. For users trying to target older engines, like Safari v14, the only option is to either use -Ctarget-cpu=mvp with build-std (as described here) or switch to the new wasm32v1-none target, which notable doesn't support std.

However, I'm not sure what that strip usage is about, which is related to debug information and not the reference type target feature.

To answer my own question: the reference type proposal changes the way tables are encoded. So by removing the table containing debug information, it triggers the output by LLVM (or wasm-opt?) to use the old encoding.

@daxpedda daxpedda unpinned this issue Nov 28, 2024
irh added a commit to koto-lang/koto that referenced this issue Dec 2, 2024
The ubuntu-latest image uses Node v18, which isn't compatible with
wasm-bindgen-test when building with Rust 1.82+ due to the use of LLVM's
`reference-types` feature, see:
rustwasm/wasm-bindgen#4211

ubuntu-24.04 includes Node v20 which doesn't have the same problem,
allowing the wasm tests to move back to stable.
irh added a commit to koto-lang/koto that referenced this issue Dec 2, 2024
The ubuntu-latest image uses Node v18, which isn't compatible with
wasm-bindgen-test when building with Rust 1.82+ due to the use of LLVM's
`reference-types` feature, see:
rustwasm/wasm-bindgen#4211

ubuntu-24.04 includes Node v20 which doesn't have the same problem,
allowing the wasm tests to move back to stable.
@torokati44
Copy link

torokati44 commented Dec 2, 2024

@daxpedda:

... For users trying to target older engines, like Safari v14, the only option is to either use -Ctarget-cpu=mvp with build-std (as described here) or switch to the new wasm32v1-none target, which notable doesn't support std.

Both of these are far from ideal though... :/
And it feels like a regression from 1.81 - the "stubby" std was just fine for us. 🥲

@torokati44
Copy link

At least if build-std was (anywhere near close to being) stable... :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants