Webpack doesn’t work well with wasm modules created with Emscripten

[surma]

Feature request

What is the current behavior?
The modularized JS emitted by Emscripten registers a global with a given name that loads the wasm file on invocation, initializes the wasm runtime and returns a Module.

Making it work with Webpack is quite hard as there seems to be interference with Webpack 4 defaults.
This is the webpack.config.js that I came up with:

const webpack = require("webpack");
const path = require("path");

module.exports = {
  mode: "development",
  entry: "./index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "bundle.js"
  },
  module: {
    defaultRules: [
      {
        type: "javascript/auto",
        resolve: {}
      }
    ],
    rules: [
      {
        test: /fibonacci\.js$/,
        loader: "exports-loader"
      },
      {
        test: /fibonacci\.wasm$/,
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
  // This is necessary due to the fact that emscripten puts both Node and
  // web code into one file. The node part uses Node’s `fs` module to load
  // the wasm file.
  // Issue: https://github.com/kripken/emscripten/issues/6542.
  plugins: [new webpack.IgnorePlugin(/(fs)/)]
};

(Here is a minimal test project in a gist that you can clone and build with npm start. Docker required!)

edit:
In the meantime, @sokra informed that that I can simplify the configuration a bit (and make it less like a sledgehammer):

module.exports = {
  /* ... */
  browser: { 
    "fs": false // ← !!
  },
  module: {
    rules: [
      /* ... */
      {
        test: /fibonacci\.wasm$/,
        type: "javascript/auto", // ← !!
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
};

Unexpected things I had to do

• I needed to overwrite defaultRules as otherwise some sort of default rule will run in addition to the ones I specified and making webpack error “Module parse failed: magic header not detected” (try it!)
• I needed to specify file-loader for the wasm file as otherwise webpack tries to resolve the names of the wasm module’s import object like env, which are provided by the JS file.
• I needed to set a locateFile() function as webpack changes the file (and potentially path) of the wasm file and Emscripten hardcodes that name (not visible here but in the gist)

I am not sure what the right course of action here is, but considering that most wasm projects are going to be built with Emscripten, I feel like it’s worth making it easier.

Happy to answer Qs to give y’all a clearer picture.

What is the expected behavior?

Ideally, Webpack would recognize the typical Emscripten JS files and automatically bundle the accomodating wasm module and make paths work.

Other relevant information:
webpack version: 4.8.3
Node.js version: 10
Operating System: Mac OS 10.13.4
Additional tools:

[xtuc]

Here's my suggestion: #7264 (comment)

We don't need to explicitly support emscripten, that would be generic enough.

Note that wasn-bindgen already does that.

[sokra]

It would be great to see a ESM target for Emscripten. This is how webpack's wasm support works. It expects WASM to be imported via import { ... } from "./something.wasm". WASM can get access to JS functions via (import "./something.js" "exportName") (import section).

wasm-bindgen creates two files abc.wasm and abc.js. The abc.js exports all exposed functions (public api) and all internal functions. It also imports the abc.wasm file. The abc.wasm imports all internal functions from abc.js.

That works great but exposes all internal functions too. I think a 3 file output would be better:

• abc.js exports all public functions, imports abc_internal.js and abc_internal.js
• abc_internal.js exports all internal functions
• abc_internal.wasm imports abc_internal.js

Even better if repeated runtime code could be moved into a npm package: This way the wasm could import it directly from this package. This could get handy if you got many wasm modules in an application.

[xtuc]

For ref emscripten-core/emscripten#6483

Also note that the same works for a module on npm. I think it would be more practical.

[mikeheddes]

I also came across this problem when using a JS library I made, that uses WebAssembly, in a React web project.

I solved it by adding some hacky pre.js code to basically hijack the wasm module instantiation from the Emscripten created glue code. This way I have all the control over initialising the WASM module. The pre.js file overrides the createWasm function and adds a custom module initialisation function which it exports using ES6. Note that I compile with the default JS glue code settings (see cpp build script) so the EXPORT_ES6 and modularize options are not set but because the pre.js code has an ES6 export statement the generated glue code is an ES6 module.

After that I combine the generated JS glue code with the JS API interface code bundled with Rollup. The entry of this can be found in index.js.

Then finally when using the library I need to provide it an arrayBuffer (node.js) or a fetch instance (browsers) of the generated dcgp.wasm file. With webpack that means I needed the following rule:

{
  test: /\.wasm$/,
  type: 'javascript/auto',
  loader: 'file-loader',
}

Some implementation references:

• Simple Library API example
• Implementation in a React app

So my conclusion being...
I agree with @surma that Webpack and Emscripten don't work well together but I think the main solution should be found in the way Emscripten generates the JS glue code and not how Webpack should handle them. I'm also not sure if @surma's suggestion in #6542 to make separate platform outputs will fix the root cause of this problem. For me the following implementation would make more sense in the JS ecosystem:

// node.js
import { initializer } from './wasm-glue-code'
import { fileToArrayBuffer } from 'emscripten-helpers'

const myLibrary = initializer(fileToArrayBuffer('./wasm-file.wasm'))

// web
import { initializer } from './wasm-glue-code'

const myLibrary = initializer(fetch('./wasm-file.wasm'))

Hope this was helpful in some way.

[VirtualTim]

Hi @surma, @sokra,
I know this is fairly old, but it seems like:

browser: { 
  "fs": false // ← !!
},

no longer works, and I can't see any reference to browser in the webpack documentation.
Any ideas what we should be using instead?

Edit: figured it out. Use this instead:

node: { 
  fs: "empty"
},

[mmarczell-graphisoft]

You can specify -sENVIRONMENT=web to stop Emscripten from including node-targeted JavaScript.

[dhdaines]

For Webpack 5, disabling the node-specific stuff is relatively easy:

resolve: {
    fallback: {
        crypto: false,
        fs: false,
        path: false
    },
},

To include the wasm files, one solution is just to use copy-webpack-plugin, which is annoying since it makes it non-transparent for people to use your module, but then again, they're already adding the magic resolve incantation...

const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
    plugins: [
        new CopyPlugin({
          patterns: [
              { from: "node_modules/MYPACKAGE/MYPACKAGE.wasm*",
                to: "[name][ext]" },
          ],
      }),
    ],
};

It would sure be swell if someone could write a webpack plugin or loader that would automate this so people could, like, just use Emscripten-compiled modules transparently. It seems like it should be relatively easy to recognize an Emscripten .js/.wasm file pair?

[marczellm]

@dhdaines
I once managed to get a working solution with setting the Emscripten build to EXPORT_ES6, and then simply importing the .js file in my Webpack based (actually Angular) project. Webpack intelligently detected that the .js module further depends on the .wasm file (and maybe even the .data file for preloaded files?)

Though this seemed to work, it printed some - to me - unintelligible, and unsilenceable warning on every build, so ultimately I used Angular's configuration options to bring in the JS file as a regular script and the WASM as a resource/asset.

[9oelM]

Hi guys, I felt extremely lost while I was digging into this issue, so after I got everything working, I made a example repo containing detailed walkthroughs:

• Repo: https://github.com/9oelM/emscripten-cplusplus-webpack-example
• Demo: https://9oelm.github.io/emscripten-cplusplus-webpack-example/

Hope it helps someone.

[dhdaines]

Note also that you still need to disable Webpack's "mocking" of __filename, at the very least, or you will not be able to load your WASM unless it is in the root directory (Webpack sets __filename = "/index.js" and then Emscripten' s preamble takes this as the path of the current script in order to find the WASM, which is, obviously, wrong):

    node: {
        global: false,
        __filename: false,             
        __dirname: false,
    },

[dhdaines]

@dhdaines I once managed to get a working solution with setting the Emscripten build to EXPORT_ES6, and then simply importing the .js file in my Webpack based (actually Angular) project. Webpack intelligently detected that the .js module further depends on the .wasm file (and maybe even the .data file for preloaded files?)

Can confirm that this works now, and it produces something like this: https://github.com/dhdaines/soundswallower-demo/tree/main/docs

...as long as you don't use Angular, because Angular doesn't include whatever magical Webpack configuration makes Webpack use the appropriate loader for the WASM file.

If you try to fix this by treating the WASM as a normal asset, Angular will use Webpack's default broken behaviour of munging the import.meta.url used in the JavaScript code to refer to the WASM to a file:// URL on your local file system. (see angular/angular-cli#22388)

So, you can use a custom Webpack config to disable this behaviour, but this breaks testing among other things, and probably means you can't target non-ES6 browsers. (see https://stackoverflow.com/questions/74038161/angular-cli-and-loading-assets-with-import-meta-url-causing-not-allowed-to-load)

Use ES6 modules, they said. It'll be great, they said.

[dhdaines]

Just to follow up on this. The problem is actually Angular and its mysterious built-in webpack configuration, which disables module.parser.javascript.url for some unknown reason! If you re-enable it in a custom webpack configuration, then EXPORT_ES6=1 creates modules that work just fine, as noted above - the call to new URL("foo.wasm", import.meta.url).href in the JavaScript code is parsed by webpack, which duly adds your WASM as an asset and rewrites the URL properly.

The correct (as far as I can tell) workaround is to enable module.parser.javascript.url in a custom webpack config: angular/angular-cli#24617

[mmarczell-graphisoft]

@dhdaines Thank you for the deep dive! I recently tried to reproduce my own previously reported success and failed. Now I can try again...