Loader .node: copy don't copy files into outdir

[1bye]

Ubuntu 22.04
Used runtimes: Bun 1.0.21v / Node.js 18

Hello!

I'm attempting to bundle the faiss-node package with esbuild. Unfortunately, after building, it doesn't copy the files (.node) from faiss-node:

Copy code
error: Could not locate the bindings file. Tried:
 → /home/mrbye/Documents/GitHub/package-builder/build/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/out/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/out/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/default/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/compiled/20.8.0/linux/x64/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/release/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/debug/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/default/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/lib/binding/node-v115-linux-x64/faiss-node.node
      at B (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:1:2455)
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:3:706
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:1:286
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:3:1208

To reproduce the issue, you should install faiss-node and copy the settings and code from here
The output dir structure is:

dir
└── index.js
# nothing here

Also I've tried your plugin native-node-modules but it don't work :(
Thanks

EDIT:

My package.json

{
  "name": "package-builder",
  "module": "index.ts",
  "type": "module",
  "devDependencies": {
    "@types/bun": "latest"
  },
  "peerDependencies": {
    "typescript": "^5.0.0"
  },
  "dependencies": {
    "esbuild": "^0.19.11",
    "faiss-node": "^0.5.1"
  }
}

[evanw]

This has nothing to do with the file loader. The .node file is never loaded in the first place. You can tell because removing --loader:.node=copy doesn't cause the build to fail. You can see the problem with --log-level=debug:

● [DEBUG] Indirect calls to "require" will not be bundled [indirect-require]

    node_modules/bindings/bindings.js:94:8:
      94 │       : require;
         ╵         ~~~~~~~

You are trying to bundle something that is not designed to be bundled. The solution is to not do that. Quoting from esbuild's getting started instructions for node:

You also may not want to bundle your dependencies with esbuild. There are many node-specific features that esbuild doesn't support while bundling such as __dirname, import.meta.url, fs.readFileSync, and *.node native binary modules. You can exclude all of your dependencies from the bundle by setting packages to external

Indeed, using --packages=external does fix your code sample. Sometimes libraries just aren't bundler-friendly. It's reasonable for a package containing platform-specific binary executables to not support bundling. For example, esbuild's JS API itself also doesn't support being bundled for a similar reason.

[1bye]

Thank you. However, could you also assist me with the following issue?

I am attempting to move the build folder to the outdir if it exists in the package. However, when I run the bundled file, it attempts to read from the root folder (I only need resolve issue with importing from root to outdir). Here is my code:

import { build } from "esbuild";
import fs from 'fs-extra';

const nativeNodeModulesPlugin = {
  name: 'native-node-modules',
  setup(build) {
    // If a ".node" file is imported within a module in the "file" namespace, resolve 
    // it to an absolute path and put it into the "node-file" virtual namespace.
    build.onResolve({ filter: /\.node$/g, namespace: 'file' }, args => {
    	console.log(args)
    	return ({
      path: require.resolve(args.path, { paths: [args.resolveDir] }),
      namespace: 'node-file',
    })})

    // Files in the "node-file" virtual namespace call "require()" on the
    // path from esbuild of the ".node" file in the output directory.
    build.onLoad({ filter: /.*/, namespace: 'node-file' }, args => ({
      contents: `
        import path from ${JSON.stringify(args.path)}
        try { module.exports = require(path) }
        catch {}
      `,
    }))

    // If a ".node" file is imported within a module in the "node-file" namespace, put
    // it in the "file" namespace where esbuild's default loading behavior will handle
    // it. It is already an absolute path since we resolved it to one above.
    build.onResolve({ filter: /\.node$/, namespace: 'node-file' }, args => ({
      path: args.path,
      namespace: 'file',
    }))

    // Tell esbuild's default loading behavior to use the "file" loader for
    // these ".node" files.
    let opts = build.initialOptions
    opts.loader = opts.loader || {}
    opts.loader['.node'] = 'file'
  },
}
/* Plugin to move build folder from packages to outdir  */
async function copyBuildFolderPlugin() {
  const pathModule = await import('path');

  return {
    name: 'copy-build-folder',
    setup(build) {
      build.onResolve({ filter: /./ }, (args) => {
        const buildFolderPath = pathModule.join(`${process.cwd()}/node_modules/${args.path}`, 'build');
        if (fs.existsSync(buildFolderPath) && fs.statSync(buildFolderPath).isDirectory()) {

          fs.copySync(buildFolderPath, `${process.cwd()}/dir/build`);
        }

        return null;
      });
    },
  };
}

await build({
	entryPoints: ['./source/index.js'],
	minify: false,
	bundle: true,
	platform: 'node',
	plugins: [nativeNodeModulesPlugin, await copyBuildFolderPlugin()],
	outdir: 'dir'
});

Log from run

120 |             throw e;
121 |           }
122 |         }
123 |       }
124 |       err = new Error(
                  ^
error: Could not locate the bindings file. Tried:
 → /home/mrbye/Documents/GitHub/package-builder/build/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/out/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/Debug/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/out/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/Release/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/build/default/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/compiled/20.8.0/linux/x64/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/release/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/debug/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/addon-build/default/install-root/faiss-node.node
 → /home/mrbye/Documents/GitHub/package-builder/lib/binding/node-v115-linux-x64/faiss-node.node
      at bindings (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:124:13)
      at node_modules/faiss-node/lib/index.js (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:184:17)
      at __require (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:4:21)
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:203:55

Here is the structure of folder:

dir
├── build
│   └── Release
│       ├── faiss-node.node
│       ├── libfaiss.a
│       ├── libgfortran.so.5
│       ├── libgomp.so.1
│       ├── libopenblas.so.0
│       └── libquadmath.so.0
└── index.js

Thanks :)

[evanw]

Sorry, I don't understand. Why are you still trying to bundle this library? You can get it to work by dropping all of the plugin stuff and using packages: 'external' instead. This library is not designed to be bundled.

[1bye]

Sorry and thank you. I'm closing this issue.