☂️ Node/npm compatibility issue for Q4 2023

[bartlomieju]

This is an umbrella issue for Node.js/npm compatibility work that we will undertake in Q4 2023.

We made a lot of progress in Q2 with polyfills for node:http, node:https and node:http2 getting more coverage, but there's still a lot of work to do.

There are several areas that we intend to improve this quarter:

• "Bring your own node_modules" (BYONM) - Deno should be able to work on projects with "node_modules" directory already installed by a 3rd party tool (npm, yarn, pnpm).

This should lower the barrier of entry to try existing projects using Deno and should fix a lot of compatibility issues from various build tools that we're experiencing at the moment (eg. Vite).

This work is well underway and is tracked in #18967.

• npm lifecycle scripts - there are many packages that use native addons (Node-API) that have either preinstall, install or postinstall scripts.

Currently these scripts are ignored by Deno which causes certain packages to fail, even though they could work without a problem in Deno (eg. sharp fails when imported, but that's only because of the postinstall script not being run). We do not intend to arbitrarily run these scripts like npm does, but instead are working on an interactive solution, that would allow users to opt-in into these scripts (potentially running in a sandboxed Deno process) or ignore them altogether.

The work on this has started, but will take several patch releases to finalize. We're starting with warnings, informing users that these scripts are currently not supported and might require manual interventation.

This is tracked in #16164.

• Make more popular packages work - this is very open-ended, but so far we agreed to work on a list prepared by @birkskyum (thank you!) that cover quite a few high profile packages - https://github.com/denoland/deno/issues/created_by/birkskyum. grpc-js/grpc and google-cloud/datastore are also targeted to be supported.

This requires polyfilling more APIs from node: builtin modules (you can see the current compatibility list at https://docs.deno.com/runtime/manual/node/compatibility).

• Support for npm workspaces - required to run monorepos locally. We are still discussing this item as workspaces are specific to the package manager used - ie. npm workspaces work differently than yarn workspaces that work differently than pnpm workspaces.

This issue will be updated, if we identify more areas we want to improve.

[rracariu]

Awesome!
This would undoubtably help deno.dev also as it allows full use of the Deno APIs such as KV and Queues when developing Next/Nuxt/SvelteKit/YouNameYourFramework web apps that get deployed with Deno deploy.

[lucacasonato]

Crypto APIs in Node are still very lackluster.

@arnauorriols and me tried to use two NPM modules this week, and both didn't work because of various crypto issues. Two issues we ran into:

• crypto.createPrivateKey() is not implemented
• crypto.createVerifier() does not support EC keys

The modules blocked by this are passport.js and web-push.

[iherman]

I hope this is the right place to report a node compatibility problem.

I have been trying to run the npm:jsdom package and I get the following error message:

error: Uncaught Error: Not implemented: isContext
    at notImplemented (ext:deno_node/_utils.ts:9:9)
    at Object.isContext (node:vm:44:3)
    at new Window (file:///Users/ivan/node_modules/.deno/jsdom@21.1.0/node_modules/jsdom/lib/jsdom/browser/Window.js:266:10)
    at exports.createWindow (file:///Users/ivan/node_modules/.deno/jsdom@21.1.0/node_modules/jsdom/lib/jsdom/browser/Window.js:96:10)
    at new JSDOM (file:///Users/ivan/node_modules/.deno/jsdom@21.1.0/node_modules/jsdom/lib/api.js:36:20)
    at file:///Users/ivan/Source/Typescript/deno/jsdom_test.ts:3:13

To be honest, I do not know what this isContext property is, but the error message seems to indicate that it is part of the node runtime...

[lenovouser]

Adding support for private npm registries would also be crucial for us. Issues are #18616, #20195 and #19900

[sant123]

Also for TLS handshakes mssql package is not working in Deno as of today v1.38.0 #20594

[ericvergnaud]

Hi there, thanks for the great work 😄.
Not sure if this belongs here or deserves a dedicated thread.
I'm looking at building modules that can run in node and/or browser.
I'm able to successfully generate a bundle using esbuild, but it won't run because it imports npm packages (using "npm:xxx" syntax).
When I add npm packages to esbuild options as: external: [ "npm:some_module" ], it will build, but at runtime the bundle tries to load a npm package called "npm:some_module", which doesn't exist.
If I don't add them, it won't build because esbuild can't find them.
How do I tell deno to drop the 'npm:' prefix when bundling for node or the browser ?

[ansarizafar]

Please implement udp.UDP.prototype.unref https://nodejs.org/api/dgram.html#socketunref I am getting this error

[ericvergnaud]

Hi there, thanks for the great work 😄. Not sure if this belongs here or deserves a dedicated thread. I'm looking at building modules that can run in node and/or browser. I'm able to successfully generate a bundle using esbuild, but it won't run because it imports npm packages (using "npm:xxx" syntax). When I add npm packages to esbuild options as: external: [ "npm:some_module" ], it will build, but at runtime the bundle tries to load a npm package called "npm:some_module", which doesn't exist. If I don't add them, it won't build because esbuild can't find them. How do I tell deno to drop the 'npm:' prefix when bundling for node or the browser ?

Found how to do that using an rebuild resolver plugin.

[ericvergnaud]

FileInfo.isFile is a boolean in Deno but a function in node.

[phoenisx]

Anyone looking for a workaround in the meantime can use deno.json overriding the importmaps for any workspaces/unresolved node imports, as follows:

{
   "imports": {
      "crypto": "node:crypto",
      "foo-package/": "../../packages/foo-package/",
      "@/": "./src/"
   }
}

This has worked for me with --unstable-sloppy-import option passed (which helps resolve imports without extension) to deno run while running any script using Deno.

[aleksanb]

Anyone looking for a workaround in the meantime can use deno.json overriding the importmaps for any workspaces/unresolved node imports, as follows:

I attempted to use this approach to workaround #18972 (comment) to no avail. I see the lib in question still fails with at SignImpl.sign (ext:deno_node/internal/crypto/sig.ts:36:33)´ and that path makes it seem like my override didn't work: "imports": {"crypto": "node:crypto"},` (in deno.json).

[bartlomieju]

With support for BYONM some of the problems listed in this issue are not so burning anymore (eg. support for postinstall scripts). We're still eye-balling npm workspace support, but again it can be worked around with BYONM. I'm going to close this issue for the time being and let's keep discussions in the respective issues.