🎯 ESM Support for PnP

[zdm]

Edit 2021-04-14 by @arcanis: In case you reach this thread by Twitter, ESM for PnP is on our roadmap. It's tricky due to some primitives currently missing in Node (they're working on this, but as you can see it's still branded as experimental), in the meantime if you need it then the node-modules linker remain a perfectly fine solution. In any case, help is appreciated, Twitter rants less.

---

What package is covered by this investigations?```
@softvisio/cli

Describe the goal of the investigation

I am trying to run script, that uses es6 syntax.

yarn add @softvisio/cli
yarn softvisio-cli

But get this error:

(node:4268) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
d:\downloads\yarn\@softvisio-cli-npm-0.1.0-efe3491d08-1.zip\node_modules\@softvisio\cli\bin\cli.js:3
import softvisioCliApi from "../lib/api.js";
^^^^^^

I am using node 13 that can load es6 modules directly.

Is it currently possible to use pure es6 modules with yarn 2 or I need to compile them to commonjs before?

[arcanis]

ESM modules go through a completely different codepath in Node (cjs / esm). The problem is that this codepath is still very new, and although modules have technically been marked stable the way to alter the resolution (aka loaders) is still in heavy discussion (nodejs/modules#351).

So we can't support them at the moment (too early), but as soon as the loader spec moves forward with an implementation we'll be ready to experiment with them 🧪

[zdm]

Thank you.
Please, don't close this issue until this feature will be implemented.

[bgotink]

I looked into the API a bit using Node 14. I've written a PoC implementation at (bgotink@34c0a7b) along with a small test repo over at https://github.com/bgotink/berry-esm.

The API is still unstable, but I think it's worth checking out to see whether there's any valuable feedback to gain from trying. I'm not sure how volatile we should consider the API, so I'm not considering "should we ship this?" at the moment, only "does it work?"

The interesting bits so far:

• It works.
• Resolving a CommonJS dependency using {format: 'commonjs'} in the getFormat hook gives us no more hooks into the file's resolution, so we can't load it from the yarn cache's zipfiles. The same goes for JSON files and {format: 'json'}. All of these need to go through {format: 'dynamic'} which is kinda sad because that means we need to replicate the Node behaviour w.r.t. named exports in these kinds of files.
• This will go nicely with support for package exports ([Case Study] Exports Support #650), where the regular .pnp.c?js file asks for CJS files and the ESM loader uses import. In fact, I would assume most packages that want to adopt ESM will do this in a backwards compatible way, in which case we need package exports to load the ESM variant of the library.

The boring bits:

• Passing the --experimental-loader flag logs a warning regardless of whether said loader is actually ever used. In other words, so to not harass users not interested in experimental ESM support we should make this opt-in
• This is probably not compatible with the native ESM loader. For instance, the loader powered by PnP is quite lenient w.r.t. file extensions, while the native ESM loader isn't.
• Some things don't work yet, e.g. import(await import.meta.resolve('typescript')) fails at the moment because the ?yarn-cache flag gets lost in the resolver
• Stuff like import(pathToFileURL(createRequire(import.meta.url).resolve('typescript'))) doesn't work.
• I've explicitly kept the pnp-loader static and use the PnP API exposed from the .pnp.c?js file

Improvements to the PnP API that would make the PoC a lot cleaner:

• Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)
• It would be great if pnp could be given an option to make file extensions required when directly referencing a file
• A function to see whether a given file path is part of the yarn cache or not would come in handy, that would allow us to use regular file: URLs instead of using a ?yarn-cache query parameter.

A node module with code below works fine in the berry root folder and showcases a lot of the basic functionality already.

// named exports from fs still work even though we patch fs
import fs, {readFileSync} from 'fs';

// JSON can be imported
import pkg                from 'typescript/package';
// CJS package import
import typescript         from 'typescript';

// importing a non-patched builtin
import {fileURLToPath}    from 'url';

console.log('');
console.log(pkg.name, typescript.version);

// Resolves to a regular file: URL
const typescriptUrl = await import.meta.resolve('typescript');
// So it can get transformed into a path and used in fs APIs or even require (BAD IDEA)
const typescriptPath = fileURLToPath(typescriptUrl);

console.log(`url:`, typescriptUrl);
console.log(`path:`, typescriptPath);

const eslintPackagePath = fileURLToPath(await import.meta.resolve('eslint/package'));

// Just to prove that the fs APIs are patched
console.log('fs.readFileSync', JSON.parse(fs.readFileSync(eslintPackagePath)).version);
console.log('readFileSync', JSON.parse(readFileSync(eslintPackagePath)).version);

$ node --experimental-loader ./.pnp-loader.mjs --experimental-import-meta-resolve --harmony-top-level-await test.mjs
(node:68602) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

typescript 3.8.3
url: file:///Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js?yarn-cache=
path: /Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js
fs.readFileSync 6.8.0
readFileSync 6.8.0

[arcanis]

Nice analysis! You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)

That seems acceptable 👍

It would be great if pnp could be given an option to make file extensions required when directly referencing a file

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

[bgotink]

You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

I've kept the module loader conservative in which paths it actually tries to handle itself versus which paths it lets the default loader handle. This requires the different hooks to be able to check whether a URL points to a path that needs to be loaded via the patched filesystem.
The ?yarn-cache query param was a quick and dirty solution for that. Another possibility is an API to see whether a given path requires loading via our patched fs module. Alternatively we could just load all files via the patched fs, though that makes the loader anything but conservative.
In a first attempt for the loader I used a custom protocol yarn-cache:, but it's a bad idea to stray away from the file: protocol for compatibility reasons: import.meta.resolve(...) and import.meta.url expose the URL of a module, changing that from a file: URL to something else will cause breakage for people using the module's URL in order to e.g. load a data file in the same folder as the code file.

I'm not sure what's the best path forward here: just load everything via our getSource hook or defer to the native hook when possible? (in the latter case we need a mechanism to devise whether a URL should be loaded by us, beit via the current ?yarn-cache query parameter or via an API call)

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

resolveToUnqualified doesn't resolve bare imports to their actual files, it resolves typescript to /.../berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript.
At the moment the PoC loader uses resolveQualified to resolve that to an actual file, but that implements too much: it automatically adds file extensions and it resolves folders to files called index.whatever.

As I've stated in my previous comment I've kept the module loader as dumb as possible, it loads everything from the regular .pnp.c?js file. Implementing both the CJS resolution algorithm and the ESM resolution algorithm in the regular pnp file is probably not the best idea. Maybe the solution is to use resolveToUnqualified and then handle the rest of the resolution in the module loader?

[dezren39]

What is the timeline for enabling ES support? Is there anything one can do to work around this? The issues linked to this suggest not. For now, I will try using berry with the nodeLinker option and see if that lets me use berry, though with it's biggest feature limited lol. The reason I want pnp is essentially what you've labeled 'zero installs', before this I used yarn 1 and the offline mirror mode to make exactly the same .zip folders that berry makes, even in the .yarn folder and everything lol allowing 1 zip per dependency and sane commit changes by just checking the .zips. Built it all with docker images lol. So I'd hoped berry would be able to work in the same way, but since there actually isn't ever a node_modules with this method, I see how it will be more difficult to patch, because the offline-mirror was just how it got stored in git, but once running it was identical to npm/node with a node_modules folder.

[desmap]

@arcanis @bgotink any news here? @arcanis left the last comment on the related thread nodejs/modules#351 on on Jul 29, 2019, so is there some hope?

[arcanis]

Of course there's hope, and it's even certain it'll eventually happen. It's however not my top priority at the moment since all the software I personally use are still CommonJS, so it'll happen faster if someone can do the legwork 🙂

[desmap]

I personally use are still CommonJS

Most of the stuff I use is also CommonJS, I agree. However, once you start to code share between frontend and backend and already bigger libs like Next default to esnext modules it's just cleaner and less headache to have all the code base as one format.

Currently, code sharing is a rocket science. But that's not your fault ;)

[desmap]

or another example which just popped up again in my project: top-level awaits, they are such a life saver and way underrated. Once you get the hang on them you can do insane stuff. Right now I try to setup conditional imports (which are of course async) and then continue depending on the condition. With top-level it would be DRY, without, a lot of .thens, code dupes and just messy.

But TS requires esnext modules for using top-level await... so yeah

[arcanis]

Hey @nnmrts - before commenting in an open source project (any open source project), please remember that the people in front of you aren't your contractors but your peers. Passive aggressiveness like "if you don't do X I won't use Y" isn't more likely to make X happen, because maintainers don't work for you (at least I haven't received the wire yet).

That being said, if you need to use ESM now, you just have to downgrade to typical node_modules installs using the nodeLinker setting and you'll have your work done for you. If you need to use ESM with PnP then it's perfect as we happen to be looking for someone with free time to help us on this. Ping us on Discord and we'll put your energy to good use for the community! 🙂

[merceyz]

I gave this a go and implemented an experimental loader in #2161, if anyone would like to test/try it you can use yarn set version from sources --branch 2161

[eric-burel]

Hello from 2022, I feel at tad lost among all tickets and doc, am I supposed to do something to have for instance an index.mjs files that imports zx/globals? I hit Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'zx' for instance.
At the moment I fallback to nodeLinker: node_modules and it works ok at least.

[merceyz]

If there are no indicators in your project other than file extensions that ESM support is required you'll need to manually enable it.

pnpEnableEsmLoader: true

https://yarnpkg.com/configuration/yarnrc#pnpEnableEsmLoader

[eric-burel]

Adding type: "module" in package.json also generate an ESM loader.
It seems to be documented only in this blog post atm: https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-packages--3hak#esm-support

However, maybe a dumb question: how do you actually use this loader? I now have a .pnpn.loader.mjs but zx package is still not found when I run node index.mjs. I guess I should also register this loader somewhere? Am I supposed to use manually the functions it exports?

[kristian]

node -r .pnp.js --loader .pnpn.loader.mjs index.mjs

Or just use:

yarn node index.mjs

Which does add all required parameters for you.

[eric-burel]

Perfect thank you!
If I should sum it up for future googlers:

• If you are migrating from nodeLinker: node-modules, and still have a "node_modules" folder, you might want to add nodeLinker: pnp explicitely so Yarn does the switch (otherwise if you just remove the nodeLinker option, it might still think you are using node modules).
• Yarn will detect type: "module" in your package.json. If you don't want to add this property, instead in .yarnrc.yml add pnpEnableEsmLoader: true
• It generates a loader, but you must still use it explictely, with either node -r .pnp.js --loader .pnp.loader.mjs index.mjs (most portable) or yarn node index.mjs
• ESM is still experimental, for instance I hit this issue [Feature] Support the imports field in the PnP API #3843 with "chalk" which is also used by "zx". But at least I could setup PNP

[nerdgore]

Thanks for the write up @eric-burel!
This is a great workaround for code dependencies.
Is it possible to somehow use this loader for scripts? In my case I am trying to run plop through a package script

[Samlv9]

Not works when i am trying to use a custom loader.

/// file: custom.loader.ts
import { extname } from "node:path";
import { create, createEsmHooks, NodeLoaderHooksAPI2, NodeLoaderHooksFormat } from "ts-node";

interface NodeLoaderHooksAPI2Context {
    conditions?: NodeLoaderHooksAPI2.NodeImportConditions;
    importAssertions?: NodeLoaderHooksAPI2.NodeImportAssertions;
    parentURL: string;
}

interface NodeLoaderHooksAPI2ReturnType {
    url: string;
    format?: NodeLoaderHooksFormat;
    shortCircuit?: boolean;
}

const { resolve: tsnodeResolve } = createEsmHooks(create());

export function resolve(specifier: string, context: NodeLoaderHooksAPI2Context, defaultResolve: NodeLoaderHooksAPI2.ResolveHook): Promise<NodeLoaderHooksAPI2ReturnType> {
    switch (extname(specifier)) {
        case ".ts" :
        case ".tsx" :
            return tsnodeResolve(specifier, context, defaultResolve)

        default: break;
    }

    return defaultResolve(specifier, context, defaultResolve);
}

$ yarn tsc --target esnext --moduleResolution node ./custom.loader.ts
$ yarn node --loader ./custom.loader.js --no-warnings --experimental-json-modules --experimental-import-meta-resolve ./test.ts

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ts-node' imported from xxxxxx
Did you mean to import ts-node-virtual-bba6bca08f/0/cache/ts-node-npm-10.8.0-e60a0a9a4f-1c22dc8dd8.zip/node_modules/ts-node/dist/index.js?
    at new NodeError (internal/errors.js:322:7)
    at packageResolve (internal/modules/esm/resolve.js:732:9)
    at moduleResolve (internal/modules/esm/resolve.js:773:18)
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:887:11)
    at Loader.resolve (internal/modules/esm/loader.js:89:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:242:28)
    at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:76:40)
    at link (internal/modules/esm/module_job.js:75:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

[merceyz]

@Samlv9 See #4044