Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: clarify module system selection #41383

Merged
merged 14 commits into from
Jan 17, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Command-line options
# Command-line API

<!--introduced_in=v5.9.1-->

Expand All @@ -11,16 +11,42 @@ To view this documentation as a manual page in a terminal, run `man node`.

## Synopsis

`node [options] [V8 options] [script.js | -e "script" | -] [--] [arguments]`
`node [options] [V8 options] [<program-entry-point> | -e "script" | -] [--] [arguments]`

`node inspect [script.js | -e "script" | <host>:<port>] …`
`node inspect [<program-entry-point> | -e "script" | <host>:<port>] …`

`node --v8-options`

Execute without arguments to start the [REPL][].

For more info about `node inspect`, see the [debugger][] documentation.

## Program entry point

The program entry point is a specifier-like string. If the string is not an
absolute path, it's resolved as a relative path from the current working
directory. That path is then resolved by [CommonJS][] module loader. If no
corresponding file is found, an error is thrown.

If a file is found, its path will be passed to the [ECMAScript module loader][]
under any of the following conditions:

* The file has an `.mjs` extension.
* The file nearest parent `package.json` file
aduh95 marked this conversation as resolved.
Show resolved Hide resolved
contains a top-level [`"type"`][] field with a value of `"module"`.
* If the program was started with a command-line flag that forces the entry
point to be loaded with ECMAScript module loader.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should explicitly refer to --input-type. I know that --loader also currently makes the entry point esm, but that's a bug (cf #33226, cc @cspotcode).

Formalizing this behaviour on --loader is problematic because it means whoever sets must also be sure that the entry point is esm. This isn't always the case: NODE_OPTIONS is typically set long before node is ever called, and by processes which aren't necessarily aware of what the entry point will be (or even if there will only be one of them).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To save readers some time, this is the comment where we established that it is, indeed, a bug: #33226 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is trying to describe the current behavior of Node.js, not its final or ideal state. I don't call out which CLI flags do trigger this behavior precisely because of that discussion you linked to. --experimental-specifier-resolution=node also triggers this behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nature of the fix isn't clear yet so idk if this PR needs to fix it right now. I wouldn't want to block this PR based upon #33226 .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only meant that documenting the interaction between --loader and entry points (even if --loader isn't explicitly referred to in this paragraph) may have a slight risk of suggesting to readers that it can be relied upon and that it'd be a regression to fix it. While loaders are experimental, I noticed people often expect experimental-induced breaking changes to be very visible in their API (like the hook refactoring) rather than subtle under-the-hood behavioural changes.

Note that this wasn't a blocker by any means, just a suggestion.


Otherwise, the file is loaded using the CommonJS module loader. See
[Modules loaders][] for more details.

### ECMAScript modules loader entry point caveat

When loading [ECMAScript module loader][] loads the program entry point, the `node`
command will only accept as input only files with `.js`, `.mjs`, or `.cjs`
extensions; and with `.wasm` extensions when
[`--experimental-wasm-modules`][] is enabled.

## Options

<!-- YAML
Expand Down Expand Up @@ -277,8 +303,8 @@ Enable experimental JSON support for the ES Module loader.
added: v9.0.0
-->

Specify the `module` of a custom experimental [ECMAScript Module loader][].
`module` may be either a path to a file, or an ECMAScript Module name.
Specify the `module` of a custom experimental [ECMAScript module loader][].
`module` may be any string accepted as an [`import` specifier][].

### `--experimental-policy`

Expand Down Expand Up @@ -1928,15 +1954,19 @@ $ node --max-old-space-size=1536 index.js
```

[Chrome DevTools Protocol]: https://chromedevtools.github.io/devtools-protocol/
[ECMAScript Module loader]: esm.md#loaders
[CommonJS]: modules.md
[ECMAScript module loader]: esm.md#loaders
[Modules loaders]: packages.md#modules-loaders
[OSSL_PROVIDER-legacy]: https://www.openssl.org/docs/man3.0/man7/OSSL_PROVIDER-legacy.html
[REPL]: repl.md
[ScriptCoverage]: https://chromedevtools.github.io/devtools-protocol/tot/Profiler#type-ScriptCoverage
[Source Map]: https://sourcemaps.info/spec.html
[Subresource Integrity]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity
[V8 JavaScript code coverage]: https://v8project.blogspot.com/2017/12/javascript-code-coverage.html
[`"type"`]: packages.md#type
[`--cpu-prof-dir`]: #--cpu-prof-dir
[`--diagnostic-dir`]: #--diagnostic-dirdirectory
[`--experimental-wasm-modules`]: #--experimental-wasm-modules
[`--heap-prof-dir`]: #--heap-prof-dir
[`--openssl-config`]: #--openssl-configfile
[`--redirect-warnings`]: #--redirect-warningsfile
Expand All @@ -1949,6 +1979,7 @@ $ node --max-old-space-size=1536 index.js
[`dns.lookup()`]: dns.md#dnslookuphostname-options-callback
[`dns.setDefaultResultOrder()`]: dns.md#dnssetdefaultresultorderorder
[`dnsPromises.lookup()`]: dns.md#dnspromiseslookuphostname-options
[`import` specifier]: esm.md#import-specifiers
[`process.setUncaughtExceptionCaptureCallback()`]: process.md#processsetuncaughtexceptioncapturecallbackfn
[`tls.DEFAULT_MAX_VERSION`]: tls.md#tlsdefault_max_version
[`tls.DEFAULT_MIN_VERSION`]: tls.md#tlsdefault_min_version
Expand Down
12 changes: 7 additions & 5 deletions doc/api/esm.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,12 @@ provides interoperability between them and its original module format,

<!-- type=misc -->

Node.js treats JavaScript code as CommonJS modules by default.
Authors can tell Node.js to treat JavaScript code as ECMAScript modules
Node.js has two module systems: [CommonJS][] modules and ECMAScript modules.

Authors can tell Node.js to use the ECMAScript modules loader
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Authors can tell Node.js to use the ECMAScript modules loader
Authors can tell Node.js to treat JavaScript code as ECMAScript modules

I just don’t think the docs should have any references to “the CommonJS modules loader” or the “ECMAScript modules loader”—no average user knows what those are. There’s just Node, and how it interprets source code.

Copy link
Contributor Author

@aduh95 aduh95 Jan 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK but using the ES module loader has more effect than on simply JS code (it no longer accepts .node and .json file as entry point, it no longer treats extensionless / unknown extension as JS files), which was the nuance I was trying to communicate here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add those other implications here, then. Most people reading these docs wouldn't know about those nuances just because the loader is mentioned.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried to address that in ce3edd0, PTAL.

via the `.mjs` file extension, the `package.json` [`"type"`][] field, or the
aduh95 marked this conversation as resolved.
Show resolved Hide resolved
`--input-type` flag. See
[Modules: Packages](packages.md#determining-module-system) for more
details.
[`--input-type`][] flag. Outside of those cases, Node.js will use the CommonJS
module loader. See [Determining module system][] for more details.
aduh95 marked this conversation as resolved.
Show resolved Hide resolved

<!-- Anchors to make sure old links find a target -->

Expand Down Expand Up @@ -1425,6 +1425,7 @@ success!
[CommonJS]: modules.md
[Conditional exports]: packages.md#conditional-exports
[Core modules]: modules.md#core-modules
[Determining module system]: packages.md#determining-module-system
[Dynamic `import()`]: https://wiki.developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#Dynamic_Imports
[ECMAScript Top-Level `await` proposal]: https://github.com/tc39/proposal-top-level-await/
[ES Module Integration Proposal for WebAssembly]: https://github.com/webassembly/esm-integration
Expand All @@ -1437,6 +1438,7 @@ success!
[WHATWG JSON modules specification]: https://html.spec.whatwg.org/#creating-a-json-module-script
[`"exports"`]: packages.md#exports
[`"type"`]: packages.md#type
[`--input-type`]: cli.md#--input-typetype
[`ArrayBuffer`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer
[`SharedArrayBuffer`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer
[`TypedArray`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray
Expand Down
35 changes: 34 additions & 1 deletion doc/api/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,38 @@ module.exports = class Square {
};
```

The module system is implemented in the `require('module')` module.
The CommonJS module system is implemented in the [`module` core module][].

## Enabling

<!-- type=misc -->

Node.js has two module systems: CommonJS modules and [ECMAScript modules][].

By default, Node.js will treat the following as CommonJS modules:

* Files with a `.cjs` extension;

* Files with a `.js` extension when the nearest parent `package.json` file
contains a top-level field [`"type"`][] with a value of `"commonjs"`.

* Files with a `.js` extension when the nearest parent `package.json` file
doesn't contain a top-level field [`"type"`][]. Package authors should include
the [`"type"`][] field, even in packages where all sources are CommonJS. Being
explicit about the `type` of the package will make things easier for build
tools and loaders to determine how the files in the package should be
interpreted.

* Files with an extension that is not `.mjs`, `.cjs`, `.json`, `.node`, or `.js`
(when the nearest parent `package.json` file contains a top-level field
[`"type"`][] with a value of `"module"`, those files will be recognized as
CommonJS modules only if they are being `require`d, not when used as the
command-line entry point of the program).

See [Determining module system][] for more details.

Calling `require()` always use the CommonJS module loader. Calling `import()`
always use the ECMAScript module loader.

## Accessing the main module

Expand Down Expand Up @@ -1047,13 +1078,15 @@ This section was moved to
[ECMAScript Modules]: esm.md
[GLOBAL_FOLDERS]: #loading-from-the-global-folders
[`"main"`]: packages.md#main
[`"type"`]: packages.md#type
[`ERR_REQUIRE_ESM`]: errors.md#err_require_esm
[`Error`]: errors.md#class-error
[`__dirname`]: #__dirname
[`__filename`]: #__filename
[`import()`]: https://wiki.developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#Dynamic_Imports
[`module.children`]: #modulechildren
[`module.id`]: #moduleid
[`module` core module]: module.md
[`module` object]: #the-module-object
[`package.json`]: packages.md#nodejs-packagejson-field-definitions
[`path.dirname()`]: path.md#pathdirnamepath
Expand Down
62 changes: 54 additions & 8 deletions doc/api/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,13 @@ along with a reference for the [`package.json`][] fields defined by Node.js.
## Determining module system

Node.js will treat the following as [ES modules][] when passed to `node` as the
initial input, or when referenced by `import` statements within ES module code:
initial input, or when referenced by `import` statements or `import()`
expressions:

* Files ending in `.mjs`.
* Files with an `.mjs` extension.

* Files ending in `.js` when the nearest parent `package.json` file contains a
top-level [`"type"`][] field with a value of `"module"`.
* Files with a `.js` extension when the nearest parent `package.json` file
contains a top-level [`"type"`][] field with a value of `"module"`.

* Strings passed in as an argument to `--eval`, or piped to `node` via `STDIN`,
with the flag `--input-type=module`.
Expand All @@ -67,12 +68,13 @@ field, or string input without the flag `--input-type`. This behavior is to
preserve backward compatibility. However, now that Node.js supports both
CommonJS and ES modules, it is best to be explicit whenever possible. Node.js
will treat the following as CommonJS when passed to `node` as the initial input,
or when referenced by `import` statements within ES module code:
or when referenced by `import` statements, `import()` expressions, or
`require()` expressions:
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved

* Files ending in `.cjs`.
* Files with a `.cjs` extension.

* Files ending in `.js` when the nearest parent `package.json` file contains a
top-level field [`"type"`][] with a value of `"commonjs"`.
* Files with a `.js` extension when the nearest parent `package.json` file
contains a top-level field [`"type"`][] with a value of `"commonjs"`.

* Strings passed in as an argument to `--eval` or `--print`, or piped to `node`
via `STDIN`, with the flag `--input-type=commonjs`.
Expand All @@ -83,6 +85,47 @@ future-proof the package in case the default type of Node.js ever changes, and
it will also make things easier for build tools and loaders to determine how the
files in the package should be interpreted.

### Modules loaders

Node.js has two systems for resolving a specifier and loading modules.

There is the CommonJS module loader:

* It is fully synchronous.
* It is responsible for handling `require()` calls.
* It is monkey patchable.
* When resolving a specifier, if no exact match is found, it will try to add
extensions (`.js`, `.json`, and finally `.node`).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* When resolving a specifier, if no exact match is found, it will try to add
extensions (`.js`, `.json`, and finally `.node`).
* When resolving a specifier, if no exact match is found, it will try to add
extensions (`.js`, `.json`, and finally `.node`) and index files
(`/index.js`, `/index.json`, and finally `/index.node`).

Copy link
Contributor Author

@aduh95 aduh95 Jan 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's true, if the specifier points to an existing directory, then an exact match is found – we have a Folders as modules section that explains it. Otherwise, we would have to specify it first tries to load the "main" field if the directory contains a package.json file, we might as well just link to that section (EDIT: which I've done in the very next item, I think we should leave it as is)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then maybe mention the folder resolution without spelling it out?

* When resolving a specifier, if no exact match is found, it will try to add
  extensions (`.js`, `.json`, and finally `.node`) and then attempt to resolve
  [folders as modules][].

Copy link
Contributor Author

@aduh95 aduh95 Jan 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that work?

node/doc/api/packages.md

Lines 97 to 99 in 93d6a23

* When resolving a specifier, if no exact match is found, it will try to add
extensions (`.js`, `.json`, and finally `.node`).
* It supports [folders as modules][].

EDIT: no you're right, we should document the order here.

node/doc/api/packages.md

Lines 97 to 100 in fe05b72

* It supports [folders as modules][].
* When resolving a specifier, if no exact match is found, it will try to add
extensions (`.js`, `.json`, and finally `.node`) and then attempt to resolve
[folders as modules][].

* It supports [folders as modules][].
* It treats `.json` as JSON text files.
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved
* `.node` files are interpreted as compiled addon modules loaded with
`process.dlopen()`.
* It treats all files that lack `.json` or `.node` extensions as JavaScript
text files.
* It cannot be used to load ECMAScript modules (although it is possible to
[load ECMASCript modules from CommonJS modules][]). When used to load a
JavaScript text file that is not an ECMAScript module, it loads it as a
CommonJS module.

There is the ECMAScript module loader:

* It is asynchronous.
* It is responsible for handling `import` statements and `import()` expressions.
* It is not monkey patchable, can be customized using [loader hooks][].
* It does no extension searching. A file extension must be provided
when the specifier is a relative or absolute file URL.
* It does not support folders as modules, directory indexes (e.g.
`'./startup/index.js'`) must be fully specified.
* It can load JSON modules, but an import assertion is required (behind
`--experimental-json-modules` flag).
* It accepts only `.js`, `.mjs`, and `.cjs` extensions for JavaScript text
files.
* It can be used to load JavaScript CommonJS modules. Such modules
are passed through the `es-module-lexer` to try to identify named exports,
which are available if they can be determined through static analysis.
Imported CommonJS modules have their URLs converted to absolute
paths and are then loaded via the CommonJS module loader.

aduh95 marked this conversation as resolved.
Show resolved Hide resolved
### `package.json` and file extensions

Within a package, the [`package.json`][] [`"type"`][] field defines how
Expand Down Expand Up @@ -1236,6 +1279,9 @@ This field defines [subpath imports][] for the current package.
[`esm`]: https://github.com/standard-things/esm#readme
[`package.json`]: #nodejs-packagejson-field-definitions
[entry points]: #package-entry-points
[folders as modules]: modules.md#folders-as-modules
[load ECMASCript modules from CommonJS modules]: modules.md#the-mjs-extension
[loader hooks]: esm.md#loaders
[self-reference]: #self-referencing-a-package-using-its-name
[subpath exports]: #subpath-exports
[subpath imports]: #subpath-imports
Expand Down
2 changes: 1 addition & 1 deletion doc/api/synopsis.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,6 @@ Now, open any preferred web browser and visit `http://127.0.0.1:3000`.
If the browser displays the string `Hello, World!`, that indicates
the server is working.

[Command-line options]: cli.md#command-line-options
[Command-line options]: cli.md#options
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved
[Installing Node.js via package manager]: https://nodejs.org/en/download/package-manager/
[web server]: http.md
4 changes: 2 additions & 2 deletions tools/doc/links-mapper.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"doc/api/synopsis.md": {
"command line options": "cli.html#command-line-options",
"command line options": "cli.html#options",
"web server": "http.html"
}
}
}