Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

esm: remove CLI flag limitation to programmatic registration #48439

Closed
wants to merge 13 commits into from
Closed

esm: remove CLI flag limitation to programmatic registration #48439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
28055ff
esm: remove CLI flag limitation to programmatic registration
JakobJingleheimer Jun 12, 2023
d4a5dd8
WIP: disentangle `loader` from `ModuleJob`
JakobJingleheimer Jun 12, 2023
378c5a5
WIP: continue to detangle ModuleJob & ESMLoader
JakobJingleheimer Jun 15, 2023
d2ad85b
WIP: add debugs
JakobJingleheimer Jun 18, 2023
02c6090
WIP: await getting module job (now always async)
JakobJingleheimer Jun 18, 2023
cb72d94
WIP: move initialize import.meta out of module loader
JakobJingleheimer Jun 18, 2023
77e91dc
WIP: tweak debugs
JakobJingleheimer Jun 18, 2023
582f50a
correct ModuleJob(`url` → `responseURL`)
JakobJingleheimer Jun 18, 2023
ec5d529
move `setCallbackForWrap` back into `translator('module')`
JakobJingleheimer Jun 18, 2023
e6e8cac
fix: restore `setCallbackForWrap` in `ModuleLoader::eval()`
JakobJingleheimer Jun 19, 2023
3819d08
fix: restore gate of `import.meta.resolve` within loader worker
JakobJingleheimer Jun 19, 2023
9ff1de7
fix: existing `job` detection
JakobJingleheimer Jun 22, 2023
6fa7dc7
WIP: investigate missing/tardy job entry in ModuleMap
JakobJingleheimer Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions lib/internal/errors.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1036,11 +1036,6 @@ E('ERR_ENCODING_INVALID_ENCODED_DATA', function(encoding, ret) {
}, TypeError);
E('ERR_ENCODING_NOT_SUPPORTED', 'The "%s" encoding is not supported',
RangeError);
E('ERR_ESM_LOADER_REGISTRATION_UNAVAILABLE', 'Programmatically registering custom ESM loaders ' +
'currently requires at least one custom loader to have been registered via the --experimental-loader ' +
'flag. A no-op loader registered via CLI is sufficient (for example: `--experimental-loader ' +
'"data:text/javascript,"` with the necessary trailing comma). A future version of Node.js ' +
'will remove this requirement.', Error);
E('ERR_EVAL_ESM_CANNOT_PRINT', '--print cannot be used with ESM input', Error);
E('ERR_EVENT_RECURSION', 'The event "%s" is already being dispatched', Error);
E('ERR_FALSY_VALUE_REJECTION', function(reason) {
Expand Down
24 changes: 16 additions & 8 deletions lib/internal/modules/esm/initialize_import_meta.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,26 @@

const { getOptionValue } = require('internal/options');
const experimentalImportMetaResolve = getOptionValue('--experimental-import-meta-resolve');
let debug = require('internal/util/debuglog').debuglog('esm', (fn) => {
debug = fn;
});

/**
* Generate a function to be used as import.meta.resolve for a particular module.
* @param {string} defaultParentUrl The default base to use for resolution
* @param {typeof import('./loader.js').ModuleLoader} loader Reference to the current module loader
* @returns {(specifier: string, parentUrl?: string) => string} Function to assign to import.meta.resolve
*/
function createImportMetaResolve(defaultParentUrl, loader) {
function createImportMetaResolve(defaultParentUrl) {
debug('createImportMetaResolve(): %o', { defaultParentUrl });

return function resolve(specifier, parentUrl = defaultParentUrl) {
const moduleLoader = require('internal/process/esm_loader').esmLoader;
let url;

debug('import.meta.resolve(%o) %s', { specifier, parentUrl }, moduleLoader.constructor.name);

try {
({ url } = loader.resolve(specifier, parentUrl));
({ url } = moduleLoader.resolve(specifier, parentUrl));
} catch (error) {
if (error?.code === 'ERR_UNSUPPORTED_DIR_IMPORT') {
({ url } = error);
Expand All @@ -31,15 +38,16 @@ function createImportMetaResolve(defaultParentUrl, loader) {
* Create the `import.meta` object for a module.
* @param {object} meta
* @param {{url: string}} context
* @param {typeof import('./loader.js').ModuleLoader} loader Reference to the current module loader
* @returns {{url: string, resolve?: Function}}
*/
function initializeImportMeta(meta, context, loader) {
const { url } = context;
function initializeImportMeta(meta, { url }) {
debug('initializeImportMeta for %s', url);

const { isLoaderWorker } = require('internal/modules/esm/utils');

// Alphabetical
if (experimentalImportMetaResolve && loader.loaderType !== 'internal') {
meta.resolve = createImportMetaResolve(url, loader);
if (experimentalImportMetaResolve && !isLoaderWorker()) {
meta.resolve = createImportMetaResolve(url);
}

meta.url = url;
Expand Down
205 changes: 109 additions & 96 deletions lib/internal/modules/esm/loader.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,19 @@ const {
} = primordials;

const {
ERR_ESM_LOADER_REGISTRATION_UNAVAILABLE,
ERR_UNKNOWN_MODULE_FORMAT,
} = require('internal/errors').codes;
const { getOptionValue } = require('internal/options');
const { pathToFileURL } = require('internal/url');
const { emitExperimentalWarning } = require('internal/util');
const { emitExperimentalWarning, kEmptyObject } = require('internal/util');
const {
getDefaultConditions,
} = require('internal/modules/esm/utils');
let defaultResolve, defaultLoad, importMetaInitializer;
let defaultResolve;
let defaultLoad;
let debug = require('internal/util/debuglog').debuglog('esm', (fn) => {
debug = fn;
});

function newModuleMap() {
const ModuleMap = require('internal/modules/esm/module_map');
Expand Down Expand Up @@ -99,22 +102,28 @@ class DefaultModuleLoader {
source,
url = pathToFileURL(`${process.cwd()}/[eval${++this.evalIndex}]`).href,
) {
const evalInstance = (url) => {
const { ModuleWrap } = internalBinding('module_wrap');
const { setCallbackForWrap } = require('internal/modules/esm/utils');
const module = new ModuleWrap(url, undefined, source, 0, 0);
setCallbackForWrap(module, {
initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, { url }),
importModuleDynamically: (specifier, { url }, importAssertions) => {
return this.import(specifier, url, importAssertions);
},
});

return module;
};
const { ModuleWrap } = internalBinding('module_wrap');
const moduleWrapper = new ModuleWrap(url, undefined, source, 0, 0);
const { setCallbackForWrap } = require('internal/modules/esm/utils');
setCallbackForWrap(moduleWrapper, {
initializeImportMeta(meta, wrap) {
importMetaInitializer ??= require('internal/modules/esm/initialize_import_meta').initializeImportMeta;
importMetaInitializer(meta, { url });
},
importModuleDynamically(specifier, { url }, importAssertions) {
const moduleLoader = require('internal/process/esm_loader').esmLoader;
debug('ModuleLoader::eval::importModuleDynamically(%o)', { specifier, url, moduleLoader });
return moduleLoader.import(specifier, url, importAssertions);
},
});
const ModuleJob = require('internal/modules/esm/module_job');
const job = new ModuleJob(
this, url, undefined, evalInstance, false, false);
url,
moduleWrapper,
undefined,
false, // TODO: why isMain false in eval? it is the root module
false,
);
this.moduleMap.set(url, undefined, job);
const { module } = await job.run();

Expand Down Expand Up @@ -142,19 +151,26 @@ class DefaultModuleLoader {
return this.getJobFromResolveResult(resolveResult, parentURL, importAssertions);
}

getJobFromResolveResult(resolveResult, parentURL, importAssertions) {
async getJobFromResolveResult(
resolveResult,
parentURL,
importAssertions = resolveResult.importAssertions,
) {
const { url, format } = resolveResult;
const resolvedImportAssertions = resolveResult.importAssertions ?? importAssertions;

let job = this.moduleMap.get(url, resolvedImportAssertions.type);

// CommonJS will set functions for lazy job evaluation.
if (typeof job === 'function') {
this.moduleMap.set(url, undefined, job = job());
// `job` may be a Promise, which the engine INSANELY _sometimes_ considers undefined and
// sometimes not. This is why we use `has` instead of `get` to check for its existence.
// ! Do NOT try to check against its value; the engine will **gladly** screw you over.
if (!this.moduleMap.has(url, importAssertions.type)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm confused why a map's "has" function would ever take more than one argument. is moduleMap something that's not a Map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! I also had to ask what this is.

The extra argument (import assertion type) is per spec required for matching cache. Under the hood, the extra argument is stored as part of the value (or maybe the key?).

I'll add some code doc for it after I get this stable/passing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it’s very confusing to have something called “map” whose method names violate the expected interface contract for a Map, but i assume that predates this PR. (iow, i think instead of using has/set/etc it should have brand new method names)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this PR does not introduce ModuleMap ;)

debug('getJobFromResolveResult: did NOT find existing entry for %s', url)
return this.#createModuleJob(url, parentURL, importAssertions, format);
}

if (job === undefined) {
job = this.#createModuleJob(url, resolvedImportAssertions, parentURL, format);
let job = this.moduleMap.get(url, importAssertions.type);
debug('getJobFromResolveResult: found existing entry for %s %o', url, job);

if (typeof job === 'function') { // CommonJS will set functions for lazy job evaluation.
this.moduleMap.set(url, undefined, job = job());
JakobJingleheimer marked this conversation as resolved.
Show resolved Hide resolved
}

return job;
Expand All @@ -163,34 +179,13 @@ class DefaultModuleLoader {
/**
* Create and cache an object representing a loaded module.
* @param {string} url The absolute URL that was resolved for this module
* @param {Record<string, string>} importAssertions Validations for the
* module import.
* @param {string} [parentURL] The absolute URL of the module importing this
* one, unless this is the Node.js entry point
* @param {string} [format] The format hint possibly returned by the
* `resolve` hook
* @param {string} [parentURL] The absolute URL of the module importing this one, unless this is
* the Node.js entry point
* @param {Record<string, string>} importAssertions Validations for the module import.
* @param {string} [format] The format hint possibly returned by the `resolve` hook
* @returns {Promise<ModuleJob>} The (possibly pending) module job
*/
#createModuleJob(url, importAssertions, parentURL, format) {
const moduleProvider = async (url, isMain) => {
const {
format: finalFormat,
responseURL,
source,
} = await this.load(url, {
format,
importAssertions,
});

const translator = getTranslators().get(finalFormat);

if (!translator) {
throw new ERR_UNKNOWN_MODULE_FORMAT(finalFormat, responseURL);
}

return FunctionPrototypeCall(translator, this, responseURL, source, isMain);
};

async #createModuleJob(url, parentURL, importAssertions, format) {
const inspectBrk = (
parentURL === undefined &&
getOptionValue('--inspect-brk')
Expand All @@ -200,16 +195,40 @@ class DefaultModuleLoader {
process.send({ 'watch:import': [url] });
}

debug('#createModuleJob (before load)', { url, parentURL, importAssertions, format })

// ! The module needing creation is not added to ModuleMap before the next `getModuleJob()`,
// ! which is a problem.
const {
format: finalFormat,
responseURL,
source,
} = await this.load(url, {
format,
importAssertions,
});

const translator = getTranslators().get(finalFormat);

if (!translator) {
throw new ERR_UNKNOWN_MODULE_FORMAT(finalFormat, responseURL);
}

const isMain = parentURL === undefined;
debug('#createModuleJob (loaded, untranslated) %o', { url, finalFormat, responseURL, source });
const moduleWrapper = FunctionPrototypeCall(translator, this, responseURL, source, isMain);

const ModuleJob = require('internal/modules/esm/module_job');
const job = new ModuleJob(
this,
url,
url, // TODO: should this be resolvedURL?
moduleWrapper,
importAssertions,
moduleProvider,
parentURL === undefined,
isMain,
inspectBrk,
);

debug('#createModuleJob (translated) %o', { url, job });

this.moduleMap.set(url, importAssertions.type, job);

return job;
Expand All @@ -224,8 +243,8 @@ class DefaultModuleLoader {
* module import.
* @returns {Promise<ModuleExports>}
*/
async import(specifier, parentURL, importAssertions) {
const moduleJob = this.getModuleJob(specifier, parentURL, importAssertions);
async import(specifier, parentURL = undefined, importAssertions = { __proto__: null }) {
const moduleJob = await this.getModuleJob(specifier, parentURL, importAssertions);
const { module } = await moduleJob.run();
return module.getNamespace();
}
Expand Down Expand Up @@ -271,12 +290,6 @@ class DefaultModuleLoader {
require('internal/modules/esm/load').throwUnknownModuleFormat(url, format);
}
}

importMetaInitialize(meta, context) {
importMetaInitializer ??= require('internal/modules/esm/initialize_import_meta').initializeImportMeta;
meta = importMetaInitializer(meta, context, this);
return meta;
}
}
ObjectSetPrototypeOf(DefaultModuleLoader.prototype, null);

Expand All @@ -285,10 +298,21 @@ class CustomizedModuleLoader extends DefaultModuleLoader {
/**
* Instantiate a module loader that uses user-provided custom loader hooks.
*/
constructor() {
constructor({
cjsCache,
evalIndex,
moduleMap,
} = kEmptyObject) {
super();

getHooksProxy();
if (cjsCache != null) { this.cjsCache = cjsCache; }
if (evalIndex != null) { this.evalIndex = evalIndex; }
if (moduleMap != null) { this.moduleMap = moduleMap; }

if (!hooksProxy) {
const { HooksProxy } = require('internal/modules/esm/hooks');
hooksProxy = new HooksProxy();
}
}

/**
Expand Down Expand Up @@ -318,12 +342,12 @@ class CustomizedModuleLoader extends DefaultModuleLoader {

async #getModuleJob(specifier, parentURL, importAssertions) {
const resolveResult = await hooksProxy.makeAsyncRequest('resolve', specifier, parentURL, importAssertions);

debug('#getModuleJob', { [specifier]: resolveResult })
return this.getJobFromResolveResult(resolveResult, parentURL, importAssertions);
}
getModuleJob(specifier, parentURL, importAssertions) {
const jobPromise = this.#getModuleJob(specifier, parentURL, importAssertions);

debug('getModuleJob', { [specifier]: jobPromise })
return {
run() {
return PromisePrototypeThen(jobPromise, (job) => job.run());
Expand Down Expand Up @@ -357,40 +381,29 @@ let emittedExperimentalWarning = false;
* A loader instance is used as the main entry point for loading ES modules. Currently, this is a singleton; there is
* only one used for loading the main module and everything in its dependency graph, though separate instances of this
* class might be instantiated as part of bootstrap for other purposes.
* @param {boolean} useCustomLoadersIfPresent If the user has provided loaders via the --loader flag, use them.
* @param {boolean} forceCustomizedLoaderInMain Ignore whether custom loader(s) have been provided
* via CLI and instantiate a CustomizedModuleLoader instance regardless.
* @returns {DefaultModuleLoader | CustomizedModuleLoader}
*/
function createModuleLoader(useCustomLoadersIfPresent = true) {
if (useCustomLoadersIfPresent &&
// Don't spawn a new worker if we're already in a worker thread created by instantiating CustomizedModuleLoader;
// doing so would cause an infinite loop.
!require('internal/modules/esm/utils').isLoaderWorker()) {
const userLoaderPaths = getOptionValue('--experimental-loader');
if (userLoaderPaths.length > 0) {
function createModuleLoader(customizationSetup) {
// Don't spawn a new worker if we're already in a worker thread (doing so would cause an infinite loop).
if (!require('internal/modules/esm/utils').isLoaderWorker()) {
if (
customizationSetup ||
getOptionValue('--experimental-loader').length > 0
) {
if (!emittedExperimentalWarning) {
emitExperimentalWarning('Custom ESM Loaders');
emittedExperimentalWarning = true;
}
return new CustomizedModuleLoader();
debug('instantiating CustomizedModuleLoader');
return new CustomizedModuleLoader(customizationSetup);
}
}

return new DefaultModuleLoader();
}

/**
* Get the HooksProxy instance. If it is not defined, then create a new one.
* @returns {HooksProxy}
*/
function getHooksProxy() {
if (!hooksProxy) {
const { HooksProxy } = require('internal/modules/esm/hooks');
hooksProxy = new HooksProxy();
}

return hooksProxy;
}

/**
* Register a single loader programmatically.
* @param {string} specifier
Expand All @@ -405,19 +418,19 @@ function getHooksProxy() {
* ```
*/
function register(specifier, parentURL = 'data:') {
// TODO: Remove this limitation in a follow-up before `register` is released publicly
if (getOptionValue('--experimental-loader').length < 1) {
throw new ERR_ESM_LOADER_REGISTRATION_UNAVAILABLE();
}
let moduleLoader = require('internal/process/esm_loader').esmLoader;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

esmLoader getter directly calls createModuleLoader if not exist, which makes it really hard to keep track of the execution flow. Would you mind adding some comments about the assumption of the state of esmLoader attribute in here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Slack we were discussing creating a function getModuleLoader which would return the current active module loader instance, whether it was an instance of the customized or the default one.

Also in general this PR isn’t ready for review yet, I think it’s just where @JakobJingleheimer finished at the end of the night. It doesn’t work yet, and we’re not sure which of the two approaches described in the top post will work or be more readable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a draft. Not sure why GitHub tagged people for review.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the bot who does the tagging, not GitHub. The webhook is executed when you open a PR, regardless of its status. You probably already know that, but you don't have to open a PR, you can also push commits to your forks and open a PR later if you don't want comments for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, it doesn't have to be configured that way (at work, we have the same kind of thing, and it only runs when the PR is not a draft—eg opened as non-draft or switches from draft to non-draft).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn’t realize there was a conversation was going on Slack. I mostly left these comments because I’m trying to familiarize myself with the loader implementation and I believe asking/commenting is the best way of learning.

Sorry for any disturbance caused by my comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just meant this isn't ready for cleanup notes yet. We need to figure out which approach to take and get it working. If you have suggestions for that (see initial post) please feel free 😀


const moduleLoader = require('internal/process/esm_loader').esmLoader;
if (!(moduleLoader instanceof CustomizedModuleLoader)) {
debug('register called on DefaultModuleLoader; switching to CustomizedModuleLoader');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

createModuleLoader makes the final decision of whether default module loader is created or customized module loader is created. But this comment and this function is called when we know CustomizedModuleLoader is going to be created. I think we should decouple the condition of conditionally creating customized or default module loader creation to their own functions and directly create them. This would potentially improve the readability and maintainability of loaders, since if in the future the behavior of createModuleLoader changes, we also need to update this line of comment.

moduleLoader = createModuleLoader(moduleLoader);
require('internal/process/esm_loader').esmLoader = moduleLoader;
}

moduleLoader.register(`${specifier}`, parentURL);
}

module.exports = {
DefaultModuleLoader,
createModuleLoader,
getHooksProxy,
register,
};
Loading