--experimental-loader breaks in Node 18.19

[simlu]

Hello ya'll. You broke something and I'm back again =)

I have the following --experimental-loader that worked great pre node v18.19.0.

However as of that version it breaks. Since I have no idea what changed and the (changelog) documentation is somewhat sparse, I would much appreciate if someone could help with what to investigate and how to fix it.

This loader is used in a test framework that is used by hundreds of repositories. So it breaking is rather inconvenient.

Expand

import assert from 'assert';
import crypto from 'crypto';
import fs from 'fs';
import { URL } from 'url';

const lookup = {};
let envVars = {};
const port = {}; // dummy variable, not used

/* serialized and passed into main process */
function createListener() {
  /* communicate process.env to loader process */
  process.env = new Proxy(process.env, {
    set(target, key, value) {
      // eslint-disable-next-line no-param-reassign
      target[key] = value;
      port.postMessage(target);
      return target[key];
    },
    deleteProperty(target, key) {
      if (!(key in target)) {
        return true;
      }
      // eslint-disable-next-line no-param-reassign
      delete target[key];
      port.postMessage(target);
      return true;
    }
  });
}

export function globalPreload({ port: p }) {
  if (process.versions.node.split('.')[0] < 20) {
    /* Skip listener, since process shared before node 20 */
    envVars = process.env;
    return '(() => {})()';
  }
  // eslint-disable-next-line no-param-reassign
  p.onmessage = ({ data }) => {
    envVars = data;
  };
  return `(${createListener})()`;
}

export const resolve = async (specifier, context, defaultResolve) => {
  const result = await defaultResolve(specifier, context, defaultResolve);
  const child = new URL(result.url);

  if (
    child.protocol === 'nodejs:'
    || child.protocol === 'node:'
    || child.pathname.includes('/node_modules/')
    || context.parentURL === undefined
  ) {
    return result;
  }

  const parentPath = new URL(context.parentURL).pathname;
  const childPath = child.pathname;

  [childPath, parentPath].forEach((p) => {
    if (!(p in lookup)) {
      lookup[p] = {
        parents: [],
        reload: false
      };
      const content = fs.readFileSync(p, 'utf8');
      if (content.includes('/* load-hot */')) {
        lookup[p].reload = true;
      } else if (content.includes('process.env.')) {
        lookup[p].reload = [...content.matchAll(/\bprocess\.env\.([a-zA-Z0-9_]+)\b/g)].map((e) => e[1]);
      }
    }
  });
  const isNewParent = !lookup[childPath].parents.includes(parentPath);
  if (isNewParent) {
    lookup[childPath].parents.push(parentPath);
    // mark all parents as reload
    if (lookup[childPath].reload !== false) {
      const stack = [parentPath];
      while (stack.length !== 0) {
        const ancestor = lookup[stack.pop()];
        if (ancestor.reload !== true) {
          if (lookup[childPath].reload === true) {
            ancestor.reload = true;
          } else {
            assert(Array.isArray(lookup[childPath].reload));
            ancestor.reload = [
              ...(Array.isArray(ancestor.reload) ? ancestor.reload : []),
              ...lookup[childPath].reload
            ];
          }
        }
        stack.push(...ancestor.parents);
      }
    }
  }

  if (!('TEST_SEED' in envVars)) {
    return result;
  }

  if (lookup[childPath].reload === false) {
    return result;
  }

  if (Array.isArray(lookup[childPath].reload)) {
    const hash = lookup[childPath].reload.reduce(
      (p, c) => p.update(c).update(envVars[c] || '<undefined>'),
      crypto.createHash('md5')
    ).digest('hex');
    return {
      url: `${child.href}?id=${hash}`
    };
  }

  return {
    url: `${child.href}?id=${envVars.TEST_SEED}`
  };
};

Appreciate all your hard work and really looking forward to loaders being stable ;-)
Cheer, L~

[GeoffreyBooth]

It’s probably one of these: https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#esm-and-customization-hook-changes

These changes landed more than a year ago on the newer release lines, so reverting them isn’t possible without breaking everyone using v20 and later. The API has changed significantly since you last used it; the recommended approach now is to use --import and register().

[simlu]

Thanks for the quick reply!

It’s probably one of these: https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#esm-and-customization-hook-changes

Yeah I had already read through that, but it wasn't very useful wrt solving my use case.

These changes landed more than a year ago on the newer release lines, so reverting them isn’t possible without breaking everyone using v20 and later. The API has changed significantly since you last used it; the recommended approach now is to use --import and register().

Gotcha, I'll check it out. We use Aws Lambda and the lts versions only just updated to that version and started breaking our test. The 20.x must still be on an older minor version, there this still works.

Are there any complete examples anywhere how one would do import hot reloading like the above loader does?

[simlu]

Looks like this thread is a never ending saga lol

I'll try my luck there again... See what people have been cooking up recently. Man I miss cjs...

[GeoffreyBooth]

Looks like this thread is a never ending saga lol

That thread is about avoiding gradual memory leaks when hot reloading (since there’s no way to garbage-collect the replaced module). Simply implementing hot reloading itself is a solved problem: Vite has been doing it for years. See https://dev.to/giltayar/mock-all-you-want-supporting-es-modules-in-the-testdouble-js-mocking-library-3gh1 for instructions on implementing it yourself, and links to lower-level libraries that implement just that part.

I’m going to close this since there’s no clear request or bug presented, but if you find one I’m happy to reopen.

Edit: The above article was written in 2020 and so refers to an older version of the API. A more updated variation can be found at https://github.com/testdouble/testdouble.js/blob/main/docs/7-replacing-dependencies.md#how-module-replacement-works-for-es-modules-using-import, with code that works in current Node.

[simlu]

Thanks for pointing me in the right direction. Ended up being a somewhat simple fix. Much appreciated!