Proposed overhaul to be more URL-based

[domenic]

Introduction

The Chrome team is keenly interested in being able to use package name maps both as a way of bringing the bare-import-specifier experience to the web, and as a way of enabling web platform features to be shipped as modules (the layered APIs project). In particular we want to enable the LAPI-related user stories in drufball/layered-apis#34.

The current proposal was created specifically to solve the bare import specifier problem, and is pretty good at that, ongoing tweaks aside. But it only has some tentative gestures in the direction of web platform-supplied modules. The proposed syntaxes are half-baked and feel tacked on to the existing proposal, instead of integrating well with it.

My best attempt to use the current package name maps proposal to solve the LAPI use cases is drufball/layered-apis#33. Its biggest drawback is the introduction of the secondary layeredapi: scheme in addition to the std/x (or @std/x) syntax for importing LAPIs. But we are forced into this awkward situation by the current proposal's separation of mapping import specifiers (the left-hand side) to URLs (the right-hand side).

The below is an alternative proposal that was developed from the ground-up to support both use cases in a unified way. It incoporates ideas from several other open issues and PRs along the way. Note that this is written as an evolution of the current proposal, for discussion and to gather thoughts. I'll probably also write a pull request that replaces the existing README with one implementing this proposal, i.e. as if we'd thought of this proposal in the first place. That'll be easier to read top-to-bottom. But I want to have this discussion first with existing collaborators, for which the below framing is probably better.

Proposal details

URL-based mapping, and the import: scheme

As alluded to in #23, it'd be ideal to have a URL scheme that says "use the package name map to resolve the contents". Let's call that scheme import:.

In the current proposal, the bare import specifiers are thought of as "primary", and import: as an add-on feature. That is, we have two namespaces: import specifiers, and URLs, and the purpose of the package name map is to map between them.

This proposal flips things around. Modules are always imported via URL. A URL is the module's primary identifier. There is just one piece of sugar: in JS import statements and import() expressions, the import: part will get auto-prepended for you, when you use a bare import specifier.

With this in hand, we reframe package name maps to be about URL-to-URL mapping. They are no longer about mapping from the import specifier namespace into the URL namespace. They operate entirely within the URL namespace. And most users will be using them to control the import: URL namespace. But you can also use them to control other parts of the URL namespace, which is useful for LAPI user story (C).

Recursive mapping

Now that we have URL-to-URL mapping, one naturally has to wonder: what happens when you map an import: URL to another import: URL? It recurses, of course!

The key question though is error-handling behavior. If you map an import: URL to some other import: URL which is known not to exist, what happens? In this proposal, the mapping gets dropped, perhaps with a warning in your dev console. This works out really well for the LAPI fallback user story (B), as we'll see.

The left- and right-hand sides of the mapping

We've talked about the above as a URL-to-URL mapping. But it's a bit more complex than that, I think.

The current proposal's setup is about mapping a class of module specifiers to a class of URLs, to support submodules. That is, "lodash": { "path": "/node_modules/lodash-es", "main": "lodash.js" } is designed to map both "lodash" -> "/node_modules/lodash-es/lodash.js" and "lodash/*" -> "node_modules/lodash-es/*".

Even if we change the left hand side to a URL (e.g. "import:lodash") instead of a module specifier (e.g. "lodash"), we want to keep this property.

Furthermore, we want to enable the fallback cases discussed in drufball/layered-apis#34 user story (B), or drufball/layered-apis#5. And personally, I want to do so in a way that isn't tied to LAPIs, and works for all modules; that seems way better if we can.

The solution is to extend the right-hand-side of the mapping to allow multiple forms:

• { path, main } tuples, as today
• strings, which behave as in Provide a sensible default path for package main sugar case #52 (i.e. they expand to { path, main } tuples derived from splitting on last path segment of the string)
• arrays of the above, which result in trying each URL in sequence and falling back on network error or non-ok fetching status.

Similarly, the left-hand side keeps its meaning today: it's not only a URL, but also a URL prefix for any submodules.

Examples

Bare import specifiers

Consider the existing examples from this repository. In this new URL-based world, they would be

{
  "mappings": {
    "import:moment": { "path": "/node_modules/moment/src", "main": "moment.js" },
    "import:lodash": { "path": "/node_modules/lodash-es", "main": "lodash.js" }
  }
}

Using the #52 behavior, we can just write this as

{
  "mappings": {
    "import:moment": "/node_modules/moment/src/moment.js",
    "import:lodash": "/node_modules/lodash-es/lodash.js
  }
}

We'll prefer this abbreviated form from now on.

Bare import specifiers with fallbacks

Let's say we wanted to use moment from a CDN, but if that CDN was down, fall back to our local copy. Then we could do this:

{
  "mappings": {
    "import:moment": [
      "https://unpkg.com/moment@2.22.2/src/moment.js",
      "/node_modules/moment/src/moment.js"
    ]
  }
}

LAPI fallbacks, user story (B)

Refresh yourself on drufball/layered-apis#34, then consider this map:

{
  "mappings": {
    "import:@std/async-local-storage": [
      "import:@std/async-local-storage/index",
      "/node_modules/als-polyfill/index.mjs"
    ]
  }
}

This assumes that LAPIs modules are registered (by the browser) at import:@std/lapi-name/*, with an index module in particular existing for each LAPI.

In browser class (1): import "@std/async-local-storage" maps to the URL import:@std/async-local-storage/index which the browser has pre-registered a module for. It works!

In browser class (2): import "@std/async-local-storage" maps to the URL "/node_modules/als-polyfill/index.mjs", after trying import:@std/async-local-storage and getting a failure. It works!

LAPI fallbacks, user story (C)

Refresh yourself on drufball/layered-apis#34, then consider this map:

{
  "mappings": {
    "/node_modules/als-polyfill": "import:@std/async-local-storage/index"
  }
}

In browser class (1): import "/node_modules/als-polyfill/index" maps to the URL import:@std/async-local-storage/index", which the browser has pre-registered a module for. It works!

In browser class (2): this mapping gets dropped from the package name map, per the "recursive mapping" rules above. So such browsers just use the original import statements, pulling in the polyfill. It works!

In browser class (3): the browser doesn't know about package name maps at all, so again the original import statements work, as desired.

Discussion

Overall this proposal accomplishes my goals. It allows package name maps to solve the LAPI use cases, while being more unified; they didn't grow any special capabilities or keys specific to LAPIs. It also solves #23, not in a tacked-on way, but in a way that gets integrated deeply into the mapping.

I see two drawbacks with this proposal:

• Polyfill packages cannot easily have file extensions, especially for submodules. The way in which we do the class-of-URLs to class-of-URLs mapping means that if we want, e.g. import:@std/virtual-scroller/virtual-content to map to /node_modules/vs-polyfill/virtual-content.mjs, we'd need a second mapping, at least.
  • Potential solution: lean into it, and get rid of the class-of-URLs-to-class of URLs mapping entirely? I.e. make everything 1:1, so you'd need to enumerate the submodules of each package, both for LAPIs and non-LAPIs.
  • Potential solution: introduce wildcard substitution? @nyaxt, our Chrome implementer, really doesn't like this path. And it muddles the meaning of these things to be more "URL templates" than "URLs", hindering our ability to reuse infrastructure like the platform's URL parser. So, meh.
  • Potential solution: pick a file extension for the web, and use that for LAPIs? Seems like a non-starter.
  • Potential solution: add an auto-extension-appending feature to the mapping? E.g. { path, main, extension }?
• The amount of ceremony, and concepts to understand, to accomplish LAPI fallback user stories (B) and (C) is high compared to a bespoke LAPI-specific solution. For example having to understand the existence of an index module for each LAPI, or having to use the form "import:@std/x": ["import:@std/x/index", fallback] to express "import:@std/x should fall back to fallback".
  • Potential solution: a dedicated fallbacks top-level section, instead of using array right-hand-sides to the mappings? Still not LAPI-specific, but it is simpler to use.

As an example, if we used the dedicated fallbacks key and the new extension key, a package name map for user story (B) might look more like this:

{
  "mappings": {
    "import:moment": {
      "path": "https://unpkg.com/moment@2.22.2/src",
      "main": "moment",
      "extension": ".js"
    }
  },
  "fallbacks": {
    "import:moment": [{
      "path": "/node_modules/moment",
      "main": "moment",
      "extension": ".js"
    }],
    "import:@std/virtual-scroller": [{
      "path": "/node_modules/virtual-scroller-polyfill",
      "main": "index",
      "extension": ".mjs"
    }]
  }
}

Thoughts welcome, either on these points or more generally.

[robpalme]

Scope

I don't mind expanding the scope of PNMs to include fallbacks. It helps to have all resolution control in one place. It's good to have things that improve runtime efficiency by making more information available early, and help tooling understand whats going on by centralizing the information.

Observability

In a module loaded via a fallback, I assume the importee module can discover its resolved/physical identity via import.meta.url. Will there be a builtin way for the importer module to know what they loaded? Given we're introducing non-determinism in loaded modules when using an array of mappings, we'll need debugging/telemetry to remotely report on what modules your app truly loaded for a particular run.

Bare specifier definition

the import: part will get auto-appended for you, when you use a bare import specifier.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

Extensions

On the file extension drawback, I have a mild personal preference to lean in and drop _class-of-URLs_mappings. It promotes deep imports into arbitrary files within external packages, that may or may not be considered implementation details. I think it best to surface inter-package dependencies in the PNM to aid static analysis. My expectation is that PNMs will primarily be tool-generated (e.g. as a pre-deployment step), so I'm not worried about extra lines.

Misc

Two typos:

• auto-appended -> auto-prepended
• right-right -> right-

[guybedford]

Very exciting to see these directions being embraced.

arrays of the above, which result in trying each URL in sequence and falling back on network error or non-ok fetching status.

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Now for my concerns :)

The two arguments for URL mapping seem to be (1) standard module are URLs and (2) For use case (C) we want to support URL mapping.

I don't think this is an adequate justification for moving to the more complex URL -> URL mapping.

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

1. Standard modules are URLs

You're introducing an indirection anyway here as well with an index for the standard modules to distinguish anyway.

If users just use a package map to map into the standard module URL instead of writing a URL (say import @std/async-local-storage), then the package name map can solve use case B fine.

2. URL mapping for browsers without package map support (use case C)

This feels to me like trying to do too much in one go. The web traditionally lets the ecosystem handle compatibility problems, and instead focuses on the future workflows. If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

[domenic]

Thanks both for the comments!

@robpalme

Observability

An interesting question. I think the platform has some things that help with this, such as the resource timing API. Ideally I'd like to reuse those.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

No. In this scheme, you import URLs, always. And package name map left-hand sides are URLs. So "/node_modules/als-polyfill" is a URL, just like "import:lodash" is. Bare import specifiers are no longer really a thing, except that there's that auto-prepending sugar which gives you the same effect.

Extensions

Interesting viewpoint. On reflection, I think I agree. That makes this all a lot simpler, which helps address some of @guybedford's concerns. We can just have string left hand sides, and pure URL right-hand sides.

Two typos:

Fixed, thanks!

@guybedford

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

URL mapping for browsers without package map support (use case C) ... If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

Unfortunately this is a hard constraint we must meet. So I think it's best we work on a way to meet it while avoiding the complexity you're worried about. @robpalme's simplification might help in that regard... I'll start writing it up.

[guybedford]

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

I really like this form, and I think the way it expands is quite predictable and intutitive.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Do you allow infinite loops? Eg:

{
  packages: {
    "/x": {
      "path": "/x/x"
    }
  }
}

how do you know when to stop the recursion in this scenario?

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

Why should we be able to map standard modules, when indirection is fine?

Unfortunately this is a hard constraint we must meet.

Who is "we"? Don't change the web to solve things it doesn't need to solve. Why not go the tried and tested polyfill route rather?

[domenic]

how do you know when to stop the recursion in this scenario?

The same way you usually do? Keeping track of where you've been?

Why should we be able to map standard modules, when indirection is fine?

I don't understand what "indirection" is in this instance, or why it is fine.

Who is "we"? Don't change the web to solve things it doesn't need to solve.

"We" is the proposal authors and implementers. We are not interested in working on package name maps if they don't also solve the important use cases for built-in modules. I understand you might not think this is important to solve, but from talking with our customers and partners we've found that without a solution for these cases, there's not enough interest in this feature to be worth our devleopment effort.

Why not go the tried and tested polyfill route rather?

I don't understand what this means, as this whole proposal is about enabling the polyfill route to even be possible.

[domenic]

Writing up the proposal in more detail I've run into one fly in the ointment. It's weird that package name maps allow remapping of all URLs, but only the remapping of import: URLs works outside of import statements and import() expression contexts. Trying to explain this is quite weird, because I talk about the benefits of import: URLs and how they are usable everywhere, and then I talk about how you configure them using package name maps, but note that you can also configure other URLs using package name maps, and those aren't usable everywhere...

Not sure what to do with this. When the namespaces were separate, import: meant "use module specifier resolution", and that story was pretty easy. Now that there's just the URL namespace, but applied unevenly, it's more troubling. Might need to back up a few steps... Sigh.

Edit: current potential solution I'm thinking of is to say that (a) the package name map only changes import: URL resolution, even in import and import(); but (b) import: always gets auto-prepended to import and import() statements. So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs". We'll see what that looks like. In this version you wouldn't use import: prefixes in the package name map.

[domenic]

Started on a writeup, but ran out of time for today. Still, should give the basic idea, for those very actively engaged: https://github.com/domenic/package-name-maps/blob/url-based/README.md

[littledan]

I'm a big fan of just about everything in this iteration of the proposal, and agree with @domenic that polyfilling built-in modules is very important. This proposal seems to work well for both the case of an entirely missing built-in module, and the ability to wrap a built-in module in another module (e.g., in case a function is added, and you want to polyfill it in old browsers; see tc39/proposal-built-in-modules#2).

Potential solution: lean into it, and get rid of the class-of-URLs-to-class of URLs mapping entirely? I.e. make everything 1:1, so you'd need to enumerate the submodules of each package, both for LAPIs and non-LAPIs.

I'm wondering whether there's something halfway to that point: Could we say that, if you use a string, you're indicating just a single mapping, and then the object can be used to permit submodules as well, if we want that feature?

[daKmoR]

To be honest, when this first started I was a little hesitant as the "current" implementation with path_prefix, packages, scopes, main and everything looked really complicated.

However, this new flatter approach is sooo much easier to understand, generate and I hope applying it would also be not too complex.

So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs"

So especially in this usecase the "import:" would do nothing right? as it's followed by a /. If it where
import:foo/bar then it would actually look up foo in the map. Just wanna be sure about my understanding.

@guybedford I'm curious if you would consider this "new" approch in your shim maybe in a branch? I would like to play around with a generator of this map... most likely on top of yarns "beta" plug and play implementation

[domenic]

Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look.

[MylesBorins]

This looks great! Any expectation for a name change?

…

On Tue, Oct 2, 2018, 8:48 PM Domenic Denicola ***@***.***> wrote: Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look <https://github.com/domenic/package-name-maps/blob/336ccbcc2000235733ce3fd93b98934b47882c8e/README.md> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV9H7CO-T8u4LF8oiFcnzZBaZmbleks5uhAlbgaJpZM4WEvVh> .

[domenic]

Yeah, tentatively "module import maps" instead of "package name maps" since it's now focused more on controlling imports in general and deemphasizing the package concept.

[ljharb]

@domenic This looks great! Not sure where to provide feedback:

• re https://github.com/domenic/package-name-maps/blob/url-based/README.md#denying-access-to-a-built-in-module, since import 'nonexistent/path'; wouldn't throw with @std/blank, it seems like @std/thrower is a better default implementation - or perhaps, @std/parseError?
• re https://github.com/domenic/package-name-maps/blob/url-based/README.md#installation, what happens if i try to dynamically add a map to the page with JS later? I'd hope/expect that would throw.
• re extension lookup, it's quite apparent that browsers can't do dynamic lookups, but in the same vein as "ends with a /", is there something we could do like a "foo/bar" scope under which all paths get a .js added? Certainly this can be handled by build tooling, but it might be nice to allow for (static, predetermined, hardcoded) extension/index resolution.
• re import:, none of the examples seem to make it explicit - can you import 'import:blah' the same as import 'blah'? Is the duality desirable, or would there be a way to prevent import: from being present in JS contexts?

[michael-ciniawsky]

What's the reasoning behind import:./x, import:../y, import:/z? Can't those not just use a 'bare specifier' aswell and left as is otherwise?

url('bare/logo')
<link href="bare/style">
import module from 'bare'

{
   modules: {
      bare: {
         '/': '/path/to/bare', // Path (Context)
         index: './lib/main.js',
         style: './lib/css/main.css',
         logo: './lib/assets/logo.svg',
         ...
      }
   }
}

[domenic]

@michael-ciniawsky I don't really understand your example, but the reasoning is to support https://github.com/domenic/package-name-maps/blob/url-based/README.md#for-built-in-modules-in-browsers-without-module-import-maps

[michael-ciniawsky]

What is unclear can you elaborate?

[michael-ciniawsky]

Why would pollyfilling need support for ./, ../, / in particular?

{
  "modules": {
    bare: {
      index: [
        "import:@std/buildin",
        "/node_modules/polyfill/index.js",
        "https://cdn.fallbacks.com/polyfill/index.js"
      ]
    }
  }
}

[ljharb]

So you can default to your own implementation but use module maps to gracefully replace it with the builtin when available, is my understanding.

[robpalme]

This new version is good. The trailing / convention is neat.

• Typo: import:query -> import:jquery
• Defining the right-hand side to be non-bare seems like a good idea to add clarity, i.e. forcing a prefix of ./ ../ / or import: Support for bare rvalues can always be introduced later.
• None of the JS code examples show the use of explicitly prepending import: to a specifier. I assume this is legal and provides identical resolution to not specifying it. I suggest adding an example.
• The existence of @std/blank seems a bit arbitrary. Blacklisting (denying) a specifier is a first-class use-case so deserves first-class syntax in my opinion. An empty string would work, right?
• you'd better be using HTTP/2 push to get that thing to us - maybe say "the client" rather that "us".
• Typo: succesfully
• Typos: othre
• I'd like to see some statement in the document on how to programatically reflect on the results of the fallback-array non-determinism. You've previously stated we might be able to reuse existing techniques.

[dcleao]

I like the uniformity and power of the proposed import: scheme, as it would be usable in all contexts where a url can be specified.

• If you load a module from a CDN that uses bare import specifiers, you'll need to know ahead of time what bare import specifiers that module adds to your app, and include them in your application's module import map. (That is, you need to know what all of your application's transitive dependencies are.) It's important that control of which URLs are use for each package stay in control of the application author, so they can holistically manage versioning and sharing of modules.

  While it is great to be able to have control, I believe it should only be an option. I should not need to know what are the libraries that a library that I depend on uses for its own implementation.

• I don't have a preference, but I wonder what are the gains of the"/" syntax when comparing it with the "main" property alternative. It does get more verbose.

[littledan]

The newest version looks even better. Two small comments:

• Interesting example for an initial self-reference of the name when expressing fallbacks for built-in modules. I'm wondering, do you have an idea for the logic to prevent an infinite loop when resolving this case? (I can think of more than one possibility.)
• It sounds like you've gone back towards remapping arbitrary URLs, not just "bare" specifiers. I'm happy with the design decisions this and other proposals have made in this space, but curious about if this opens up other possibilities, e.g.,
  • (Although I personally prefer @std/) Would it work out to use std: as the prefix, and permit remapping of this pseudo-scheme as well?
  • Should bare specifiers like 'bare.mjs' in module imports actually have the semantics of'./bare.mjs', if the two are equally remappable?

[nyaxt]

The latest draft lgtm.

FYI: @yutakahirano

I think we need to clarify:

• How the extended import URL scheme maps to SecurityOrigin.
• The scope resolution algorithm: Given the map and a context script URL, resolve scope.
  • IIRC, the advantage of the explicit package based mapping was that scope resolution behavior is intuitive.
  • Giving it an another thought, may be it wouldn't be a blocker as long as the algorithm is deterministic.
• "sensitivity to the current script file"
  • Is this the referencing script's base URL == the script's response URL?

Would it make sense to implement this by having import: URLs issue redirect to the resolved URLs?

[RangerMauve]

How would this new import: scheme work with the work being done on changing navigator.registerProtocolHandler to use a blocklist instead of an allow list?

I assume import: would need to be added to the blocklist.

[kinu]

The proposal looks exciting. Let me list some of the comments/questions I had (some might have been already discussed or explained).

• This says that it should be able to used to fetch any resources, which seems to imply that this is actually more about 'fetch resource map' rather than import map, is my interpretation right?
• It's a little unclear what should happen if the import URL is used for navigations or worker / service worker instantiations.
• Related, how the origin concept should be handled with this URL?
• It's already mentioned but it's unclear how importmap should be specified for modules in/with workers, and then for service workers.
• What should happen if a resource loading triggered in a context where a importmap is set up redirects to an import: URL?
• How should the import: URL loading interact with Service Workers?
• How should it work with preloads/prefetches that come with link headers?