feat: Import maps #2360

bartlomieju · 2019-05-15T09:17:27Z

This PR aims to add support for import maps as described in #1921

For now code is rather poor quality, I intend to correct that in the next days.

It's still very early stage, things to be done:

~~reading import map from JSON (waiting Add serde_derive crate #2359)~~
scopes

cli/import_map.rs

bartlomieju · 2019-05-16T21:31:36Z

@ry please take a look. Got parsing working (there is still some cases with std:* to handle) and most of resolving, now I'd like to start integrating import maps, a natural place for me to use them would be:

impl Loader for Worker {
  type Error = DenoError;

  fn resolve(specifier: &str, referrer: &str) -> Result<String, Self::Error> {
    if import_map {
      let result = import_map.resolve(specifier, referrer);
      ...
    }
    resolve_module_spec(specifier, referrer).map_err(DenoError::from)
  }

  ...
}

But it's not straight forward as Worker implements Loader that lives in core/modules.rs. Any tips?

bartlomieju · 2019-05-19T14:07:48Z

Okay this is more-or-less ready for some review.

cli/import_map.rs is basically one-to-one implementation of reference implementation. There are two differences:

no builtin modules - we don't support them, so one can't remap specifier to builtin module

{
 // this will not work
  "imports": {
     "foo": ["std:blank"]
  }
}

not all fetch schemes are supported - Deno supports only reading from http:, https: and file:

{
 // this will not work
  "imports": {
     "foo": ["data:<asdfasdf>"],
     "bar": ["blob:<asdfasdf>"],
  }
}

Now it's time to integrate this remapping to module loading flow. @ry I will definitely need some help with that

EDIT: Regarding use of serde_derive; I'm pretty happy with current manual parsing of JSON - it's pretty straightforward and readable, after all I think it's not worth to use it in this case.

bartlomieju · 2019-05-19T20:05:04Z

I tried again to integrate import map resolving inside core/modules.rs - this can basically be done changing signature of Loader::resolve, here's "almost" working diff (at the end loader is taken and cannot be used in resolve_cb):

--- a/core/modules.rs
+++ b/core/modules.rs
@@ -41,7 +41,12 @@ pub trait Loader {
   /// When implementing an spec-complaint VM, this should be exactly the
   /// algorithm described here:
   /// https://html.spec.whatwg.org/multipage/webappapis.html#resolve-a-module-specifier
-  fn resolve(specifier: &str, referrer: &str) -> Result<String, Self::Error>;
+  fn resolve(
+    &self,
+    specifier: &str,
+    referrer: &str,
+    is_root: bool,
+  ) -> Result<String, Self::Error>;

   /// Given an absolute url, load its source code.
   fn load(&mut self, url: &str) -> Box<SourceCodeInfoFuture<Self::Error>>;
@@ -105,7 +110,8 @@ impl<L: Loader> RecursiveLoad<L> {
     referrer: &str,
     parent_id: Option<deno_mod>,
   ) -> Result<String, L::Error> {
-    let url = L::resolve(specifier, referrer)?;
+    let loader = self.loader.as_mut().unwrap();
+    let url = L::resolve(&loader, specifier, referrer, true)?;

     let is_root = if let Some(parent_id) = parent_id {
       let loader = self.loader.as_mut().unwrap();
@@ -261,13 +267,14 @@ impl<L: Loader> Future for RecursiveLoad<L> {
     }

     let root_id = self.root_id.unwrap();
+    let imm_loader = &self.loader.unwrap();
     let mut loader = self.take_loader();
     let (isolate, modules) = loader.isolate_and_modules();
     let result = {
       let mut resolve_cb =
         |specifier: &str, referrer_id: deno_mod| -> deno_mod {
           let referrer = modules.get_name(referrer_id).unwrap();
-          match L::resolve(specifier, &referrer) {
+          match L::resolve(imm_loader, specifier, &referrer, false) { // this bit is not working 
             Ok(url) => match modules.get_id(&url) {
               Some(id) => id,
               None => 0,
@@ -634,7 +641,12 @@ mod tests {
   impl Loader for MockLoader {
     type Error = MockError;

-    fn resolve(specifier: &str, referrer: &str) -> Result<String, Self::Error> {
+    fn resolve(
+      &self,
+      specifier: &str,
+      referrer: &str,
+      _is_root: bool,
+    ) -> Result<String, Self::Error> {
       eprintln!(">> RESOLVING, S: {}, R: {}", specifier, referrer);
       let output_specifier =
         if specifier.starts_with("./") && referrer.starts_with("./") {

ry

It's a good start - nice work.
I'm most concerned about Value types propagating beyond from_json and the use of IndexMap.

cli/import_map.rs

cli/state.rs

bartlomieju · 2019-05-20T16:31:47Z

@ry I've updated the PR according to your feedback, please take a look at not-resolved comments to settle on the rest of them. Also if you could suggest something with respect to #2360 (comment) I'd be very happy

ry · 2019-05-20T18:04:08Z

this can basically be done changing signature of Loader::resolve

I'm ok with adding the &self parameter, but why do you need is_root ?

bartlomieju · 2019-05-20T18:29:44Z

this can basically be done changing signature of Loader::resolve

I'm ok with adding the &self parameter, but why do you need is_root ?

Yes, is_root is needed - import map shouldn't be applied to main module or web worker's main module, so a check if needed in resolve.

I'm having trouble making borrow checker work in RecursiveLoad::poll() - Loader::resolve is called inside callback to isolate.mod_instantiate(root_id, &mut resolve_cb), could you check patch above?

core/modules.rs

This patch makes it so that RecursiveLoad doesn't own the Isolate, so Worker::execute_mod_async does not consume itself. Previously Worker implemented Loader, but now ThreadSafeState does. This is necessary preparation work for dynamic import (denoland#1789) and import maps (denoland#1921)

bartlomieju · 2019-06-07T21:38:31Z

Working example:

// importmap.json
{
  "imports": {
    "moment": "./moment.ts",
    "lodash": ["./lodash.ts"]
  }
}

// moment.ts
console.log("I'm remapped moment");

// lodash.ts
console.log("I'm remapped lodash");

// script.ts
import "moment";
import "lodash";
console.log("Import maps in action!");

$ deno --importmap=importmap.json script.ts
I'm remapped moment
I'm remapped lodash
Import maps in action!

ry · 2019-06-07T21:41:55Z

A better example would be to map "http" to "https://deno.land/std/http/" , so that it looks like

import { serve } from "http/server.ts";

async function main() {
  const body = new TextEncoder().encode("Hello World\n");
  for await (const req of serve(":8000")) {
    req.respond({ body });
  }
}

main();

Would be a nice one for the manual.

ry · 2019-06-08T16:00:17Z

cli/flags.rs

+      Arg::with_name("importmap")
+        .long("importmap")
+        .value_name("FILE")
+        .help("Load import map file")


Can you link to the spec, gist, or manual so people can learn more about the file format?

--importmap <FILE> Load import map file Specification: https://wicg.github.io/import-maps/ Examples: https://github.com/WICG/import-maps#the-import-map

Caveat - visible only with deno --help

build_extra/rust/BUILD.gn

piscisaureus · 2019-06-08T16:30:18Z

cli/import_map.rs

+    }
+  }
+
+  pub fn from_json(


I wonder if it would be easier to use serde_derive to automatically generate a parser for this.
Might be a lot less code and reduce risk of bugs.

I went that way, but after several passes I decided that it's actually easier and less code to do it manually (coercing different types to array turned out to be a lot of code in serde_derive). Test suite is pretty big, so this shouldn't be a problem

piscisaureus · 2019-06-08T16:32:38Z

cli/ops.rs

@@ -498,10 +499,30 @@ fn op_fetch_module_meta_data(
  let use_cache = !state.flags.reload;
  let no_fetch = state.flags.no_fetch;

+  // TODO(bartlomieju): I feel this is wrong - specifier is only resolved if there's an


What does the spec say about modules that don't appear in the import map?
Is there some default behavior to fall back to?

Yes, it's fall-through - in our case ImportMap::resolve returns None and specifier is resolved as if there was no import map

But my comment concerns whole op_fetch_module_meta_data - I believe that there should be resolving of specifier even if there is no import map - to see that, please tak a look at tests/error_004_missing_module.ts. It not returning proper error. More for reference in this comment

website/manual.md

bartlomieju · 2019-06-09T11:39:33Z

@ry @piscisaureus feedback addressed

ry

LGTM - awesome work!

I think we’ll still need to iterate on this, but let’s do it in master and not have you keep rebasing.

bartlomieju · 2019-06-09T13:08:44Z

🎉 Thanks! Great to have it landed!

vadimalekseev · 2020-06-24T17:27:18Z

Great work, but I have a question:

Why import_map.json with underscore? People usually use camelCase or kebab-case on JavaScript/TypeScript projects

bartlomieju force-pushed the feat-import_maps branch from f10b9bf to cbf35ad Compare May 16, 2019 15:37

bartlomieju commented May 16, 2019

View reviewed changes

cli/import_map.rs Outdated Show resolved Hide resolved

bartlomieju force-pushed the feat-import_maps branch from abb0eb6 to cdc15a1 Compare May 16, 2019 21:15

bartlomieju marked this pull request as ready for review May 16, 2019 21:32

bartlomieju force-pushed the feat-import_maps branch 2 times, most recently from 5c8ba86 to ffc298d Compare May 17, 2019 18:59

ry reviewed May 20, 2019

View reviewed changes

bartlomieju commented May 21, 2019

View reviewed changes

core/modules.rs Outdated Show resolved Hide resolved

bartlomieju force-pushed the feat-import_maps branch from 238812f to 051ceaf Compare June 5, 2019 09:21

bartlomieju added 14 commits June 5, 2019 11:21

prototype - implement bare specifier

c34ef21

simple resolve in import maps

0a24861

port tests, and package match

f9c2cc0

ImportMap.resolve works for empty map

09767fd

remove ParsedSpecifier

75860c1

add TODOs

3d70b88

fmt

83bb648

manual JSON parsing

73ca1ba

more tests

895d3a2

remove junk files

0ad1378

lint rust

23567af

use longest match algo

5a4f582

preserve reading order

dd8b7ee

use IndexMap for import map

8e5d6e5

refactor flags

145cb26

bartlomieju force-pushed the feat-import_maps branch from fe80eb2 to 145cb26 Compare June 7, 2019 14:01

ry mentioned this pull request Jun 7, 2019

1.0 #2473

Closed

43 tasks

bartlomieju added 4 commits June 8, 2019 13:15

impl ImportMapError for DenoError

da80393

more complex integration test

2f79dc4

fmt

42df29c

update website with import map example

f19d9c9

bartlomieju force-pushed the feat-import_maps branch from 3defdf9 to f19d9c9 Compare June 8, 2019 11:40

ry reviewed Jun 8, 2019

View reviewed changes

piscisaureus reviewed Jun 8, 2019

View reviewed changes

build_extra/rust/BUILD.gn Show resolved Hide resolved

piscisaureus reviewed Jun 8, 2019

View reviewed changes

ry reviewed Jun 8, 2019

View reviewed changes

website/manual.md Outdated Show resolved Hide resolved

bartlomieju added 5 commits June 9, 2019 12:57

filenames with underscores

ee75aca

link to spec and examples of import map in CLI

5e888c6

add feature to serde_json

cedef8b

Merge branch 'master' into feat-import_maps

5936767

Merge branch 'master' into feat-import_maps

bcbc92d

bartlomieju mentioned this pull request Jun 9, 2019

Rewrite deno_dir.rs #2057

Closed

ry approved these changes Jun 9, 2019

View reviewed changes

ry merged commit a115340 into denoland:master Jun 9, 2019

bartlomieju deleted the feat-import_maps branch June 9, 2019 13:08

bartlomieju mentioned this pull request Jun 9, 2019

Implement import-maps #1921

Closed

Silic0nS0ldier mentioned this pull request Jul 14, 2019

JS: Import maps support Silic0nS0ldier/tspm-concept#1

Open

imaginamundo mentioned this pull request Nov 24, 2019

Permissions per modules with importmap like configuration #3397

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Import maps #2360

feat: Import maps #2360

bartlomieju commented May 15, 2019 •

edited

Loading

bartlomieju commented May 16, 2019 •

edited

Loading

bartlomieju commented May 19, 2019 •

edited

Loading

bartlomieju commented May 19, 2019

ry left a comment

bartlomieju commented May 20, 2019

ry commented May 20, 2019

bartlomieju commented May 20, 2019

bartlomieju commented Jun 7, 2019

ry commented Jun 7, 2019 •

edited

Loading

ry Jun 8, 2019 •

edited

Loading

bartlomieju Jun 9, 2019

piscisaureus Jun 8, 2019

bartlomieju Jun 9, 2019 •

edited

Loading

piscisaureus Jun 8, 2019

bartlomieju Jun 9, 2019

bartlomieju Jun 9, 2019

bartlomieju commented Jun 9, 2019

ry left a comment

bartlomieju commented Jun 9, 2019

vadimalekseev commented Jun 24, 2020

feat: Import maps #2360

feat: Import maps #2360

Conversation

bartlomieju commented May 15, 2019 • edited Loading

bartlomieju commented May 16, 2019 • edited Loading

bartlomieju commented May 19, 2019 • edited Loading

bartlomieju commented May 19, 2019

ry left a comment

Choose a reason for hiding this comment

bartlomieju commented May 20, 2019

ry commented May 20, 2019

bartlomieju commented May 20, 2019

bartlomieju commented Jun 7, 2019

ry commented Jun 7, 2019 • edited Loading

ry Jun 8, 2019 • edited Loading

Choose a reason for hiding this comment

bartlomieju Jun 9, 2019

Choose a reason for hiding this comment

piscisaureus Jun 8, 2019

Choose a reason for hiding this comment

bartlomieju Jun 9, 2019 • edited Loading

Choose a reason for hiding this comment

piscisaureus Jun 8, 2019

Choose a reason for hiding this comment

bartlomieju Jun 9, 2019

Choose a reason for hiding this comment

bartlomieju Jun 9, 2019

Choose a reason for hiding this comment

bartlomieju commented Jun 9, 2019

ry left a comment

Choose a reason for hiding this comment

bartlomieju commented Jun 9, 2019

vadimalekseev commented Jun 24, 2020

bartlomieju commented May 15, 2019 •

edited

Loading

bartlomieju commented May 16, 2019 •

edited

Loading

bartlomieju commented May 19, 2019 •

edited

Loading

ry commented Jun 7, 2019 •

edited

Loading

ry Jun 8, 2019 •

edited

Loading

bartlomieju Jun 9, 2019 •

edited

Loading