Unified imports proposal

[stefnotch]

Background

For future features, such as re-exports, and inline modules, we want an import system that has a concept of a module! One where import foo::bar::baz does not necessarily translate to foo/bar/baz.wesl. There could have been a re-export after all.

The basic idea of an import is

1. Resolve the import. Examples for what import a::b can mean, depending on the rest of the code
   • It points at a/b.wesl.
   • It points at a function b from a.wesl
   • Due to a re-export, it points at a/utils/b.wesl
2. Bring it into scope.
   e.g. import a::b can bring a module named b into scope. Or a function named b

Caching import resolutions is explicitly an implementation detail. The algorithms described below will work without caching, but caching can speed them up considerably.

Resolving the source of an import is not always obvious. So here is a proposed algorithm.

Sparse Web Projects

Some web projects scatter .wesl files throughout their hierarchy. Usually the .wesl files are reasonably concentrated, but there occasionally are folders do not have a wesl file.
e.g.

wesl.toml
src/
  components/
    HelloWorld.vue
    shaders/
      glitch.wesl
  utils/
    math.wesl
    rgb.wesl
    hsv.wesl

Basic Observation

Even without re-exports, or anything advanced, we need a resolution algorithm for import a::b!
It can either import the module b, from the file a/b.wesl.
Or it can import the function b, from the file a.wesl.

To solve that, we go over the import, step by step. At each step, we decide where to go next to find the correct module.

Step by step resolution

At the end of the resolution, we have a tree of modules. Each module has exactly one fully qualified path.
(Future: Module re-exports will get resolved to the fully qualified path.)

First comes the root step.

• . means "absolute path of the current module"
• (Optional: self means the same thing. self is a reserved word in WGSL, so it'd work.)
• .. is a shorthand for ./..
• (Optional: super)
• some_name will look at the wesl.toml wesl.toml for LSPs #57
  • If it is a library name, then we go to the library and restart the resolution from there.
  • If it is the name of a file, we go to that file.

Now we are at a valid module that we can parse.

Then comes the next step.

• .. means "go one up in the tree of modules". If we escape the root folder of a dependency, we throw an error.
• (Optional: super means the same thing. super is also a reserved word in WGSL.)
• some_name. Now we look at the current module.
  • If it is a function/struct/constant/..., then we are done. We found a concrete item.
  • (Future: If it points at a re-export, we go there.)
  • (Future: If we have an inline module, we go there.)
  • If none of the above were true, we go to the next file. current/module/path/some_name.wesl
    • Every time the next file does not exist, we pretend that an empty file was found. And then we continue to the next file until we find one.
      • (Future: If the need arises, we can add an explicit "skip those folders" statement to WESL. Aka proposal A.)
    • (Future: If we were inside an inline module, then this last rule doesn't apply. We just throw an error.)

Finally, we repeat that step until we are finished with our import. At the end, we either found a valid import, or ran into an error.

[k2d222]

Great, this proposal is roughly what I want too! Some comments:

If none of the above were true, we go to the next file.

what if the file does not exist? do we throw or do we try the next path component?

In my understanding, a project could get away with not writing a single export declaration if the file tree is well structured: all subfolders are adjacent to a file with the same name. This is close to what python does: a module is a folder that contains a file __init__py (but python is more cryptic!). I'm slightly concerned that it would encourage ppl to create empty files just for that purpose. like in your example, I would be tempted to create an empty src/utils.wesl so all files in src can import util::anything.

Proposal A

• it seems to me that export works like the mod statement in rust, except that it can skip subfolders and can rename. Is this correct?
• does an export also act as an import for the current module, i.e. brings some_name into scope?

Proposal B

• does it mean that a file can have only one wildcard export statement?
• could you mix and match proposal A exports and a proposal B export, the latter acting as a catch-all?
• does it mean that the import can only resolve to a file module, and not a declaration inside the file? e.g. import ./src/utils/anything/blah would resolve to /src/utils/anything/blah.wesl, but never to declaration blah in file /src/utils/anything.wesl
• similarly, does it mean that if a submodule declares an inline module, it could never be imported from the outside?.

Additional notes

• re-exports could simply be aliases, since we can import aliases. export "./src/utils" as utils; alias math = utils::math;
• similarly for proposal A, this could be an alternative equivalent syntax: alias some_same = "./src/utils/shader". Essentiallly the quoted path is a regular ident. Then perhaps we could do things like accessing file declarations directly: let x = "./src/utils/shader"::sqrt(2.0);. Actually you can do that in naga_oil!

[stefnotch]

Great, this proposal is roughly what I want too! Some comments:

If none of the above were true, we go to the next file.

what if the file does not exist? do we throw or do we try the next path component?

That is a really good question. Initially I was assuming that it's an immediate error, but after thinking about it for a bit, I don't actually see an issue with "just continue and try the next path component".

Maybe that should be proposal C?

• Everything is a module import
• If we don't find a file, we go to the next component and try that

Proposal A

* it seems to me that `export` works like the `mod` statement in rust, except that it can skip subfolders and can rename. Is this correct?

Yes, pretty much

* does an export also act as an import for the current module, i.e. brings `some_name` into scope?

No, it doesn't bring some_name into scope. If the user wants some_name, they can now import it with the usual mechanisms.

Proposal B

* does it mean that a file can have only one wildcard `export` statement?

Yes, exactly

* could you mix and match proposal A exports and a proposal B export, the latter acting as a catch-all?

I'm thinking that we should only pick one of them. Currently I prefer proposal A over proposal B.

* does it mean that the import can only resolve to a file module, and not a declaration inside the file? e.g. `import ./src/utils/anything/blah` would resolve to `/src/utils/anything/blah.wesl`, but never to declaration `blah` in file `/src/utils/anything.wesl`

Yes. That's a major limitation of proposal B. I actually haven't thought about this limitation, I only thought about the limitation below vvv.

* similarly, does it mean that if a submodule declares an inline module, it could never be imported from the outside?.

Also yes. Also a limitation of proposal B.

Additional notes

* re-exports could simply be aliases, since we can import aliases. `export "./src/utils" as utils; alias math = utils::math;`

Oh yes, that would also work! We could also do export file("./src/utils") as utils;, juuuust in case the WGSL peeps ever want to add strings to WGSL so that we can have print("quoted string").

* similarly for proposal A, this could be an alternative equivalent syntax: `alias some_same = "./src/utils/shader"`. Essentiallly the quoted path is a regular ident. Then perhaps we could do things like accessing file declarations directly: `let x = "./src/utils/shader"::sqrt(2.0);`. Actually you can do that in `naga_oil`!

Similarly to above, we could let the syntax be let x = file("./src/utils/shader")::sqrt(2.0);
We can, of course, bikeshed over whether the function should be called file or mod or import or ...

[mighdoll]

I think there are concerns here we might discuss separately, maybe as separate github issues. But I may be misunderstanding too. Anyway, here's my first take:

• We want some way for packages to publish wgsl elements for use by other external importers from other packages and to customize the visible names and paths that. Allowing packages to choose their API seems like the most important use case here. Is that right?

  • If it's the important case, let's characterize it and hopefully resolve that use case first.
  • Is renaming/repathing at the package boundary best solved by creating an internal module/path renaming mechanism and then adding visibility control atop it? I recall some of the alternatives we discussed awhile back that include renaming/repathing at the same moment as selecting things to part of the package api like @publish(some/name). Best to consider in the context of the use case we're trying to solve.

• The import system ought to be extensible to handle some future variant of namespaces/submodules. That's import should be able to address namespaces / submodules within a file #25 I think.

• Do we also want to enable some kind of remapping of import paths for internal to the package modules? e.g. some way to declare that henceforth all the elements in ./foo/bar/deep/deeper/zap.wgsl can be imported from any module internal to the package with some shorter path? If we want this feature, I see how export "./src/utils/shaders/starry_shader" as starry would get us there.
  • I think that's a separate use case (and probably lower priority).
  • Given that we're trying to enable a distinction between internal and external names and paths, wouldn't we still need two mechanisms? e.g. we might want to use import starry internally for convenience, but publish so that other packages import pkg/star_shader;
  • Also we might also consider something that works per file, sort of like how naga_oil would allow #define_module_path some_name in the shader.wgsl file. I can see how per file repathing/renaming would be a nice convenience for projects that like keeping the shaders in strings for example.
  • I'm uncomfortable with the idea that import starry/foo could refer to a local module in my package if I have the export line above. If I later npm add a package named starry, my code breaks mysteriously. Better to keep separate package names in a separate namespace.

• Also implied here as @k2d222 mentioned on discord and @ncthbrt brought up in Is star in import syntax actually necessary for resolution? #35, we might want replace the currently spec'd wildcard syntax. Worth considering, and also orthogonal probably.

[stefnotch]

@k2d222 @mighdoll @ncthbrt
I updated the proposal, now it really is just

1. We have the segments of the import path
2. Start at root
3. Look at file
4. Go to next file (and skip missing ones)
5. Repeat

[k2d222]

LGTM