[RFC] Support source field in package.json to enable babel on symlinked modules

[devongovett]

Related issues: #13, #948

Summary

When developing several modules locally, it is useful to symlink them into your project with npm link or similar. This is also common with monorepos managed by yarn workspaces and lerna. If those linked modules need to be compiled with babel, it is annoying to have to run a manual build step after each change since Parcel does not run babel on node_modules by default.

This PR adds support for a new source field to package.json. When a module with a source field is symlinked, we enable babel compilation on that module. If the module is not symlinked but installed normally, babel compilation is disabled as usual. This is a development only feature, and should not be relied on for production use.

The source field also operates as an alias so you can change the resolution of files within a module when it is being used as source code. For example, it is common to pre-compile the src folder of a module to lib or something prior to publishing to npm. The main field in package.json, would then point to e.g. lib/index.js. When developing locally, you would want this to point to source code instead of compiled code. The source field lets you do this.

Examples

1. Treat all files as source code, don't change the resolution

   {
     "main": "foo.js",
     "source": true
   }

2. When compiling from source, use bar.js as the entry point

   {
     "main": "foo.js",
     "source": "bar.js"
   }

3. When compiling from source, alias specific files

   {
     "main": "foo.js",
     "source": {
       "./foo.js": "./bar.js",
       "./baz.js": "./yay.js"
     }
   }

4. When compiling from source, alias using glob patterns

   {
     "main": "foo.js",
     "source": {
       "./lib/**": "./src/$1"
     }
   }

The last example allows you to e.g. replace your entire lib directory with src so import 'my-module/lib/test.js would resolve to my-module/src/test.js. You could also use a top-level catch-all pattern like "**": "./src/$1" for packages like lodash that have many files in the root to replace e.g. lodash/cloneDeep with lodash/src/cloneDeep.

Questions

1. Should we require the source field, or is symlinking enough to enable babel? I think it will be pretty common to need a source field anyway for aliasing (i.e. the first example will be uncommon), so maybe not a big deal to require one.
2. What if you actually want to import a precompiled file from a symlinked package? e.g. import "my-module/dist/index.js". Currently, babel would still be applied as there is no way to know which files are source files and which are compiled. Again, I don't think this should be very common but curious to hear opinions.

Feedback

I think this feature will be very useful for local development. Please let me know your feedback!

cc. @tj @CentaurWarchief @Vanuan @gaearon

[gre]

nice writeup & very interesting idea. This would be super useful for dev productivity.

Would you use babel by default? How does parcel know which babel plugins need to be applied for the module? When using lerna, I usually will have all my scripts at the project top level and things like .babelrc is not even present in the module.

[gaearon]

cc @bradfordlemley

[Janpot]

In development, you'd still want a watcher on all those files, to start off a build when a change happens, right?
Would another road be to keep packages "isolated" as they are, each producing a built artefact. And have the workspace, or lerna distribute a watch command over the whole repo. In a similar way it redistributes the test command right now?
Scratch my comment, I'm all for a solution like this

[benjamn]

I like this idea, and I would be happy to make Meteor respect the source field if packages start to use it, in addition to our current default behavior of compiling linked packages if the source code resides within the application (outside of node_modules): explainer, PR.

Clarifying question: who decides what Babel configuration is used to compile the package, if it uses the source property? The package author might want to ensure that a certain set of plugins/presets are used, at minimum, using a .babelrc file. But presumably the application developer wasn't entirely happy with those minimum .babelrc settings, since otherwise the compiled version of the package would have worked well enough. What I'm missing in this proposal is how the application developer might provide additional/different Babel configuration, while also respecting the assumptions of the package author.

Should we require the source field, or is symlinking enough to enable babel?

I'm having trouble imagining why a package author would not want to provide a source field, given that it applies only for linked npm packages, and most linked packages are just clones of the source repository, which probably needs to be compiled in order to be usable.

If there's really no reason for a package author not to provide the field, then it might be better just to compile all linked npm packages, instead of requiring a special field, because there's a cost to getting package authors to adopt a new convention.

I'm also a little skeptical that the source configuration should be the concern of the package author, rather than the application developer, since bundling is typically a concern of the application developer (the person who runs Parcel or Webpack or whatever). What if there's a package that you need to compile that hasn't opted into the source convention? Why depend on cooperation from the package author?

To ask the same question from a different angle: what's the benefit to the package author of opting into the source configuration? If it's a way to make everyone happy, and finally put a stop to bugs about inadequate Babel compilation, that's great, but it could also easily become another configuration surface area that doesn't quite work for everyone, despite the implicit promise of making the package "just work" with more tools.

As a package author, I might prefer to let my more ambitious users figure out compilation for themselves, rather than making promises I can't keep about use cases I can't anticipate.

What if you actually want to import a precompiled file from a symlinked package? e.g. import "my-module/dist/index.js". Currently, babel would still be applied as there is no way to know which files are source files and which are compiled. Again, I don't think this should be very common but curious to hear opinions.

I think this is common enough and important enough to consider, especially because compiled dist files are often very large, and it would great to avoid recompiling them somehow.

With Meteor's approach, since the linked package is viewed as part of the application, you can ignore arbitrary files or directories by putting them in a .meteorignore file.

Meteor also supports creating symbolic links to packages that are installed within node_modules, so you could link directly to the src directory of some package, to enable compilation of just the files in that directory, rather than enabling compilation for the whole package. Roughly speaking, if Meteor's bundler encounters two files with the same fs.realpath, and one of them has been compiled by Meteor, the same compiled code will be used for both modules, so the direction of the symbolic linkage doesn't really matter.

[devongovett]

Thanks for all the feedback. I'll try to address the questions below.

How does parcel know which babel plugins need to be applied for the module?

We respect the .babelrc file in the linked package. We do NOT respect the .babelrc of the parent app when compiling linked node_modules. Since the linked package will presumably be published to npm separately from the app, it should not have any implicit dependencies on the app's code even if it lives in the same monorepo since when used outside that app they won't be there.

In development, you'd still want a watcher on all those files, to start off a build when a change happens, right?

Yep, we'd watch the source files for changes rather than the dist files.

Who decides what Babel configuration is used to compile the package, if it uses the source property?

As described above, the linked package describes its own build configuration, not the application. Applications shouldn't need to know how to build other node_modules.

What if there's a package that you need to compile that hasn't opted into the source convention? Why depend on cooperation from the package author? What's the benefit to the package author of opting into the source configuration?

By default, node_modules are considered precompiled. The source field lets package authors describe how the bundler should reach source files instead of precompiled files. It's necessary for package authors to describe this in their package for it to work correctly, otherwise we'd just be compiling the precompiled files again (which might not exist in the linked package if it hasn't been built).

I'd expect many linked packages in an application to actually be owned by the same author as the app itself. This is definitely the case in a monorepo, and probably common in other cases as well.

If a linked package doesn't have a source field, we'd just depend on a manual build step in that package to build the precompiled files just as we do today, so this at least wouldn't make that any worse.

[bradfordlemley]

We're discussing something similar for the create-react-app build system in facebook/create-react-app#4092. In that proposal, the consumer specifies which packages are source packages and builds those packages according to the consumer's (CRA's) build spec -- it doesn't honor the source package's .babelrc.

This would conflict with this parcel approach because the package could contain a "source" entry point that contains to JSX/etc., but the package itself doesn't have the .babelrc to transpile it.

A couple ideas to make them compatible: (1) default to use the consumer's build if the source package doesn't have .babelrc, or (2) in addition to what is already proposed, support the same "sourcePackages" config that facebook/create-react-app#4092 proposes which would explicitly list which packages should be built according to the consumer's build spec.

Another related question with the current approach: Is there a way for thesource package's build to honor the consumer's desired target environment? E.g., for the consumer to provide babel-preset-env settings? It doesn't seem like a huge issue, but kind of annoying if you're using something as source, but not to be able to build it to the desired target spec. This is one nice feature of delegating the entire build to the consumer as proposed in facebook/create-react-app#4092.

[TimNZ]

Got here from #948

Not a symlink'd module.
Getting 'unexpected token' from React parsing errors on a nested node_module where "jsnext:main" is specified in the nested node_module package and pointing to src.

app
-- node_modules
---- module A ("main": "dist/Jam.js")
        Jam.js:  'require('module B')'
------- node_modules 
----------- module B ("jsnext:main": "src/index.js")
                    index.js: JSX -> 'unexpected token'

[devongovett]

@bradfordlemley

1. I don't think we should use the consumers build environment. That might work well for monorepos where the code is colocated, but for modules which are simply npm linked, you'd want the .babelrc of the linked package to apply. The linked package should always have its own build config - the whole point of having separate packages is so that they can be published/installed independently of the app. If they rely on build settings from the app, they are coupled together.
2. I don't think sourcePackages is necessary. I can't think of a case where you'd want to link a module but NOT use its source code. We can always add this field later if it is a problem. What am I missing here?

As for your question about babel-preset-env settings, Parcel already uses the browserslist config from the host app to compile all dependencies that need compiling. Previously this was only dependencies that specified a higher target than the app, but now this will also apply to custom configs - the env settings always come from the app, not the module.

[pravdomil]

I'm afraid that we introduced new property under package.json. Do you consider use esnext property? http://2ality.com/2017/06/pkg-esnext.html#package-authors
I think that esnext is now being used for original package sources.

[devongovett]

The source field could refer to source files other than JavaScript, e.g. TypeScript or some other language. Seems like using esnext for that would be weird.

[rodoabad]

@devongovett is source a Parcel only feature?

[rodoabad]

Oh, this only works when you're linking. Not when it's published. I see.

[aleclarson]

@devongovett WDYT about publishing packages with a source field so they can be used in bundles that target modern browsers only? I once believed the module field served that purpose, but I found out recently that it's meant for import/export syntax only, not other ES6+ syntax.