Distribution Formats

[kitsonk]

Thinking about how we should format distributions, I was thinking something like this:

dist
└ umd - individual UMD modules with external sourcemaps
└ es - individual ES6+ targeted modules in es6 module format
│      with inline sourcemaps and sources (essentially for
│      rebundling)
└ src - source TypeScript files
└ typings - Any typings for the package, including the packages typings
  index.js - a UMD module that contains all the packages modules in a single bundle
  index.min.js - a UMD module that contains everything minified
  index.js.map - A sourcemap file for index.js
  index.min.js.map - A sourcemap file for index.min.js
  package.json - CommonJS package meta-data
  bower.json - bower package meta-data
  README.md - Package's readme
  LICENSE - Package's license

I think it will be important to consumers to have easy ways to consume the packages in different workflows. I would expect our build pipeline would eventually just grab the /src and keep rolling things up into a final distribution build, but I can see where people will want to grab things like /es and use Babel or something else to do their targeting won't need to have TypeScript installed.

[dylans]

Is umd forever a strict es5? Or do we ever try to include some ES6 features that are widely adopted, but wrapped in umd given the missing ESM loader spec?

[kitsonk]

With the current TypeScript compiler (1.8), it is all or nothing. So my recommendation would be UMD would be ES5 target.

Even if there is granular targeting in TypeScript (which is likely for 2.0), we could create distributions that include some ES6 features, but which ones? While the expectation that the UMD would work in any environment (assuming that we plan to support IE11 for a while). I think anything finer grained would be dependent upon a build pipeline which would either use "pure" ES6+ or TypeScript as input.

I guess the only thing worth considering distributing is a "modern" build, which I would recommend being a single UMD bundle with certain modern features and be only for Node.js 4+, Chrome, Firefox, Edge, Safari 9+, iOS 9+, Android 5+, but that might be a rabbit hole.

[devpaul]

Will typings be provided in ES5 or ES6 format or both? Currently dojo/core doesn't seem to refer to any ES6 types like Symbols, Iterators, Iterable, or Generators. An example of this is in array.ts where we support anything ArrayLike, but don't provide support for Iterables.

[kitsonk]

Currently, but I have my PR for Symbols, Iterators, Set, forOf. There are already shims for Map, WeakMap and Promise. Once my PR is in, I think there are some refactoring to do make it easy for iterators to be supported well other places. There are some conflicts when targeting ES5/ES6, specifically in regards to Symbol, which is why there is a separate .d.ts that has to be loaded when targeting ES5 which isn't required when targeting ES6.

Generators are syntax and TypeScript are working on being able to down emit them.

[devpaul]

I took a look at the PR. I'm not sure what our approach should be for implementing shims in Dojo 2. I understand the value in supporting Promise and functionality that dojo depends on. However, if we plan on shimming all of ES6 and plan on keeping up with yearly ES releases it seems like this can quickly take over the dojo/core API and become overwhelming for development. How long will it be until most of dojo/core becomes redundant due to browsers catching up? For example, dojo/array in Dojo 1 is mostly redundant in an ES5 environment. How long until core/Promise and core/array befall the same fate?

My thought regarding typings were more about finding ways to add ES6 typings to modules where it might make sense. For instance, Symbol and Symbol.iterator are supported on every platform > IE11, so for project with a browser matrix Edge 12+ will dojo/core provide a good experience? Furthermore, how do we write dojo in such a way that TypeScript code in src can be written in ES6/modern ES and still provide typings for the down-emitted ES5 JS in umd?

For instance, in core/array.from() the definition is:

function from<T>(arrayLike: (string | ArrayLike<T>), mapFunction?: MapCallback<T>, thisArg?: {}): Array<T>

But for developers using ES6 the first argument should include iterables:

function from<T>(arrayLike: (Iterable<T> | ArrayLike<T>), mapFunction?: MapCallback<T>, thisArg?: {}): Array<T>

One possible solution is to selectively compile in es6 specific code that handles current ES features like iterators; e.g. core/src/es6/array.ts Another is to add an additional layer of typings; e.g. core/typings/core.es6.d.ts that adds handling for Iterable in array.from and just assume the native function will handle it. We could also try shimming everything, but what do we do with things like Symbol and Proxy that cannot be shimmed?

Going back to the original question, I think we can delay some of these decisions. If we do, I think the above proposal provides a good starting point. However, given this discussion we may want to remove the es targeted folder until we have good ES6 typings in Dojo 2 and a strategy for releasing down-emitted ES5 targeted code along-side ES6+ TypeScript and ES6+ Javascript.

Also, please let me know if I'm missing the point. In Dojo 1 we simply wrote everything to work in the lowest targeted browser and created shim or feature detection to use later methods. With TypeScript it seems that we must also take into account the most modern features for typings while writing code against the lowest common denominator of browser support.

[kitsonk]

However, if we plan on shimming all of ES6 and plan on keeping up with yearly ES releases it seems like this can quickly take over the dojo/core API and become overwhelming for development.

No, we never said all. We said those parts of ES6+ which we found useful.

How long will it be until most of dojo/core becomes redundant due to browsers catching up?

Lots of people say that, but Dojo has never been able to afford the assumption that browsers are supported. IE11 is still 23% of the market share, IE8 still 8.16%. Not supporting IE8 is still "controversial" for Dojo 2. Plus you have Android 4.4. It isn't all about desktop browsers.

For example, dojo/array in Dojo 1 is mostly redundant in an ES5 environment.

That only took 6 years.

How long until core/Promise and core/array befall the same fate?

See above.

But for developers using ES6 the first argument should include iterables:

If we believe we can shim Iterables (we can) and we like Iterables (we do) we should adjust the APIs so that someone can depend on core and write their code in a way where they don't have to think about it. TypeScript takes care of the syntax and dojo-core should take care of the functional shimming where appropriate. Specifically going back and updating the APIs post the landing of a iterable shim would be a task we should accomplish.

I think we need to bisect this issue though. There are really two parts to this I realise now:

• How do create our distributions in a more usable fashion? I think RxJS does this well. If you take a look the main npm distribution they include a fairly well structured set of distributables.
• How do we deal with ES6+ only distributions effectively. For that, I think we need a better understanding of how we are going to elide code with the has API as well as we need Conditional Decorators which are targeted for TypeScript 2.1 and we need Granular Targeting which doesn't have an item yet on the TypeScript roadmap but it is committed (I suspect it is an omission, likely 2.1).

Let's keep this issue focused on distribution formats and open a new issue regarding the second point. But in summary, we need to still write our core libraries shimming as much ES6 functionality as we need or find useful. End users can easily narrow the targeting of what UAs they can deal with, we don't have that luxury.

[devpaul]

I took a look at how RxJS handles ES6 types like symbol and found the same issue there as I'm trying to resolve here.

When RxJS uses an iterator it does so with a shim created in RxJS: https://github.com/ReactiveX/rxjs/blob/master/src/symbol/iterator.ts

As a result, RxJS often uses any types when referring to ES6 behaviors (https://github.com/ReactiveX/rxjs/blob/master/src/observable/IteratorObservable.ts#L68).

In dojo/core/array.from() we have an opposite, but related issue since from()'s typings are more specific than the ES6 implementation (while RxJS's is too broad).

I think Conditional Decorators and Granular Targeting offer good solutions for this. Thank you for pointing me to those. Does this mean we are going to wait to address ES6 typings until TypeScript 2.1? If we wanted, we could create a separate array.core.es6.d.ts typings to address this for ES6 users, which will in-turn provide proper intellisense and type hints for ES6 Dojo users. If we don't do this, should we broaden some of our typings (like RxJS does) to allow developers of ES5 and ES6 to use the same typings? And if we don't do any of this, is it appropriate for us to release an ES6 dist?

[kitsonk]

Yes, I agree, we will have to put ES6 typings on hold until we have the Conditional Decorators and Granular Targeting. @tomdye is working on the ES6 compiling stuff. Ideally we should integrate whatever solution we come up with into SitePen/dts-generator.

So for now, I propose we do this:

dist
└ umd - individual UMD modules with external sourcemaps
└ src - source TypeScript files
└ typings - Any typings for the package, including the packages typings
  index.js - a UMD module that contains all the packages modules in a single bundle
  index.min.js - a UMD module that contains everything minified
  index.js.map - A sourcemap file for index.js
  index.min.js.map - A sourcemap file for index.min.js
  package.json - CommonJS package meta-data
  bower.json - bower package meta-data
  README.md - Package's readme
  LICENSE - Package's license

I will create a separate issue for ES6+ typings.

[kitsonk]

@bryanforbes is going to do some thinking on this too... moving to him for now.

[kitsonk]

I have done some thinking about this and @novemberborn, @tomdye, and myself had a chat which provoked me. Also, @kfranqueiro brought up some thoughts and considerations.

Here is what I propose:

• Public npm and bower packages contain:
  • All modules compiled in UMD format with external sourcemaps which contain the TypeScript source as part of the source map.
  • A built single "rolled-up" UMD module, a minified version of this and built minified separate UMD modules all with appropriate source maps.
  • Package typings
• @dojo name spaces packages contain (essentially this is the source repo with a few things built):
  • TypeScript source
  • UMD built modules
  • ES6 modules
  • Unbuilt tests
  • Package typings
• The GitHub source repo would not have any built versions, but the npm script would have a post install script that would essentially build the package to replicate the @dojo layout. That way, people could depend upon the GitHub repo and have a nice built version installed from source.

I suspect that this would require us to change the layout of our current packages so that the structure does not differ significantly from the built and unbuilt packages. I would propose that we:

• Put compiled UMD source modules into /lib
• The compiled UMD "rolled-up" module and minified version would be named /main.js and /main.min.js
• Put compiled ES6 modules into /esm

[novemberborn]

The recent .jsm kerfuffle makes me wonder whether each module format should be distributed separately, so cjs, amd and es6. UMD can lead to weird issues as we saw with Esprima. Admittedly that was a bug, but if a Dojo package is used as a dependency it might similarly and unexpectedly be affected by a global define.

Dojo packages tend to contain multiple public modules, e.g. dojo-core/Promise and dojo-core/async/Task. Requiring them like that is preferable over dojo-core/lib/Promise. I imagine AMD loaders and WebPack (for ES6 hopefully) can be configured to map dojo-core to dojo-core/lib, but Node/CJS can't. This would point point to writing CJS or UMD files directly in the package root.

What would the export be of the rolled-up main? require('dojo-core').async.Task? It will lead to problems if that class is a different object from dojo-core/async/Task, especially in Node. Most Dojo packages don't have a clear main (dojo-compose being one notable exception). Perhaps it'd be better to place rolled-up and minified versions in a dist folder, e.g. dojo-core/dist/core.js.

Tests typically shouldn't be distributed. You can't run them anyway without the dev dependencies.

I'm not 100% sure, but I suspect that when installing a package from GitHub, npm won't install dev-dependencies. This would make it harder to automatically build the @dojo equivalent upon installation.

[kitsonk]

The recent .jsm kerfuffle makes me wonder whether each module format should be distributed separately, so cjs, amd and es6. UMD can lead to weird issues as we saw with Esprima. Admittedly that was a bug, but if a Dojo package is used as a dependency it might similarly and unexpectedly be affected by a global define.

Possibly, but then I would do CJS, AMD, ES6 and UMD. I still don't think there is a big risk with UMD. We had a loader defect, which we fixed. I still think we should have a "ready to go" distribution that focuses on the main way we expect the modules to be consumed. We have modules that are dependent on AMD plugins, therefore we could say the "official" public distribution is only AMD.

Dojo packages tend to contain multiple public modules, e.g. dojo-core/Promise and dojo-core/async/Task. Requiring them like that is preferable over dojo-core/lib/Promise. I imagine AMD loaders and WebPack (for ES6 hopefully) can be configured to map dojo-core to dojo-core/lib, but Node/CJS can't. This would point point to writing CJS or UMD files directly in the package root.

My concern was spitting modules in the root directory when sources are elsewhere in certain distributions is problematic. Having them located in the lib is not necessarily problematic given an AMD loader (and I assume SystemJS has a similar functionality) given a package map of:

{
    packages: { 'dojo-core': 'node_modules/dojo-core/lib/' }
}

What would the export be of the rolled-up main? require('dojo-core').async.Task? It will lead to problems if that class is a different object from dojo-core/async/Task, especially in Node. Most Dojo packages don't have a clear main (dojo-compose being one notable exception). Perhaps it'd be better to place rolled-up and minified versions in a dist folder, e.g. dojo-core/dist/core.js.

Asking realistic questions, shame on you. 😄 I assumed they would be an AMD layer bundle, therefore exporting the mids versus the export structure. Therefore in CJS land I would assume it would be like this:

require('dojo-core')['dojo-core/async/Task'];

But in TypeScript land it would just be the same old thing and in AMD, it would be loading a layer.

I'm not 100% sure, but I suspect that when installing a package from GitHub, npm won't install dev-dependencies. This would make it harder to automatically build the @dojo equivalent upon installation.

Google Foo suggests an approach

[novemberborn]

Possibly, but then I would do CJS, AMD, ES6 and UMD. I still don't think there is a big risk with UMD.

Sure. I'm just being overly precise 😄

My concern was spitting modules in the root directory when sources are elsewhere in certain distributions is problematic. Having them located in the lib is not necessarily problematic given an AMD loader (and I assume SystemJS has a similar functionality) given a package map

It's just that CJS doesn't have a package map, and Node may not gain one for ES6 either.

in CJS land I would assume it would be like this:

require('dojo-core')['dojo-core/async/Task'];

The problem would be that that Task !== require('dojo-core/lib/async/Task').

[kfranqueiro]

Can someone explain the benefit of including a "single rolled-up UMD module"? Seems to me that packages are more often than not a collection of semi-disparate things that users are more likely not interested in loading 100% of, and would be better off relying on a build or rollup step of their own to generate a layer/bundle of everything they want. The proposed "CJS land" way of exposing/loading stuff from it also seems super-unorthodox (and for that matter, on the Node side of things, the entire thought of a rolled-up module seems less relevant).

I'm also concerned that providing the rolled-up module OOTB might again contribute to false claims of "oversized kitchen sink" like we already get in Dojo 1 by people who don't/didn't understand how modularity and builds worked (though admittedly the lack of pruning in Dojo 1's build system also contributed to that in a sense).

[kitsonk]

Can someone explain the benefit of including a "single rolled-up UMD module"?

I will admit I am 50/50 on it. I suggested it because it would be a simple "take it and run" scenario, especially if many of our packages are not semi-disparate things. Yes, core will be, but most of our other packages so far are starting to look like handful of modules where the modules are mainly to make it easier to maintain the code.

One thing I thought about writing was "if it is appropriate for the package" to have one consumable rolled up module, but the "if appropriate" would cause the distributables to be inconsistent.

[tomdye]

I think that a rolled up (or at least an optimised rolled up) module may well be the only 'easy' way for us to provide webpack support unless we create a work around for the AMD plugins.

But I would hope that an end user would create their own selectively from source to avoid bloat

[novemberborn]

most of our other packages so far are starting to look like handful of modules where the modules are mainly to make it easier to maintain the code.

I think it's fine to have a main which reexports some of those modules. With proper documentation you can then either load the individual modules or just the main.

Rolling up implies duplicating the code in the main, which seems wrong.

[devpaul]

We should also consider targeting CDNs (i.e. google) for distribution. Not sure if that would add weight to having a rolled-up minified UMD release or not.

[kfranqueiro]

Something that came up RE potentially publishing dgrid to npm may also have bearing on the idea of double-publishing these packages (once under @dojo, once outside):

There is no way to alias packages in npm, which I presume would mean you would have to reference each of the two packages differently (or move/rename/symlink one manually), which seems like it could be highly annoying and lead to inconsistent docs/usage between the two versions.

[kitsonk]

@gbaumgart I took a look at your guide lines. There are a few things to consider:

• Dojo 2 is being built on TypeScript, therefore the source code will have to be transpiled into JavaScript to run in an environment.
• We will unlikely every check that transpiled source code into the GitHub repos, in part because it follows your first guideline that we shouldn't "waste" disk space, therefore what gets checked into the GitHub repo is the minimal needed to create the end consumable code, but does not include that.
• Because of that, installing a package from GitHub will always require some sort of post-install step, which, given certain package managers, we could attempt to automate, otherwise it will be up to the package consumer to perform the necessary complication.
• We are trying to make the available distribution packages available in a way that meet most general use cases, but the biggest challenge we have is like we had with Dojo 1. Because our distributions were loads of individual modules to avoid duplication of code, we found many developers then felt that format was "production ready" and people would then complain "Dojo takes forever to load and is so slow", which it isn't, it is just if you try to load 100 modules over an HTTP 1.1 3G connection, it will be slow, but a "built" version with most of the modules built into layers is highly efficient.
• We agree that no package manager lock-in is a good idea.

[kitsonk]

Ok, several of us discussed this and this is what we came to:

• We will provide distributions that are individual UMD modules that "ready to go" that are the straight transpiles from TypeScript, targeted at both ES5 and ES6. The will inline source maps and sources.
• We will provide the original TypeScript modules in the distributions for people to build
• The Dojo 2 build tooling will consume the TypeScript modules and provide minified built layers.
• We may provide alternative "ready to go" distributions of something like dojo/widgets which would include all the dependent packages into a built, minified layer with the dojo/loader so that people can use a key part of Dojo 2 without additional steps.