Proposal: Modularize Library

[yuit]

Problem

Currently TypeScript compiler accepts a single compiler option, target, which specify both version of library to be included during compilation and version of JavaScript for emitting code together.
The behaviour presents limitations in two parts of the pipeline: consuming the default library and granularly controlling how JavaScript will be emitted (e.g. what features to be down-level etc.).
The proposal will mainly focus on the first issue regarding using the default library.

The current behavior of target option doesn't allow users to have a fine-grain control on what library or feature of the library to be included, and user cannot independently control library's version from emit JavaScript version. As summarized in #4692 Proposal: Granular Targeting and #4168 Normalize our lib files by compiler setting

• Emit ES5 JavaScript while using ES6 library during design-time (and vice-versa).
• Include only sub-part of the ES6/ESnext library
• Modifying default libraries can conflict with users-defined ones.
• Using host-specific library.

Workaround

As @yortus points out in the #4692 Proposal: Granular Targeting , there are possible workarounds for above issues by maintaing customized version of the default library while target another version of emit JavaScript.
Such approach is cumbersome to maintain and consume any necessary fixes of the default library.

Related Issues:

• #4692 Proposal: Granular Targeting
• #4168 Normalize our lib files by compiler settings
• #3215 New APIs added to lib.d.ts may break client codes. Allow duplicated members in interfaces? Make lib.d.ts overridable?
• #3005 Using ES6 type default library when targetting ES5 output

Proposed Solution

There are two parts to the proposed solution:

1. Introducing a new compiler flag --lib. The flag will be used to allow users to granularly control the default library.
   Note: target flag will remain. Its behaviors with --lib flag will be discussed below.
2. Breaking down the default library into smaller files, especially ES6 library and any future JavaScript library.

Compiler Flag --lib

The flag's options (see below for the full list) allows users to specify what library to be included into the compilation.
The flag's options can be separated into following categories:

• JavaScript only:

  es3

  es5

  es6

• Host only:

  node

  dom

  dom.iterable

  webworker

  scripthost

• ES6 or ESNext by-features options:

  es6.array

  es6.collection

  es6.function

  es6.generator

  es6.iterable

  es6.math

  es6.number

  es6.object

  es6.promise

  es6.proxy

  es6.reflect

  es6.regexp

  es6.string

  es6.symbol

  es6.symbol.wellknown

  es7.array.include

The --lib flag is an optional flag and multiple options can be used in combination (e.g --lib es6,dom.iterable) and each option will be mapped to associated library.
So in this example, files: lib.es6.d.ts and lib.dom.iterable.d.ts will be loaded into the compilation..
When --lib is specified, --target will be used solely to indicate version of emitting JavaScript.

In additional to above behavior, we still need to determine what to be included when --lib is not specified, should it remain the same, including version of library specifying by --target and all host libraries, or including every library. -> _Current implementation is to include version of the library using --target

The proposal will only affect when the compiler attempts to create compilation in Program.ts. Other parts of the pipeline will not be affected by the proposal.

Usage:

The --lib flag will take an options in the form of string separated by comma. Space will not be allowed as a separator. This rule apply to both command line and tsconfig

Valid

tsc --target es5 --lib es6
tsc --target es5 --lib es5,es6.array

"compilerOptions": {
    ...
    "lib": "es5,es6.array"
}

Invalid command-line

tsc --target es5--lib es5, es6.array  // es6.array will be considered a file-name
tsc --target es5 --lib es5 es6.array  // es6.array will be considered a file-name

"compilerOptions": {
    ...
    "lib": "es5, es6.array"
}

Working Items:

• Breaking up the library (PR Library modularization #6990, this PR is folded into Proposal: Modularize Library #6974)
• Adding compiler options (PR Proposal: Modularize Library #6974)
• Language Service support in VS
• Add node.d.ts and browser.d.ts

[yortus]

Thanks @yuit, great to see progress on this.

I'm wondering about how this handles cases where there is crossover between features. For example, consider the following project setup:

Example Project

• Browser-based, so it must either avoid unsupported features or ensure they are adequately polyfilled.
• Promise is polyfilled by the project as necessary so we can rely on that.
• Generic iterables are not widely supported (eg not supported in any IE or Safari version) so should not be used in source code. This means we want only one of the following two definitions:

interface PromiseConstructor {
    ...
    all<T>(values: ArrayLike<T | PromiseLike<T>>): Promise<T[]>;  // Do include this
    all<T>(values: Iterable<T | PromiseLike<T>>): Promise<T[]>;   // DO NOT include this
}

• Well-known symbols like Symbol.species and Symbol.toStringTag are not widely supported so should not be used in source code.

How to get just the right type definitions?

For the above project, I'd like to tell tsc: "I want to include definitions for Promise in the build, but NOT definitions for well-known symbols or methods that take generic iterables". That way, I get type safety for Promises and compile-time failure (rather than runtime failure) if I try to use unsupported constructs like Promise.all(someIterable).

I believe situations like this really get to the point of granular targeting. I've used Promise as an example but there are similar crossover issues with RegExp, Date, typed arrays, Map, Set, and other builtins. We want early failure for referencing things we already know aren't going to work reliably at run time. The compiler can really help here by letting us catch these errors at transpile time, rather than runtime.

How to handle feature crossover?

I may have missed something, but I don't see any handling of feature crossover in this proposal as it is currently stated. It looks to me like the following would be the best that can be done:

• Promise in es6.d.ts is currently all-or-nothing, including definitions that use well-known symbols and generic iterables, even if we explicitly want to avoid them.
• Alternatively, there could be a separate symbols.d.ts with well-known symbol declarations placed there, and taken out of places like promise.d.ts It could use declaration merging to add to existing declarations, like interface PromiseConstructor { [Symbol.species]: Function; }.
• But the declaration-merging approach will sometimes introduce declarations that we've told the compiler not to include. E.g. if we said we want to include symbols but not promises, then we'll still have a Promise and PromiseConstructor defined by symbols.d.ts.

This is where a tsc-internal mechanism that does the equivalent of conditional compilation shines - it starts with all known definitions and filters out anything not supported on a line-by-line basis, which covers cross-over between features perfectly at arbitrary levels of granuarity. See for example this es6.d.ts. I'm NOT suggesting conditional compilation, just a purely internal mechanism that does effectively the same thing to get the correct definitions, with no more and no less than what the user asked for via --lib.

[yuit]

@yortus Thanks for the feedback and pointing out some holes in the proposal.

How to get just the right type definitions?

For the above project, I'd like to tell tsc: "I want to include definitions for Promise in the build, but NOT definitions for well-known symbols or methods that take generic iterables".

Yes you are right that the current proposal when you pick feature such as promise.d.ts, it is all or nothing. You cannot just pick only sub-part of the promise.d.t.

So building upon your proposed solution, we can 1) separate symbol into its own file (probably same as iterator) 2) separate symbol in Promise or other es6 features in to their own files like es6.promiseWithSymbol.d.ts which will get included when you specified that you want es6.symbol. I am trying to not to overly granularly separate the files so we don't end up creating large amounts of tiny lib. though for this as you suggest here there may not be too bad. Also, to clarify, I think these smaller files like es6.promiseWithSymbol.d.ts should not be exposed as options to users.

[yortus]

@yuit sounds reasonable :)

I am trying to not to overly granularly separate the files so we don't end up creating large amounts of tiny lib

I agree there are definitely diminishing returns to supporting finer granularity. However, I do think the choice of "grains" should be informed by how the browsers/JS runtimes actually add feature support over time, which is what this issue is meant to address after all.

For example, the Symbol builtin is widely supported now, but well-known symbols still have very little support. I can definitely make use of Symbol in my node projects today, but I must avoid pretty much all well-known symbols. And they have pretty different use-cases anyway.

So, while I would not suggest having a separate flag for each of the 2 dozen or so well-known symbols, I would suggest having a separate flag for ES6.Symbol and for ES6.WellKnownSymbols, based on their actual availability.

I think these smaller files like es6.promiseWithSymbol.d.ts should not be exposed as options to users

Agreed, the number and names of all these .d.ts files can just be an implementation detail known to tsc. It can pick the right set of files according to a naming convention based on the supplied --lib flags.

[alexeagle]

@mhegazy note that it would be nice for one of the new lib.es6* files to be the minimal one for
tsc --target es6 --noLib node_modules/typescript/lib/lib.d.ts app.ts
(from #5504)

[yuit]

@alexeagle what you mean by lib.es6.* to be minimal? Do you mean you just want es5 lib? ( like what you describe in the issue #5504)?

[alexeagle]

First preference would be to support --target es6 with only the es5 lib, with a fix for #5504

If that's not possible for some reason, then I'd like to have the lib.d.ts that expresses just the types needed by the compiler to make it work, to make the workaround less painful. (I have Duplicate identifier in our internal build because of the types I had to specify in our custom std lib)

[yuit]

@alexeagle Thanks for the clarification. The new compiler flag will allow you to get your preference fix for #5504 😄
You will now be able to do --target es6 --lib es5 (and include any es6 per-feature library)

[mhegazy]

This should be fixed by #7715. --lib support should be available now in typescript@next and in the next TypeScript release.

[basarat]

Going through and adding to alm. I know it says "lib": "es5, es6.array". But wouldn't "lib": ["es5","es6.array"] be better in tsconfig.json? Is this how future array options (if there aren't any yet) be represented in tsconfig.json's compilerOptions?

Thanks! 🌹

[basarat]

More questions:

• What would be the result of getDefaultLibFileName in language service host (perhaps we should start going with FileNames (notice s) for these apis.
• Similarly getDefaultLibFileName should have an s suffix.
• Also does the user type es6.array or es2015.array. Note: the file is called lib.es2015.array.ts

Here is a sample of the function I needed to delegate to eventually:

export const getDefaultLibFilePaths = (options: ts.CompilerOptions): string[] => {
    if (options.noLib) {
        return [];
    }
    if (options.lib) {
        /** Note: this might need to be more fancy at some point. E.g. user types `es6.array` but we need to get `es2015.array` */
        return options.lib.map((fileName) => fileFromLibFolder(`lib.${fileName}.d.ts`));
    }
    return [fileFromLibFolder(ts.getDefaultLibFileName(options))];
}

With this I have it working in alm. Feedback appreciated 🌹

[basarat]

Is this how future array options (if there aren't any yet) be represented in tsconfig.json's compilerOptions

In fact rootDirs is an array from the discussion : #8245 (comment) Maybe lib should be too in tsconfig.json

[basarat]

Verified by @2426021684 (TypeStrong/atom-typescript#918 (comment)) it is an array, as it should be 🌹

[yuit]

Update our decision on to ship node.d.ts or not to ship, the outcome is that we will not ship the node.d.ts as our default lib for the following reasons:

• what version of node.d.ts we will support?
• nodejs constantly ship new version and cause change in node.d.ts, how we will ship the update in node.d.ts that we already ship (especially problematic in VS case)?
• additional effort to maintain yet a separate node.t.s instead of just in "@types"
• shipping node.d.ts doesn't seem to provide any much easier workflow, in comparison to just install one from "@types"