Let's discuss TypeScript support.

[danielkcz]

I am well aware that this is Babel world project and it was meant like that from the start. However, I would like to initiate the discussion here what it would take to have macros available in TypeScript world. Macros are such a great concept it would be almost a crime to leave it like this.

Problem

I am mostly coming from the situation I am trying to solve now. The LinguiJS is going to be using macros to transform i18n elements. While in a Webpack world it's possible to run ts-loader and babel-loader in tandem, it already introduces challenges, eg. keep JSX untransformed so Babel can have enough information and do JSX transformation on its own later along with Macros.

Also, other toolings needs to solve the same. Be it extraction process of Lingui itself or Jest testing or any other bundler I suppose. None can use Webpack loaders and have its own way around. This is no good for a whole ecosystem in my opinion.

I am sure this situation is not unique and with rising adoption of TypeScript there will surely be more projects popping out trying to solve the same. I feel it would be rather unhealthy to keep ecosystem separated like this. Not only for library authors, but for users as well.

Solutions?

TypeScript has it's own "plugin" system or rather it's called custom transforms. There is some AST apparently, but I do have trouble finding a real documentation for that. Honestly, I've never been fiddling with AST in Babel, so I am probably the least capable person to deliver any valid information in here.

Would it be viable to basically have a Typescript transform that can understand babel macros out of the box? Most likely that due to AST difference the authors of macros would need to account for both variants, but on the user end it would mean complete transparency as you could use either language without hassle.

Goal

To prepare a TypeScript plugin/transform that would work in a similar fashion to babel-plugin-macros allowing user code driven transformation instead of relying on bunch of hard-to-configure plugins. Ultimately, if we can achieve some level of compatibility between these projects and bridge the gap, it make life easier for everyone.

Current stage

None :) We are seeking some smart people who have insight into TypeScript transforms and AST. We need to create a prototype that mimics the behavior of babel-plugin-macros at some basic level to verify the core idea.

Why not to use Babel-Typescript support?

One might argue that since there is TypeScript support in Babel now, it's a clear path and no need to bother with this. I would agree in case that support would be building on a real TypeScript compiler. Since it's completely separate implementation, it might introduce unexpected errors and incompatibilities. VSCode checking can tell A and then Babel will transform it into B. Current lack of some features is an example of that and more might come.

Personally, I think it's a wrong path, but that's not the point of this discussion. Perhaps I am mistaken on this and I would love to be proven otherwise.

[kentcdodds]

I'm definitely in favor of supporting macros in TypeScript, but I'm totally the wrong person to work on that because I don't use TypeScript at all. I don't know whether it'd be better to have TS support in this same project or if it should be a different project.

I think it would be cool if macros authors could publish a single package that supports both though. Here's what I think we should do:

1. Explore supporting macros using TypeScript's plugin system
2. Implement it
3. Implement an existing babel macros with the TypeScript one
4. Report back on what was found.

I think this would be great. The lack of custom transforms is one of the biggest reasons I don't want to use TypeScript (it may be the only remaining reason). If we can get macros to work then I'll probably have no more reason to resist :)

[danielkcz]

@kentcdodds I am we aware that you are more of the FlowType guy and I respect that. I just felt this is a good place as any to open the discussion :)

I like your proposed route. Would love to get some people on board here who have a more insight into how is that working. There are very few of those typescript transforms and it's a kinda uncertain area.

Apparently, there is also a long standing discussion on simplifying the use of transforms. As of now, it's not as simple as adding them in config. It has to go through API layer. The ts-loader for a Webpack allows that and probably some other tools as well, but it's far from ideal.

Also, I found this interesting extension to VSCode. Just from that image it looks like that TypeScript AST is not that much different, but I am no expert for sure.

[kentcdodds]

Interesting!

Well, as I said, I wont be working on this, but I'd love it if someone were to do it :)

Considering config is so complicated, that makes the case for macros even more compelling. Because people don't have to touch config more than it takes to add the macros plugin, then they can use all the macros they want without messing with config.

[danielkcz]

@kentcdodds Feel free to point some people in here. I certainly do not have such a good network like you and not sure who to ask for help.

Perhaps consider RT: https://twitter.com/danielk_cz/status/1071761665631367168 (or make your own)

[sibelius]

As Typescript is supported by Babel 7, can’t we just use Babel to keep this plugin compatible with pure js, flow and TS?

[danielkcz]

@sibelius Please read the last paragraph in initial comment...

One might argue that since there is TypeScript support in Babel now, it's a clear path and no need to bother with this. I would agree in case that support would be building on a real TypeScript compiler. Since it's completely separate implementation, it might introduce unexpected errors and incompatibilities. VSCode checking can tell A and then Babel will transform it into B. Current lack of some features is an example of that and more might come. Personally, I think it's wrong path, but that's not the point of this discussion. Perhaps I am mistaken on this I would love to be proven otherwise.

[vhfmag]

I have been rooting for macros in Typescript for quite a long time and that feature request has been discussed for at least 3 years (since 2015 - microsoft/TypeScript#4892 (comment)), so I would love to see this being worked on! That said, as close as their ASTs may be, a hybrid Babel/Typescript macro plugin would have to implement a unified API for them, which could greatly improve complexity and maintenance burden (but that should be further studied before arriving at conclusions).

As for @sibelius suggestion, I think that's at the very least a viable option for the near future, especially because babel plugins and macros do have type information and can use it to generate code. It's been something I've had in my mind since I've first heard about Babel Typescript support, so I decided to finally implement a proof of concept.

With tsguard.macro, I'm trying to implement type guards generated at compile time (it's my first ever babel plugin or macro, so be careful down there). I did it with babel-plugin-macros and it works just fine for that use case. There are some disclaimers about this approach, though:

• Although it is possible to modify type information in the AST, that isn't used for type checking, since it is done by typescript (that is the most important use case for direct typescript macros).
• Advanced typescript features, like conditional and inferred types, require either reimplementing their logic or converting Babel's AST to Typescript's AST, then work with it in Typescript and finally convert it back (which kind of nullifies my point in the first paragrah)

Even with its limitations, the Babel Typescript macro approach can enable several use cases:

• Generating React's propTypes from Typescript types 🎉
• Otherwise generating values from types, like ts-transformer-keys (which generates an array with all of an interface's keys) or ts-transformer-enumerate (which generates an array with all possible values of a union type of string literals), etc.

[longlho]

What are you trying to achieve specifically w/ macro support in TS? The transformer API already does what you're looking for.

• If you want things like scope in the example it's already doable, given that you provide the correct type definition
• If you want custom syntax like CSS in JS or graphql it's also already doable.
• If you want to tap into type checker it's not doable right now, for good reason, cause it's super complex and will likely slow down compilation.

If you're talking about cross-tooling problem, it's kinda moot bc it'll always be a problem, unless all tool developers synchronize on their development initiatives & API. Say there's a working solution now, will in work w/ TS@4, webpack@18 & React@20 or whatever.

Some build systems aren't even compatible at all, e.g: PostCSS plugins are async (can be sync but few support it), while TS transformers/jest custom plugins are sync as it modifies AST synchronously. Not to mention things like generating proper sourcemaps.

A lot of production build systems also don't go directly from source to bundled assets (at least at places where I've work at). It's typically source code from a bunch of libs -> ES6 -> published to internal npm -> app code consumes -> webpack -> bundled assets so ES6 is basically the cross-platform binary. The build problem basically breaks down to lib -> ES6, then ES6 -> chunking, which is a lot more sane and is kinda the only solution I've seen that scales.

[kentcdodds]

@vhfmag

that feature request has been discussed for at least 3 years

Just to be clear for everyone who's new to babel-plugin-macros... When we say "macros" here, we're not talking about typical macros that teaches the complier new syntax tricks. These macros aren't adding syntax to the language, they're simply providing an easier way for manipulating the AST that the compiler creates.

With that in mind, @longlho, when you say "The transformer API already does what you're looking for" that's great news! I'm not proposing that we change anything in TypeScript to support this, we just need to use what's already provided by TypeScript to allow for the kinds of macros people build with babel-plugin-macros.

We're not really talking about a cross-tooling problem yet. Right now we're just trying to see whether it's feasible to have a tool like babel-plugin-macros for TypeScript. From the look of things, the biggest road block is:

Apparently, there is also a long standing discussion on simplifying the use of transforms. As of now, it's not as simple as adding them in config. It has to go through API layer. The ts-loader for a Webpack allows that and probably some other tools as well, but it's far from ideal.

[longlho]

@kentcdodds It's already doable to have macros in TS. I think the problem that @FredyC is talking about is:

1. You can't inject transformer in TS compiler via tsconfig. You have to write a wrapper on top of the Compiler API yourselves.
2. babel & TS AST aren't compatible.
3. TS Transformers can't tap into typechecker so while you can introduce custom syntax during transpiling process, it won't compile.

[danielkcz]

@vhfmag

That said, as close as their ASTs may be, a hybrid Babel/Typescript macro plugin would have to implement a unified API for them, which could greatly improve complexity and maintenance burden (but that should be further studied before arriving at conclusions).

I agree. This is more like a wet dream than something that can become reality. I would definitely like to put the main burden on macro maintainers so they can decide if it's worth for them to support TypeScript or Babel or both. There should be some common API and helpers, but AST differences would always mean double work. Ultimately, it means less burden for users of such universal macros and that's the most important.

As for @sibelius suggestion, I think that's at the very least a viable option for the near future, especially because babel plugins and macros do have type information and can use it to generate code.

Please, let's not discuss this approach anymore here. If someone likes it, they can use it just fine. I don't like it and I've explained my reasons in a first comment. No need to further come up with theories in this matter.

[danielkcz]

@longlho

What are you trying to achieve specifically w/ macro support in TS? The transformer API already does what you're looking for.

I would quote Goal from initial comment

To prepare a TypeScript plugin/transform that would work in a similar fashion to babel-plugin-macros allowing user code driven transformation instead of relying on bunch of hard-to-configure plugins. Ultimately, if we can achieve some level of compatibility between these projects and bridge the gap, it make life easier for everyone.

I assume you haven't been using/writing macros yet. The major difference is that instead of having bunch of separate transform that you need to configure (which is painful) you would have just a single transform and everything else is driven from user code. You just install/write macro and use it directly in the code without any further configuration.

Some build systems aren't even compatible at all, e.g: PostCSS plugins are async (can be sync but few support it), while TS transformers/jest custom plugins are sync as it modifies AST synchronously. Not to mention things like generating proper sourcemaps.

That's a bit troubling for sure. I assume it would require to make synchronous version of such plugins. I really don't expect to have a plug & play of currently existing plugins 😄

A lot of production build systems also don't go directly from source to bundled assets (at least at places where I've work at).

I am not sure if I understand how is it related to discussion. In the end macros are just another transformation plugin and it's up to you where in your pipeline it will be.

[danielkcz]

@kentcdodds Thank you for clearing that out, much appreciated.

From the look of things, the biggest road block is

It may not be such a big problem after all. There is already ttypescript project which shows how it can be done. End users could either use that or I can imagine making something similar that would just insert macros transform and nothing else to ease on a configuration. Point being it's actually solved problem and we don't need to focus on that right now.

[danielkcz]

@longlho

3. TS Transformers can't tap into typechecker so while you can introduce custom syntax during transpiling process, it won't compile.

I don't think we need that. Let's leave type checking to Typescript alone. Macros should be really only about transformation of a syntactically valid code into something else, also syntactically valid. For example in case of Lingui project it's capable of transforming <Trans>text</Trans> into <Trans id="text" /> to basically unify stuff. These small bits and pieces is what's this should be about. We are not going to transform whole file to something completely different :)

[kentcdodds]

I would definitely like to put the main burden on macro maintainers so they can decide if it's worth for them to support TypeScript or Babel or both. There should be some common API and helpers, but AST differences would always mean double work. Ultimately, it means less burden for users of such universal macros and that's the most important.

I agree with this. The nice thing about babel-plugin-macros is that often the interaction with the actual AST can actually be pretty minimal, so I see a great opportunity for code reuse here. And even if there weren't much opportunity for code reuse, just having the capability for TS would be a great win.

---

Anyway, we're not here to justify the existence or usefulness of this style of macros with TypeScript, just how to make it work.

[longlho]

Actually I've written a bunch of ts transformers (like https://github.com/longlho/ts-transform-css-modules) so I do feel the pains you're talking about. TS compiler isn't designed in the same way as babel & its compiler API can be backwards incompatible during minor version (or even patch version). It's designed to be monolithic, transformer API was a huge deal, primarily driven by Angular bc they use TS in their AOT compiler.

You have to tap into typechecker for some macros especially when you want to maintain type safety in your macro. E.g: The <Trans/> component, when being replaced during compilation, will have to produce something of the same type. At the same time, it'll also have to modify the emitted type declaration files to adjust to the new type as well so it's not as simple as just changing AST syntax.

I added transformer support to ts-loader (TypeStrong/ts-loader#535) and ttypescript is very similar. The majority of the code is wrapping around the CLI and it's quite a hassle to upgrade everytime TS compiler changes.

But anw, if babel-plugin-macros wants to support TS, looks like it'll just have to bridge the AST API. The bulk of the work would be on macro authors to make sure typecheck works.

Oh, and accurate source map is a whole other can of worms....

[kentcdodds]

Thanks for the input @longlho!

So I think we're back to this:

I think it would be cool if macros authors could publish a single package that supports both though. Here's what I think we should do:

1. Explore supporting macros using TypeScript's plugin system
2. Implement it
3. Implement an existing babel macros with the TypeScript one
4. Report back on what was found.

Once we've done that we can see whether including this within babel-plugin-macros makes sense or if it'd be better to just be a separate project (I think it'd probably be better to be separate).

[longlho]

Yeah I agree. If we abandon the whole using Babel AST to write macros & just use TS AST to write them, it's completely doable. The work would just be recreate the babel-plugin-macros interface in TS.

The scope example in this repo is very easy (we already did stuff like that like https://github.com/longlho/ts-transform-react-intl/blob/master/src/macro.ts) where _('asd') tags it for translation.

Even https://github.com/kentcdodds/babel-plugin-preval is also fairly straightforward. As long as the macros don't involve complicated types, they're all doable. The value really comes in though when you can change types. That's way more powerful.

[danielkcz]

You have to tap into typechecker for some macros especially when you want to maintain type safety in your macro. E.g: The <Trans/> component, when being replaced during compilation, will have to produce something of the same type. At the same time, it'll also have to modify the emitted type declaration files to adjust to the new type as well so it's not as simple as just changing AST syntax.

Ah, haven't thought that about, you are completely right. It depends of complexity of concrete plugin, but key point here is that the resulting transformation is not some "in the wind" code. It's perfectly working code that user can import instead of macro one and use that. Macro would just need to switch that import and it would work just fine. It's definitely extra layer of concern, but nothing too huge imo.

The majority of the code is wrapping around the CLI and it's quite a hassle to upgrade everytime TS compiler changes.

Is API of the compiler really changing that much often? I wasn't really following that. It would be definitely better if TS would support plugins out of the box, but until then I guess that hassle needs to be accepted.

Oh, and accurate source map is a whole other can of worms....

Is it? I mean if TypeScript can do source map by comparing source code with final output, it shouldn't matter how many transforms are in the middle? I don't know source maps on such level to recognize an issue here. Is it actually necessary for each transform to somehow instruct how the source map should look like?

[kentcdodds]

The work would just be recreate the babel-plugin-macros interface in TS.

It's a good thing that babel-plugin-macros is actually pretty small and has had very minimal changes in the last year :)

[longlho]

@FredyC https://github.com/Microsoft/TypeScript/wiki/API-Breaking-Changes should give u an idea on how often the Compiler API changes. Smaller breaking changes like type union aren't documented. E.g: They change transformer return type from TransformerFactory<ts.SourceFile> to TransformerFactory<ts.SourceFile | ts.Bundle> during some commits.

Source map includes line #s & column #s and will have to be mapped differently when u modify the AST. The complexity comes in when you pull in external resources (like CSS for example), where, ideally you've have to bridge to the CSS source map as well. Source map also doesn't work perfectly in a transformation chain.

[longlho]

side note @kentcdodds what are the macros that use babel-plugin-macros right now? is there a registry?

[kentcdodds]

There's a list here: https://github.com/jgierer12/awesome-babel-macros

There's also a keyword people are supposed to use: https://www.npmjs.com/search?q=keywords:babel-plugin-macros

[danielkcz]

I want to make one thing clear here. I started this discussion and I certainly would like to guide it through, but I won't be much involved regarding coding except some code reviews if needed. I have really shallow knowledge about all this and I don't feel like I want to dive into that too much.

So what we need right now is some volunteer(s) who could examine how macros are working and try to implement some very basic transform that can handle simple macros. Let's ignore details like source mapping and anything that's not really important right now.

Personally, I would pick a macro like https://www.npmjs.com/package/ms.macro to be an initial target for an implementation. Or if you have some other preference, I don't really mind. Let's have it in a separate repo for now and later we can see how much common ground is there and if it's worth it to have a single project.

So ... volunteers? :)

[xiaoxiangmoe]

@FredyC https://github.com/xiaoxiangmoe/typescript-macros/

Babel-plugin-macros use mutable API, this does not seem to be compatible with TypeScript transformer, so I chose to use immutable API.

[danielkcz]

@xiaoxiangmoe I had just a brief look, but I am bit confused. Is there a separate transform there or is it all bundled in that single package there? It's definitely cool if it works, but next step then should be to separate common transform (macros engine) and the actual code required for macro.

Edit: Duh, nevermind me, should have read the checklist in readme more carefully :) Go on, you are definitely on a good path there.

[xiaoxiangmoe]

@FredyC
I'll only publish the packages/typescript-macros in npm latter.
package in the examples directory is private and it only works for debug macros engine.

[danielkcz]

@xiaoxiangmoe Ok I understand now, haven't thought that examples contain code for actual macros :) This is looking really well. Amazing how fast you managed to put it together.

I am amused by uppercase macro, that's crazy overkill, especially seeing how the macro itself looks inside 😮 However the hooks macro looks fairly useful already. Nice work indeed!

[xiaoxiangmoe]

Well, it still doesn't explain why there is a duplicit declaration of uppercase function, in macro itself and in the interop as well. Doesn't make any sense.

Ok, you are right, it is useless now.
I remove function body of this.

see:
xiaoxiangmoe/typescript-macros@91c5e10

[kentcdodds]

Ok, so I'm not totally sure I understand what's going on here, but if you'd like to make a pull request on babel-plugin-macros for what changes we would need to change to make this work (without breaking things) that would help me understand I think.

[xiaoxiangmoe]

@kentcdodds Nothing will change for babel-plugin-macros.
Maybe you could try this

git clone https://github.com/xiaoxiangmoe/typescript-macros.git
cd typescript-macros
yarn
yarn build
cat examples/lowercase.usage.js/dist/main.js

And you'll see

// file: examples/lowercase.usage.js/src/main.js
import lowercase from '@-.-/lowercase.macro';

export const van = lowercase(
  "My name is Van. Uh...I'm an artist. I'm a performance artist."
);

transformed by babel-plugin-macros into

// file: examples/lowercase.usage.js/dist/main.js
const van = "my name is van. uh...i'm an artist. i'm a performance artist.";

export { van };

[xiaoxiangmoe]

This would require a small amount of change on my end, but it would not be a breaking change for macro authors or users.

Your end and macro authors or users will have no change if they don't want to support TypeScript

If they want to support TypeScript in single package, see example lowercase.macro

[danielkcz]

@xiaoxiangmoe It still doesn't make sense to me. I am confused the most by the interop package. Does it mean that alongside the regular macro package you need the extra one to do what actually? If the interop package is meant to be some kind of universal macro to generate macro then it shouldn't contain any reference to "uppercase" macro.

And what about that weird @-.- scope? Is that going to be some official scope for all macros? :)

[xiaoxiangmoe]

I say it just a prototype.

see https://github.com/xiaoxiangmoe/typescript-macros/blob/master/examples/README.md

all package's start with scope @-.- which is private scope for my own use.

I'll provide an officially interop macro package without @-.- scope future.

I still don't know why you think interop package didn't make any sense. It make sense for our TypeScript users. It help us generate dts file.

[xiaoxiangmoe]

If you like to write dts file like JavaScript users, then it really doesn't make sense to you now.

[danielkcz]

I am sorry, @xiaoxiangmoe, it's kinda hard to follow your thoughts on this. I don't want to offend you or anything, but your weaker English is not helping it either :)

Um, can you like remove uppercase from the interop package to make it's more clear that it's the universal package? Let's un-prototype it a more to get a full picture, please.

[xiaoxiangmoe]

I'm sorry for my poor English. I don't konw how to express my thought to make it more clear.
Maybe I should write document for users. Could you please help me polish the wording and grammar? Thanks.

[xiaoxiangmoe]

@FredyC Oh, the uppercase in the interop package is useless, Thank you.
I'll refactor the code to make it more clean and clear.

[xiaoxiangmoe]

@FredyC see codes now.

If there are still some things that make you wonder, please tell me, thank you.

[danielkcz]

Maybe I should write document for users. Could you please help me polish the wording and grammar?

That would be a great next step. For users, I assume you mean macros authors? The end users should probably get documentation from macro author.

Please document why is the interop macro needed and ideally show how it would look like without it to have a comparison.

[danielkcz]

@xiaoxiangmoe Do you have some progress? Don't worry if you don't, I am just curious :)

[tricoder42]

I haven't seen it mentioned it thread and I've just came across typescript-eslint:

This parser is somewhat generic and robust, and could be used to power any use-case which requires taking TypeScript source code and producing an ESTree-compatible AST.

I'm still hoping to have generic macros working both with babel and TS…

[longlho]

I wouldn’t bank on it. Linter doesn’t care about types (thus not needing tychecker) but macros do.

…

On Sat, Jan 19, 2019 at 4:01 AM Tomáš Ehrlich ***@***.***> wrote: I haven't seen it mentioned it thread and I've just came across typescript-eslint <https://github.com/typescript-eslint/typescript-eslint/tree/master/packages/typescript-estree> : This parser is somewhat generic and robust, and could be used to power any use-case which requires taking TypeScript source code and producing an ESTree-compatible AST. I'm still hoping to have generic macros working both with babel and TS… — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMGb7FOvBmBRr3kBlERb6oULIMdUyU_ks5vEt7egaJpZM4ZKIG4> .

[tricoder42]

Ah, sorry. The package is named typescript-estree. Yes, it's used internally by eslint, but doesn't have to be used only for linters. It generates ESTree-compatible AST from TypeScript source code. Babel AST format is based on ESTree with few deviations.

Theoretically it could work:

• Babel: babel-parser -> macro build on top Babel AST
• TypeScript: typescript-estree -> convert ESTree to Babel AST -> macro build on top Babel AST

The middlestep for typescript could be omitted in future according to this comment:

I work closely with TypeScript Team and we are gradually aliging the AST of this project with the one produced by Babel's parser. To that end, I have created a full test harness to compare the ASTs of the two projects which runs on every PR, please see the code for more details.

[xiaoxiangmoe]

@tricoder42 What I am worried about is that eslint may can only parse AST, and can't do some semantic conversion like typescript language service.

[xiaoxiangmoe]

@FredyC I plan to use a library (ts-creator) to simplify my code.

[xiaoxiangmoe]

@FredyC see https://xiaoxiangmoe.github.io/typescript-macros/

[danielkcz]

@xiaoxiangmoe Thanks, looks like nice kick off for the documentation. I've left some preliminary comments in the commit. Not sure about ts-creator and how it fits in the whole thing, but if you think it will make things easier, then go ahead :)

[xiaoxiangmoe]

Hi all, do you think we should support an ts.LanguageService param for macro authors?

ttypescript not support type=ls now.

[danielkcz]

Well, seems this has kinda died. Honestly, I've already migrated to Babel TypeScript after some hurdles and lost motivation to move this forward. Big thanks to @xiaoxiangmoe for taking such heroic stab at it 🥇

I will close it so Kent does not need to suffer such long term issue sitting here 🙈 Feel free to ping me in typescript-macros repo.

[xiaoxiangmoe]

@FredyC I also migrated to Babel TypeScript now.

[danielpza]

Checkout https://github.com/LeDDGroup/typescript-transform-macros