Get ideas from other projects

[ishitatsuyuki]

There are several other Rust projects (some launched recently) that deals with JavaScript sources. Let's see if we can get some ideas from their design.

https://github.com/nathan/pax by @nathan
https://github.com/swc-project/swc by @kdy1
https://github.com/FreeMasen/RESS by @FreeMasen

On this thread, let's focus on differences or ideas on:

• Parser structure
• Crate separation/modularization
• Project goals: beyond parsers, what should we focus on? Transpiling, minification/optimization, and bundling is what I'm aware of currently.

Library authors: are you interested in joining force with Ratel? Are there anything on library design that we should change? (Pardon this friendly ping, sorry if you get annoyed by a notification. Feel free to unsubscribe in any case.)

[maciejhirsz]

There is also https://github.com/dherman/esprit by @dhermann

I'll try to set up some benchmarks across the Rust-based parsers to see how Ratel fares. If anything, we can at least be the testbed for performance.

[FreeMasen]

Thanks for reaching out! I am not sure I am willing to abandon my crates at this point but I think communicating is a key to success.

One of the big reasons I started down the rust+js path was because I was looking for some modular crates to pick and choose what part of the process I was trying to work through. For example, if I wanted to transpile from rust to js it would be a lot easier if I could just transform a syn::TokenStream into a ratel::ast and then pass that off to ratel-codegen. This solution wouldn't need any other parts of ratel.

Modularity being the key here, by having a reader(parser) and a writer(codegen) built around a solid AST you empower the community to build things you might never think of. If someone wanted to write a codegen library that read in ratel::ast and output cranelift IR, no one involved with ratel would need to be invested in that project. I see that as a huge win for you and the community.

Another thing that I have been getting questions about/thinking about is the ability to use your parser to on supersets of js, primarily JSX and typescript.

Looking over your crates here (not super thoroughly) I don't see any kind of comment handling mechanism. A number of js dev-tools utilize comments so including a way to communicate them might be valuable.

Thanks again for reaching out!

[maciejhirsz]

Aye, modularization is definitely somewhere Ratel has to improve. I've already tried some stuff in the token_maps branch, but I've abandonned that for now and am now focusing on Logos, once that's in place I think I can make a proc_macro for creating Lookup Tables for tokens that should help writing different specializations for Ratel.

In token_maps I was exploring the possibility of doing a runtime generated tables for feature sets that can be either generated dynamically or loaded from lazy_static and passed to the parser by reference. Right now I'm thinking it might be more interesting to make the entire AST completely generic in the following fashion:

trait Module<'ast>: Parse<'ast> {
    type Statement: Statement<'ast>;
}

trait Statement<'ast>: Parse<'ast> {
    type LoopStatement: Statement<'ast>; // For specialization that involves `break` and `continue`
    type Expression: Expression<'ast>;
}

trait Expression<'ast>: Parse<'ast> {
    type AsyncExpression: Expression<'ast>; // For specialization within async function bodies that permits `await` expressions
}

It might be a terrible idea and going the dynamic route might be easier (even if it means that the types permit constructing potentially invalid grammar), but I'm curious what other people think.

[kdy1]

Author of swc here. Can we join our efforts?
Project's goal is almost identical.

I've implemented almost all transcompilers in swc, but it's quite slow. So I want to make it fast, or merge it into ratel.
(Mainly because I need arena allocator, and changing ast to have lifetime seems harder than merging swc into ratel).

[maciejhirsz]

@kdy1 I'd be more than happy to join, I also don't particularly care about branding so the merge can be either way.

I'd say have a look at the AST on master and let us know if there is anything there that you are lacking (we do miss comments #103, but I don't know if swc is actively using comments in any way?) On my end, I'm happy to offload all visitor / transformer stuff and just focus on the parser.

[kdy1]

Currently ratel's span does not contain hygiene information. swc use span hygiene to implement block scoping. hygiene based variable mangement makes implementing other passes simple. They can just declare a variable named ref and hygiene pass will rename it. Otherwise each pass should check if there's a variable named ref.

Also note that swc uses rustc's libsyntax internally which is quite heavy, so I think right way to go is merging swc into ratel and droping the dependency.

[maciejhirsz]

I'm hoping to look into adding hygiene to the AST today. My idea would be to create a new ratel-ast crate in the workspace with any changes swc might need, and then have lexer and parser follow suit (lexer will get a full refactor anyway).

[kdy1]

Does ratel targets stable rust?
If using nightly rust is ok, I want to integrate swc's specialization based visitor into ratel.

It's (currently) done by putting derive(Fold) on ast nodes and it allows doing things like https://github.com/swc-project/swc/blob/222bdc191fcab77143197fb0da455c9905d1f983/ecmascript/transforms/src/compat/es2015/arrow/mod.rs#L118-L142

Also note that rustc is quite good at optimizing the visitor.

[ishitatsuyuki]

I'm not sure specialization based visitor is a good idea. Specialization is a very hot topic currently, and whether and how it would be stabilized is uncertain.

I'd prefer the traditional approach utilizing hardcoded trait methods. If you are concerned about optimization, would you mind benchmarking both specialization based method and the one without?

[kdy1]

I'll just go with traditional one.

I thought there's no way to visit ArrowBody, but rustdoc says that it implements Visitable, so I were wrong.