Sequence eDSL replacement

[goetzrrGit]

Current Challenge:

Aerie currently utilizes a TypeScript eDSL for sequencing, which has proven to be difficult to maintain. It requires a considerable level of TypeScript knowledge, and debugging is particularly challenging due to the lack of straightforward debugging tools for the eDSL. Additionally, the process of generating SeqJSON or importing SeqJSON back into the eDSL requires server interactions.

Proposed Solution:

The proposed DSL offers a streamlined approach to code maintenance. We're leveraging CodeMirror (editor) and Lezer to create a highly customizable DSL grammar that seamlessly integrates with our Svelte web application.

This enables:

• SeqJSON generation on the fly with no server interaction
• Highly customizable grammar and DSL style
• A smaller codebase with significantly increased code reusability
• The ability to leverage current eDSL code for faster feature implementation

Considerations:

However, adopting this approach comes with its own set of considerations:

• Loss of type checking, which necessitates the development of a robust linter for user typing validation
• Development of a custom content assist, as opposed to an out-of-the-box solution like Monaco provides
• Loss of SeqJSON validation, prompting the need for a dedicated SeqJSON validator
• Lezer grammar syntax may require some adjustment, with documentation availability improving but still posing a learning curve for some users

Trade-offs:

Investing in time and resources to build a new tool versus continuing with the fully implemented eDSL
Balancing the simplicity of the proposed DSL with the potential maintenance overhead of maintaining two sequencing DSLs
Ripping out the current eDSL to support a small subset of the spec, resulting in a significantly smaller commandlib file, while potentially losing the ability for users to utilize the commandLib library outside of AERIE in their own IDE, impacting template and code reusability.

UPDATE 10/31/23

Overall

Both DSLs and eDSLs have their own advantages and disadvantages. DSLs are more modular and reusable, easier to maintain, and can add checks without a TypeScript custom worker. However, they require custom context assist implementation and the Lex grammar can be complex to set up. eDSLs have out-of-the-box context assist and a more powerful TypeScript custom worker, but they are more complex to maintain and less flexible than Lex grammar.

Feature	DSL	eDSL
Lex grammar	More difficult to learn, but more powerful to define a custom format and style	Easier to learn, but harder to define a custom language
Modularity	More modular and reusable code to/from seqJson	Less modular and reusable code, with more duplicate code
Complexity	Less complex, easier to maintain	More complex, more difficult to maintain
Context assist	Requires custom implementation	Context assist out of the box
Type checking	Limited type checking capabilities, but easier to implement	More powerful type checking capabilities, but complex to implement
Additional features	Limited additional features	Supports more advanced features such as auto-complete, formatting, and time conversions

Recommendation:

I recommend that we stay with the eDSL for sequence creation. Several other parts of AERIE are already in TypeScript and have their own eDSLs. Maintaining both an eDSL and a DSL would be a burden, as we would need to support two different languages. It is true that the DSL is easier to work with, but we would not be saving any overhead, as we are still using the eDSL for sequence expansion.

Instead, we should focus on making the eDSL simpler and more reusable. We could also spend a sprint refactoring the code to make it more maintainable. Other parts of the system, such as Constraints and Goals, are just as complex as the sequencing system, and they have a steep learning curve due to their use of Java V8 and TypeScript eDSL on the Merlin server.

[ewferg]

Nice write up @goetzrrGit! Can you provide a little more detail on the commandLib library and its purpose and why users may want to use it in their own IDE?

[dyst5422]

@ewferg The commandLib library is the typescript library generated from a command dictionary that furnishes the eDSL with all of the command information for a mission. A user could use it in their own IDE to get all of the type information needed for authoring/checking/validating a sequence/fragment typescript file as it is done both on the server and in the Aerie UI.

[dyst5422]

I haven't seen what kind of grammer/structures you are proposing for the DSL, so take the following input with that in mind.

The value in using an eDSL as compared to a DSL is primarily in having the power of a full programming language to programmatically create and reuse fragments of sequences.

Sequencing gets more and more complicated as a mission progresses, so being able to reuse sequencing fragments and isolate complexity becomes very important.

I'm going to suggest you look into how masters and submasters are generated on M2020 or MSL as a view into what this can look like for a mature mission.

Sure, for simple templated sequences, perhaps a DSL is simpler, but it's insufficient for the more complex cases and, as you stated, maintaining two completely different ways to create sequences is more maintenance cost.

So the driving factor for the eDSL in the first place was something that was going to be powerful and flexible enough to allow for the most complicated cases while providing the declarative simplicity desired for the simpler cases.

[goetzrrGit]

Sorry for not respondingWe've put this on the backburner for a bit, but I wanted to share with you the proposed DSL that Chris wrote. I took a look at it to assess its feasibility.

https://github.com/NASA-AMMOS/aerie-sequence-editor

For context, this DSL is intended to replace the standalone editor for sequence creation, but not the expansion editor. The expansion editor will continue to use a TypeScript eDSL, which I don't believe this DSL can't handle. I am going to add more to my finding and @joswig is going to decide whether we should move in this direction.

[dyst5422]

Conversions between SeqJSON and eDSL shouldn't require server interactions, just the part of the commandLib that contains the logic for those conversions, which should be executable in the browser with a small bit of work.

[goetzrrGit]

This is true, as of right now we send it off to the server to generate the json, but the UI can invoke the toSeqJSON method on the fly.