Provide Javascript port of TokenStreamRewriter

[eirikur-grid]

What

Add a JavaScript port of the TokenStreamRewriter that includes most, but not all, of the methods from the Java implementation.

Why

So that JavaScript developers can benefit from having it to implement all sorts of neat things (refactorings, translation etc.)

Scope

Included

• insertAfter
• insertBefore
• replace
• delete
• getText
• getProgram
• getTokenStream

Omitted

The following methods were omitted because
a) I couldn't find any tests for them and
b) their utility is not immediately obvious to me.

• rollback
• deleteProgram
• getLastRewriteTokenIndex

If someone needs them then we can obviously add support for them in a subsequent PR.

How

By

• Porting most of the TokenStreamRewriter class
• Porting tests and supporting fixtures (there are a couple of toy grammars used by the tests)
• Adding more tests to get close to 100% coverage

Credit to @sharwell for having ported the TokenStreamRewriter and the corresponding unit tests to TypeScript (see https://github.com/tunnelvisionlabs/antlr4ts). That has made creating a JavaScript port relatively straight-forward.

Drive-bys

• For added convenience, I added a Makefile within the spec folder to automate the code generation from the toy grammars.
• I added a dev dependency for c8 and a run target in package.json that generates a code coverage report. This I used to find uncovered sections in the TokenStreamRewriter.
• I added an explicit dev dependency on eslint (seems it was transitive before) and upgraded it to the latest version. I then tweaked the parser options in order for it to recognize class static fields. See this SO thread for more context.
• I made a few additions to .gitignore to exclude autogenerated files that don't need to be placed under source control.
• I added an npm run lint target

Closes #1767

[eirikur-grid]

Automatic code formatting kicked in on this file.

[ericvergnaud]

@eirikur-grid sorry if it's taken us so long to look at this
would you be able to rebase from this PR #4027
and provide an associated d.ts file

[eirikur-grid]

This week is going to be quite hectic for me. I'll take a look shortly after Christmas.

[eirikur-grid]

@ericvergnaud I've rebased and added the type definitions file.

[ericvergnaud]

@eirikur-grid we're about to release, and this is a good time to finalize things.
Could you:

• squash your commits and gpg-sign off the squashed commit (this replaces the contributors.txt strategy)
• move your changes in index.js to index.node.js and index.web.js
• add an export in index.d.ts
• move your spec/* files under spec/rewriter/* to clarify their purpose
• not mandatory but a .ts test would also be welcome (to validate the ts api)

If you can do that rapidly enough, this PR will make it into the next release

[eirikur-grid]

@eirikur-grid we're about to release, and this is a good time to finalize things. Could you:

* squash your commits and gpg-sign off the squashed commit (this replaces the contributors.txt strategy)

* move your changes in index.js to index.node.js and index.web.js

* add an export in index.d.ts

* move your spec/* files under spec/rewriter/* to clarify their purpose

* not mandatory but a .ts test would also be welcome (to validate the ts api)

If you can do that rapidly enough, this PR will make it into the next release

You caught me at a very busy time, hence the late reply. I should be able to get around to this before the end of this week.

[ericvergnaud]

HeyToo bad, version 4.12 just went out Please rebase and we’ll ship this in the next releaseEricEnvoyé de mon iPhoneLe 19 févr. 2023 à 23:13, Eiríkur Fannar Torfason ***@***.***> a écrit : @eirikur-grid we're about to release, and this is a good time to finalize things. Could you: * squash your commits and gpg-sign off the squashed commit (this replaces the contributors.txt strategy) * move your changes in index.js to index.node.js and index.web.js * add an export in index.d.ts * move your spec/* files under spec/rewriter/* to clarify their purpose * not mandatory but a .ts test would also be welcome (to validate the ts api) If you can do that rapidly enough, this PR will make it into the next release You caught me at a very busy time, hence the late reply. I should be able to get around to this before the end of this week. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[eirikur-grid]

@ericvergnaud I believe I've addressed all remarks.

[ericvergnaud]

I think we have a small problem here
The purpose if the CI is not to test against the latest official version, rather against the version being tested.
So the makefile should use the tool from the local maven repo, not from the antlr web site.

[eirikur-grid]

OK, how do I do that? Are there any examples within the repo that I can follow? I can see that the Java tests are capable of generating lexers/parsers from a grammar at run-time but what about other languages?

When I started working on this, I found Python tests using generated code (runtime/Python2/tests/mocks/TestLexer.py) that has been commited to the repo, so I assumed this was OK.

[eirikur-grid]

I think my contribution ends here. Over a year ago, I found myself needing a JavaScript version of the TokenStreamRewriter. That is no longer the case, so I have no vested interest in putting more effort into this.

[ericvergnaud]

All generic grammar tests generate parsers at CI time, but they are run from java.

Every GitHub action runs the "Build ANTLR with Maven" step, which builds the tool and registers it in the local maven repo, which should be found at ~/m2/repository/org/antlr/antlr4/${version}/antlr4-${version}-complete.jar.

You are correct in observing that many target-specific tests do not follow the expected policy, but let's not make things worse :-)

[ericvergnaud]

Sorry to hear that, and apologies for taking so long to consider this PR. I hope your interest will grow again soon. This is useful work for the community.

[ericvergnaud]

this needs to be regenerated during tests or it will fail with a version mismatch

[ericvergnaud]

same here

[ericvergnaud]

@parrt ready to merge, I suggest we deal with the versioning issue at a global level since it affects all targets

[parrt]

Re "derived from antlr4ts", license appears to be "The ANTLR Project" so we're good there: https://github.com/tunnelvisionlabs/antlr4ts/blob/master/LICENSE