fix(ts/js) use identifier to match potential keywords

[joshgoebel]

• fix(typescript) use identifier regex to match potential keywords
• fix(javascript) use identifier regex to match potential keywords

The treats expressions like $class as a whole identifier and therefore
avoids the class portion of that identifier incorretly being highlighted
as a keyword.

Resolve #2516.

[joshgoebel]

This adds $lexemes:

  var KEYWORDS = {
     $lexemes: ECMAScript.IDENT_RE,
     keyword: ECMAScript.KEYWORDS.join(" "),
     literal: ECMAScript.LITERALS.join(" "),
     built_in: ECMAScript.BUILT_INS.join(" ")

I can document this syntax, but I think this makes a lot of sense as the whole purpose of lexemes is to process keywords - it's not used for anything else. And many grammars use a KEYWORD constant that they embed over and over in many places... making this a very useful thing to have to avoid repeating yourself.

I would suggest either:

• mode level lexemes (if present) would supersede keywords/$lexemes
• having both is a hard error and the language does not compile (ie, grammar designers would not be allowed to do this when designing grammars)
• Although perhaps no special behavior is needed here... we could let $lexemes always win and it can always be destroyed with inherit if they REALLY need an exception case for some deep mode - the same as you would do for any other key.

Long term I think perhaps we should deprecate MODE.lexemes in favor or MODE.keywords.$lexemes.

[joshgoebel]

Also the name "lexemes" drives me nuts and we could use this as an opportunity to start renaming it... $pattern?

[allejo]

• mode level lexemes (if present) would supersede keywords/$lexemes

Mode level lexemes currently exist and are potentially in use right now. Wouldn't we want keywords/$lexemes to supersede that? i.e. we're introducing new behavior and the old behavior will potentially be deprecated, so shouldn't that new behavior be more important?

• having both is a hard error and the language does not compile (ie, grammar designers would not be allowed to do this when designing grammars)

Do you know if there's a use case of when anyone would need to use both? If not, then I would think that discouraging this behavior would be ideal so a hard error.

Unless there's a use case, this has my vote 👍

• Although perhaps no special behavior is needed here... we could let $lexemes always win and it can always be destroyed with inherit if they REALLY need an exception case for some deep mode - the same as you would do for any other key.

If supporting both methods at the same time is important, then I think this idea is the same as my thoughts on the first option.

---

Long term I think perhaps we should deprecate MODE.lexemes in favor or MODE.keywords.$lexemes.

If my understanding is correct, lexemes are currently defined per mode (as a sibling to keywords). However, a common practice in writing grammars is that keywords is a single object/string and the lexemes for said keywords are outside that object, correct? If so, I'm in favor of deprecating MODE.lexemes.

---

Also the name "lexemes" drives me nuts and we could use this as an opportunity to start renaming it... $pattern?

👍 for $pattern. What this feature is defining is the pattern for a lexeme or token, not the lexemes themselves.

[egor-rogov]

This adds $lexemes

It looks reasonable, but given we already have one method (`MODE.lexemes)), do we really need another one? I'm not strongly against it, just in a doubt.

Long term I think perhaps we should deprecate MODE.lexemes in favor or MODE.keywords.$lexemes.

We'll have troubles doing so, because a lot of grammars use it.

Also the name "lexemes" drives me nuts and we could use this as an opportunity to start renaming it... $pattern?

I agree, "lexemes" always seemed strange to me. keywords.$pattern looks better. But why do you want to use this dollar?

[joshgoebel]

keywords.$pattern looks better. But why do you want to use this dollar?

Because it's a reserved namespace. Someone might be writing a grammar for a pattern matching library and want to have a "pattern" type of keyword and have "pattern" as the className... so we have to have some type of "reserved" namespace if we are going to allow configuration options right next to class names.

but given we already have one method (`MODE.lexemes)), do we really need another one? I'm not strongly against it, just in a doubt.

I was cutting and pasting the code 4-5 times for Javascript and then about to have to do it gain for TypeScript - and this is literally why we program, to eliminate this type of manual cutting and pasting - and I'm about ready for grammars to get a bit easier to write. Plus cutting and pasting is error prone, and it's too easy to miss a spot, etc...

This exact "cut and paste" problem is the ONLY reason that's stopped me from fixing the __keywords issue with C++, it's just so yucky a solution, and C++ is even worse (number of times you'd need to copy/paste).

Do you know if there's a use case of when anyone would need to use both? If not, then I would think that discouraging this behavior would be ideal so a hard error.

We could always discourage it UNTIL someone finds a use case. :-)

---

The other idea here would be some type of inheritance setting for keywords/lexemes... but I'm not sure what that would look like. You'd want it at a single level (top) to prevent hunt and find and cut and paste over and over...

keywords: { ... },
lexemes: /[a-z]+/,
inheritKeywords: true,
inheritLexemes: true,
// or
keywords: { 
  $inherit: true,
  $lexemes: /[a-z_]+/,
  keywords: " ... ",
  built_ins: " ... "
}

I'm already liking this idea of keeping the configuration with the keywords themselves though, as honestly it seems the most natural place to me.

So once inherit was turned on the keywords would be inherited down... and then (say inside a string or comment) you'd have to say keywords: null to stop the inheritance chain.

My worry is there we might start getting people accidentally highlighting keywords inside strings/comments just because they fail to turn off inheritance... but I'm not sure that's much worse than the current behavior of not highlighting keywords at all because people forgot to do the keyword inheritance manually?

I've kind of felt for a while that keywords should perhaps even "auto-inherit"... and you've have to manually intervene to disable that behavior...

---

And the more I think about it we don't need both at all:

lexemes: /[a-z+]/,
keywords: {
  $pattern: /[a-z]+/

Should throw a hard error ("Don't use both lexemes and keywords.$pattern in the same block.").

[joshgoebel]

We'll have troubles doing so, because a lot of grammars use it.

We'd deprecate it like you do anything... with a major version, first to deprecate it - add warnings... and then in the next major version to turn it off completely. But I'm fine with it living for a long while if need be...

If the new location becomes the blessed one it's one line of code to just copy lexemes into keywords.pattern when compiling modes... and I think we can maintain that one line of code many years into the future if we MUST. :-)

[egor-rogov]

Okay, I'm fine with a hard error.
So we agreed on $pattern, I think? And it should be documented, of course.

[joshgoebel]

No worries I will document it. :)

[joshgoebel]

How is that?

[joshgoebel]

Better?

[egor-rogov]

Oh yeah, that's awesome.