set of strings: longest match

[markusicu]

In the TC39 meeting today (2021-may-26) there was some discussion of how to match character classes that contain multi-character strings, inspired by the slide that showed the examples

• [\p{RGI_Emoji}--(🇧🇪)]
• [a-zA-Z(ch)(m̀)(か゚)(🇦🇺)(🇧🇪)(🇫🇷)] ≍ [a-zA-Z(ch|m̀|か゚|🇦🇺|🇧🇪|🇫🇷)]

The proposal is to match longest strings first, so that a prefix string does not pre-empt matching a longer string. This needs to be done in runtime semantics after evaluating a set of strings (as a modified CharSet, or as a StringSet, whichever that goes).

In particular, we do not want to match strings in the order that they are written in the regular expression.

Reasons:

1. A character class defines a set of characters/strings in the mathematical sense: no order, no duplicate elements
2. The regex spec and its proposed changes are written in terms of set operations.
3. Source order would be confusing in a character class with unions, intersections, subtractions, and nested classes.
4. A Unicode property defines a set of characters/strings in the mathematical sense; in particular, no order. Thus, there is no order of the strings in [\p{RGI_Emoji}--(🇧🇪)] that we could preserve.
5. Implementation experience: ICU class UnicodeSet has supported string literals (though not properties of strings) since 2002. Unicode CLDR has used UnicodeSet syntax nearly that long (e.g., in exemplar character sets and transform/transliteration rules). There has been no discussion or confusion about these being sets in the mathematical sense.

As for the longest match specifically, note that users may have no idea how many Unicode code points it takes to write a “character” like m̀, か゚, 👧🏿, or 🇧🇪 — they just want it to “work”. (I even had a discussion this week with a Slovak colleague who expected there to exist a single-code point way to write "ch".)

[mathiasbynens]

cc @msaboff and @waldemarhorwat

[mathiasbynens]

As discussed in yesterday’s meeting, I’ve kicked off a PR to add this to the FAQ, pointing here for the more detailed rationale that @markusicu has posted. Once the PR lands, I’ll close this issue.

[mathiasbynens]

https://github.com/tc39/proposal-regexp-set-notation#whats-the-match-order-for-character-classes-containing-strings