\p{Lc} fails to compile, but should be equal to \p{Cased_Letter}

[Aloso]

What version of regex are you using?

1.7.1

Describe the bug at a high level.

The syntax of regex seems to follow the ECMAScript spec. Upon checking, I noticed that 3 General Categories are not supported that work in other regex engines:

• Surrogate
• Cs (abbreviation for Surrogate)
• Sc (abbreviation for Currency_Symbol)

But while the lack of support for Surrogate may be intentional, \p{Sc} should work, and does work when not abbreviated. It also works when prefixed with gc= or General_Category=:

Regex	compiles?
\p{Currency_Symbol}	yes
\p{Sc}	no
\p{gc=Sc}	yes
\p{GC=Sc}	yes
\p{General_Category=Sc}	yes

This bug might be related to the Script/sc property (e.g. \p{sc=Greek}), which happens to have the same abbreviation as the Currency_Symbol General Category.

I noticed that Unicode properties are case insensitive in regex, which is in violation of the ECMAScript spec. But since changing it would be a breaking change, it's probably not worth it.

What are the steps to reproduce the behavior?

Try to compile \p{Sc}

What is the actual behavior?

error: regex parse error:
    \p{Sc}
    ^^^^^^
error: Unicode property not found

What is the expected behavior?

It should work.

[BurntSushi]

Thanks for the report. This is a duplicate of #835, although that's only about the Sc abbreviation. I have it fixed in some ongoing work that I hope to get merge at some point soon.

With respect to Surrogate, it is excluded explicitly. The regex crate only supports Unicode scalar values, which correspond precisely to the subset of all codepoints that exclude the surrogate codepoints. The reason for doing that is that surrogate codepoints are unrepresentable in UTF-8, and this crate only supports UTF-8. Since regex compilation converts Unicode classes into UTF-8 automata and since surrogate codepoints cannot be encoded in UTF-8, there is no way to proceed there. Either classes that contain surrogates should fail to compile, or they would otherwise have to be silently removed. The regex crate chooses the former behavior.

Popping up a level, what are you hoping to achieve with the \p{Surrogate} class?

The syntax of regex seems to follow the ECMAScript spec.

Could you please say what you gave that impression? I'd love to fix it, because it is most certainly not true.

I noticed that Unicode properties are case insensitive in regex, which is in violation of the ECMAScript spec. But since changing it would be a breaking change, it's probably not worth it.

Again, this is intentional and it is even explicitly documented in the Unicode support documentation:

In all cases, property name and value abbreviations are supported, and all names/values are matched loosely without regard for case, whitespace or underscores.

Indeed, it is actually ECMAScript that does not follow UTS #18.

[Aloso]

With respect to Surrogate, it is excluded explicitly

That is what I assumed, and it makes perfect sense since a surrogate is not a Unicode scalar value. I just noticed it because I iterated over all General Categories to see which are supported by regex.

Popping up a level, what are you hoping to achieve with the \p{Surrogate} class?

Nothing. I just needed to know which properties are supported, so I can emit an error in Pomsky when an unsupported property is used. Unicode support in Pomsky is slightly broken at the moment, which I intend to fix in the next release.

The syntax of regex seems to follow the ECMAScript spec.

Could you please say what you gave that impression?

The syntax just looks similar, but upon second thought there aren't that many similarities. I think what animated this statement was my realization that Rust and ECMAScript support almost the same set of Unicode properties and use the same syntax to match them.

[BurntSushi]

The syntax just looks similar, but upon second thought there aren't that many similarities.

Ah yeah, most regex flavors are quite similar, but can diverge in interesting and surprising ways. Some are definitely more similar than others.

I think what animated this statement was my realization that Rust and ECMAScript support almost the same set of Unicode properties and use the same syntax to match them.

Yeah, the key there is the common inspiration: UTS#18.

[BurntSushi]

I'm going to close this because I think the Surrogate question is answered, and the Sc bug is tracked by another ticket and should be resolved soon.

[Aloso]

@BurntSushi I found another one: \p{LC} doesn't work, but \p{gc=LC} does. Not sure how I missed this before. LC (short for Cased_Letter) combines Uppercase_Letter, Lowercase_Letter and Titlecase_Letter.

[BurntSushi]

Nice find! I have a fix incoming for that now too. The main issue is that lc is an abbreviation for both Cased_Letter and Lowercase_Mapping. Since we don't support the latter (but it is in the regex's Unicode table of property name aliases), I added a manual case for it to be mapped to Cased_Letter. (Just like I've done for cf and sc.)