Disallow surrogate halves in escape sequences of string and character literals

[HertzDevil]

See #10440.

The lexer independently ensures the escaped character is a valid UTF-8 codepoint instead of relying on the exception raised by Int#chr (in other words the lexer could probably use #unsafe_chr here). Int#chr will belong in a different PR.

[sdogruyol]

Thanks @HertzDevil 🙏

[bcardiff]

So, this breaks markd.

• https://github.com/icyleaf/markd/blob/master/src/markd/html_entities.cr#L61
• https://github.com/icyleaf/markd/blob/master/src/markd/mappings/entities.cr#L18-L19

Those seem to be valid use cases to me.

Should we revert this PR?

[straight-shoota]

The use case is probably fine. But I question if unicode escape sequences should support invalid unicode characters. This change seems like a good restriction.

Now, regarding the use case: Strings and string literals can be invalid unicode (for now at least, see #2886). The \x escape sequence allows using any byte value in a string literal. So I think the best solution would be to rewrite the literals without unicode escape sequences:

"[\xed\xa0\x80-\xed\xaf\xbf][\xed\xb0\x80-\xed\xbf\xbf]" # equivalent to "[\uD800-\uDBFF][\uDC00-\uDFFF]"
"\xed\xa0\xb5\xed\xb4\x84" # equivalent to "\uD835\uDD04" (Afr)
"\xed\xa0\xb5\xed\xb4\x9e" # equivalent to "\uD835\uDD1E" (afr)

ping @HertzDevil @icyleaf

[straight-shoota]

This is how to fix string literals with surrogate halves after this change: Byte escape sequences need to replace unicode escape sequences. String#dump takes care of that, it automatically presents invalid unicode codepoints as byte escape sequences.

"[\uD800-\uDBFF][\uDC00-\uDFFF]".dump # => "[\xED\xA0\x80-\xED\xAF\xBF][\xED\xB0\x80-\xED\xBF\xBF]"