RFC: Allow full unicode range

[leebyron]

This spec text implements #687 (full context and details there) and also introduces a new escape sequence.

Three distinct changes:

1. Change SourceCharacter to allow points above 0xFFFF, now to 0x10FFFF.
2. Allow surrogate pairs within StringValue. This handles illegal pairs with a parse error.
3. Introduce new syntax for full range code point EscapedUnicode. This syntax (\u{1F37A}) has been adopted by many other languages and I propose GraphQL adopt it as well.

(As a bonus, this removes the last instance of a regex in the lexer grammar!)

Reference implementation graphql/graphql-js#3117

Closes #687
Closes #214

[andimarek]

One thing I am not sure yet: should a single surrogate pair be really an error.

Wikipedia says:

It is possible to unambiguously encode an unpaired surrogate (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this then when translating between encodings.[citation needed]

(But without clear citation as you can see)

Looking at the Java and JS internal encoder it seems that they don't throw errors which is consistent with the wiki page.

So I am asking if we are trying to be to strict here. As a side effect: not handling single surrogate code points as errors would also simplify implementations.

[leebyron]

I'll be honest that this is on the edge of my expertise with Unicode. I did a ton of reading while working on this and stupidly did not take any notes. I'll try to retrace some steps.

Here's an excerpt from the Unicode FAQ:

Q: How do I convert a UTF-16 surrogate pair such as to UTF-8? As one 4-byte sequence or as two separate 3-byte sequences?

A: The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single 4-byte sequence. However, there is a widespread practice of generating pairs of 3-byte sequences in older software, especially software which pre-dates the introduction of UTF-16 or that is interoperating with UTF-16 environments under particular constraints. Such an encoding is not conformant to UTF-8 as defined. See UTR #26: Compatability Encoding Scheme for UTF-16: 8-bit (CESU) for a formal description of such a non-UTF-8 data format. When using CESU-8, great care must be taken that data is not accidentally treated as if it was UTF-8, due to the similarity of the formats. [AF]

Q: How do I convert an unpaired UTF-16 surrogate to UTF-8?

A different issue arises if an unpaired surrogate is encountered when converting ill-formed UTF-16 data. By representing such an unpaired surrogate on its own as a 3-byte sequence, the resulting UTF-8 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in a valid data stream. Therefore a converter must treat this as an error. [AF]

That is about UTF16 and UTF8 conversions, which is not exactly the domain we are working in the spec. GraphQL does not have an official encoding, it stays at the abstraction of a "sequence of code points"

I suppose if your implementation relied on UTF8 under the hood then you would expect the behavior described here, only an error would be appropriate when interpreting the string value. If your implementation relied on "older software" UTF16, as both Java and JavaScript do, then you might just expect them to pass through without error, since that's what the host environment does.

---

I think either way we need to solve for:

Given a string which includes a surrogate pair (directly or escaped), we need to describe the string value of code points in a well defined way.

So we will have to have this bit of complexity one way or another. Our choice is then what to do with unpaired surrogates:

1. Treat them as an error (current)
2. Pass them through as code points (potentially invalid for some systems?)
3. Leave it to the implementation (ie. "A service may raise an error or pass them through")

I'm not sure which of the latter two you'd suggest, though I think we may need to call in for more Unicode brain power 🧠

[andimarek]

I have two open questions, see inline comments.

[andimarek]

I don't have enough background to judge that. How often and in which areas is that used these days that we need to consider that?

[leebyron]

UTF-8 doesn't need a BOM, so since that's become the dominant encoding this comment is a little anachronistic (BOM is common in other encodings like UTF-16), but I think still important for completeness.

This same clause appears in the JavaScript spec for similar reasons.

[leebyron]

I should point out that this is an editorial change. The current spec has this same clause, it was just in an awkward location before we had a space to discuss Unicode support directly.

[andimarek]

Yeah ... GraphQL Java actually already handles that :-)

[andimarek]

This is the one change I can see so far which I am not sure about.
I can see how you want to escape control character in a String, because you really want to be able to express ALL unicode scalar values.
But why unescaped in a document itself? In 6 years GraphQL Java nobody ever wanted to that afaik.

[leebyron]

I think this earlier change to exclude control characters was well intentioned but ultimately more costly than helpful.

In practice I agree with your point, but there are corner cases which have caused real confusion.

First, it only excluded the ascii-range control chars, and not the control chars > 0x7F, so while the intent was to exclude control chars, in practice there were others allowed.

But I think more importantly, the real cost is being different from other languages as a source of complexity and confusion. No other modern language has this kind of source level exclusion, and this ends up being a land-mine for dynamic queries with string concatenation which need to then remember to do additional input sanitation.

Ultimately this previous attempt to exclude control chars was just a source of implementation complexity of dubious value. My intent with the change is to make Unicode source chars no different from nearly all other modern languages.

[andimarek]

Thanks for explaining. I am happy with it.

[andimarek]

It is getting a bit meta but I don't think character is completely correct here. It is either a sequence of Unicode Scalar Values or a sequence of Code Points depending on how you look at it.

[netlify]

✅ Deploy Preview for graphql-spec-draft ready!

Name	Link
🔨 Latest commit	6871305
🔍 Latest deploy log	https://app.netlify.com/sites/graphql-spec-draft/deploys/62999a0184aff8000826c763
😎 Deploy Preview	https://deploy-preview-849--graphql-spec-draft.netlify.app/draft
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.