Implement JSON token stream deserializer #454
Conversation
| } | ||
|
|
||
| /// Reads a JSON string out of the stream. | ||
| fn read_string(&mut self) -> Result<String, Error> { |
There was a problem hiding this comment.
the XML parsing library I used didn't unescape in the tokenizer. That enabled the tokenizer to be allocation-free (and for input that you know won't need to be escaped (like base64), you can even skip the unescape step later)
I don't think we want to be allocating for every string we read out of the input stream, especially since lots of them (eg. a key in a map) we never need to own
There was a problem hiding this comment.
This makes a lot of sense. Will refactor.
| pub enum Token { | ||
| StartArray, | ||
| EndArray, | ||
| ObjectKey(String), |
There was a problem hiding this comment.
these almost certainly want to be &'a str that refer into the input
|
Looks like the unescaping has a bug around UTF-16 surrogate pairs. Will examine fixing that in addition to decoupling parsing from unescaping. |
|
I will add fuzzing in a follow-up PR. |
This adds a JSON token streaming deserializer to smithy-json for #161.
It passes all tests in JSONTestSuite, except for some notable ones:
There are some test cases around invalid Unicode sequences which are expected to pass but always fail since this implementation is coercing all JSON strings into Rust strings, and Rust wants valid UTF-8. I don't think accepting these invalid sequences would be valuable for our SDK implementation since the protocols call for valid UTF-8 in strings.I was wrong here.[][], which is totally valid for this implementation since it would result instart_array, end_array, start_array, end_array.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.