-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070
Comments
On macOS, I generated a 13M of $ node -e "fs.writeFileSync('/tmp/response.json', JSON.stringify({customers: [...Array(1000000).keys()].map(x=>({id: x}))}))" and ran your code sample (adjusting $ ~/.dotnet5/dotnet --version
5.0.100-rc.2.20459.18
$ ~/.dotnet5/dotnet run
Reading Json in memory - deserialized 1000000 customer records.
Reading Json from stream - deserialized 1000000 customer records. |
@am11 - thanks - I have checked and I can reproduce on MacOS with RC2. The error is consistently replicable, but is sensitive to the data. I spent some time trying to find an example of data that does not include private information and reproduces the error, but could not find anything else that does it.
|
It's possible this is fixed by #42089. |
With 125M of response.json with 1M elements in |
Here is a repro with fake data: https://github.com/frankbuckley/net5-json-repro Running gives results similar to:
To do this reliably, the |
This happens, when the buffered data ends exactly while reading a null token. 237 if (!isNullToken || !IgnoreDefaultValuesOnRead || !Converter.CanBeNull) Will try to fix it without regression or at least add a simple repro. |
* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx
Re-opening for consideration in .NET 5. |
* Repro dotnet#42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from dotnet#42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx
* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx Co-authored-by: devsko <devsko@users.noreply.github.com>
Fixed for .NET 5 in #42359. |
* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * refactoring * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx * Using test data from SR * Generalize continuation tests for payloads of any length Tweak the payload and expect `JsonException` * merge * Test deserialize with Utf8JsonReader and ReadOnlySequence * Again with value typed nested object * Add tests for splitted whitespaces * Addressed feedback Added dictionary test * Validate line and position of failure in tweaked payloads more tweaks * Fixed comment Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com>
While evaluating .NET 5.0 (Preview 8), I have encountered a problem with the
JsonSerializer
deserializing a particular payload, which works fine with .NET Core 3.1.Description
Specifically, the payload is the response from a call to the Shopify Customer Search API. If we call this with a limit of 200 results, it returns a ~330KB response body and an exception is thrown. If we call this same API with a limit of 100 results, deserialization of the response completes successfully.
I have managed to reproduce the error, reading the response data from a file. It appears that the error only occurs if
JsonSerializerOptions.IgnoreNullValues = true
and deserializing(async) from a stream. If the Json is read into memory andJsonSerializer.Deserialize
is called, it works fine.I have tried reproducing the error with another large payload without success, so it seems to be something specific to this Json data (and this configuration). Unfortunately, it includes personal data so I cannot share it publicly, but I would be happy to find a way to help debug the problem.
Configuration
Regression?
Yes - this works with .NET Core 3.1.
Other information
The exception thrown is:
InnerException stack trace:
Repro (excluding data):
I found a similar issue (#41604) but this error differs in that it is occurring in an ordinary Console app on Windows and also running in Linux (Ubuntu Fedora) containers.
The text was updated successfully, but these errors were encountered: