Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

Closed
frankbuckley opened this issue Sep 10, 2020 · 8 comments · Fixed by #42158

Comments

@frankbuckley
Copy link
Contributor

While evaluating .NET 5.0 (Preview 8), I have encountered a problem with the JsonSerializer deserializing a particular payload, which works fine with .NET Core 3.1.

Description

Specifically, the payload is the response from a call to the Shopify Customer Search API. If we call this with a limit of 200 results, it returns a ~330KB response body and an exception is thrown. If we call this same API with a limit of 100 results, deserialization of the response completes successfully.

I have managed to reproduce the error, reading the response data from a file. It appears that the error only occurs if JsonSerializerOptions.IgnoreNullValues = true and deserializing(async) from a stream. If the Json is read into memory and JsonSerializer.Deserialize is called, it works fine.

I have tried reproducing the error with another large payload without success, so it seems to be something specific to this Json data (and this configuration). Unfortunately, it includes personal data so I cannot share it publicly, but I would be happy to find a way to help debug the problem.

Configuration

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-preview.8.20417.9
 Commit:    fc62663a35

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-preview.8.20417.9\

Host (useful for support):
  Version: 5.0.0-preview.8.20407.11
  Commit:  bf456654f9

.NET SDKs installed:
  3.1.402 [C:\Program Files\dotnet\sdk]
  5.0.100-preview.8.20417.9 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0-preview.8.20414.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0-preview.8.20407.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0-preview.8.20411.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Regression?

Yes - this works with .NET Core 3.1.

Other information

The exception thrown is:

System.Text.Json.JsonException
  HResult=0x80131500
  Message=The JSON value could not be converted to System.String. Path: $.customers[135].default_address.province | LineNumber: 0 | BytePositionInLine: 229465.
  Source=System.Text.Json
  StackTrace:
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.<ReadAsync>d__19`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at JsonDotnet5Repro.Program.<Main>d__4.MoveNext() in C:\dev\frankbuckley\JsonDotnet5Repro\JsonDotnet5Repro\Program.cs:line 43

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
InvalidOperationException: Cannot get the value of a token type 'StartObject' as a string.

InnerException stack trace:

   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.Serialization.Converters.StringConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

Repro (excluding data):

using System;
using System.Collections.Generic;
using System.IO;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;

namespace JsonDotnet5Repro
{
    class Program
    {
        private static readonly JsonSerializerOptions s_serializerOptions = new JsonSerializerOptions
        {
            IgnoreNullValues = true // works if set to false
        };

        static async Task Main(string[] args)
        {
            // Read into memory (always works)

            var json = File.ReadAllText("response.json");
            var result1 = JsonSerializer.Deserialize<CustomerCollectionResponse>(json, s_serializerOptions);

            Console.WriteLine($"Reading Json in memory - deserialized {result1.Customers.Count} customer records.");

            // Read from stream (fails if IgnoreNullValues = true)

            var stream = File.OpenRead("response.json");
            var result2 = await JsonSerializer.DeserializeAsync<CustomerCollectionResponse>(stream, s_serializerOptions);

            Console.WriteLine($"Reading Json from stream - deserialized {result2.Customers.Count} customer records.");
        }
    }

    public class CustomerCollectionResponse
    {
        [JsonPropertyName("customers")]
        public List<Customer>? Customers { get; set; }
    }

    public class Address
    {
        [JsonPropertyName("first_name")]
        public string? FirstName { get; set; }

        [JsonPropertyName("address1")]
        public string? Address1 { get; set; }

        [JsonPropertyName("phone")]
        public string? Phone { get; set; }

        [JsonPropertyName("city")]
        public string? City { get; set; }

        [JsonPropertyName("zip")]
        public string? Zip { get; set; }

        [JsonPropertyName("province")]
        public string? Province { get; set; }

        [JsonPropertyName("country")]
        public string? Country { get; set; }

        [JsonPropertyName("last_name")]
        public string? LastName { get; set; }

        [JsonPropertyName("address2")]
        public string? Address2 { get; set; }

        [JsonPropertyName("company")]
        public string? Company { get; set; }

        [JsonPropertyName("latitude")]
        public float? Latitude { get; set; }

        [JsonPropertyName("longitude")]
        public float? Longitude { get; set; }

        [JsonPropertyName("name")]
        public string? Name { get; set; }

        [JsonPropertyName("country_code")]
        public string? CountryCode { get; set; }

        [JsonPropertyName("province_code")]
        public string? ProvinceCode { get; set; }
    }

    public class Customer
    {
        [JsonPropertyName("id")]
        public long Id { get; set; }

        [JsonPropertyName("email")]
        public string? Email { get; set; }

        [JsonPropertyName("accepts_marketing")]
        public bool AcceptsMarketing { get; set; }

        [JsonPropertyName("created_at")]
        public DateTimeOffset? CreatedAt { get; set; }

        [JsonPropertyName("updated_at")]
        public DateTimeOffset? UpdatedAt { get; set; }

        [JsonPropertyName("first_name")]
        public string? FirstName { get; set; }

        [JsonPropertyName("last_name")]
        public string? LastName { get; set; }

        [JsonPropertyName("orders_count")]
        public int OrdersCount { get; set; }

        [JsonPropertyName("state")]
        public string? State { get; set; }

        [JsonPropertyName("total_spent")]
        public string? TotalSpent { get; set; }

        [JsonPropertyName("last_order_id")]
        public long? LastOrderId { get; set; }

        [JsonPropertyName("note")]
        public string? Note { get; set; }

        [JsonPropertyName("verified_email")]
        public bool VerifiedEmail { get; set; }

        [JsonPropertyName("multipass_identifier")]
        public string? MultipassIdentifier { get; set; }

        [JsonPropertyName("tax_exempt")]
        public bool TaxExempt { get; set; }

        [JsonPropertyName("tags")]
        public string? Tags { get; set; }

        [JsonPropertyName("last_order_name")]
        public string? LastOrderName { get; set; }

        [JsonPropertyName("default_address")]
        public Address DefaultAddress { get; set; }

        [JsonPropertyName("addresses")]
        public IList<Address> Addresses { get; set; }
    }
}

I found a similar issue (#41604) but this error differs in that it is occurring in an ordinary Console app on Windows and also running in Linux (Ubuntu Fedora) containers.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Sep 10, 2020
@am11
Copy link
Member

am11 commented Sep 10, 2020

On macOS, I generated a 13M of /tmp/response.json with:

$ node -e "fs.writeFileSync('/tmp/response.json', JSON.stringify({customers: [...Array(1000000).keys()].map(x=>({id: x}))}))"

and ran your code sample (adjusting response.json as /tmp/response.json) in a console app generated with SDK 5.0.0-rc.2.20458.14 (and host 5.0.100-rc.2.20459.18), it seems to to be working fine:

$ ~/.dotnet5/dotnet --version
5.0.100-rc.2.20459.18

$ ~/.dotnet5/dotnet run
Reading Json in memory - deserialized 1000000 customer records.
Reading Json from stream - deserialized 1000000 customer records.

@frankbuckley
Copy link
Contributor Author

@am11 - thanks - I have checked and I can reproduce on MacOS with RC2. The error is consistently replicable, but is sensitive to the data. I spent some time trying to find an example of data that does not include private information and reproduces the error, but could not find anything else that does it.

PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro> dotnet --info
.NET SDK (reflecting any global.json):
Version:   5.0.100-rc.2.20460.9
Commit:    c8ff60c134

Runtime Environment:
OS Name:     Mac OS X
OS Version:  10.15
OS Platform: Darwin
RID:         osx.10.15-x64
Base Path:   /usr/local/share/dotnet/sdk/5.0.100-rc.2.20460.9/

Host (useful for support):
Version: 5.0.0-rc.2.20458.14
Commit:  482494f9ec

.NET SDKs installed:
3.1.401 [/usr/local/share/dotnet/sdk]
3.1.402 [/usr/local/share/dotnet/sdk]
5.0.100-preview.8.20417.9 [/usr/local/share/dotnet/sdk]
5.0.100-rc.2.20460.9 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-preview.8.20414.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-rc.2.20459.20 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-preview.8.20407.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-rc.2.20458.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download
PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro> dotnet run
Reading Json in memory - deserialized 200 customer records.
Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.customers[135].default_address.province | LineNumber: 0 | BytePositionInLine: 229465.
---> System.InvalidOperationException: Cannot get the value of a token type 'StartObject' as a string.
at System.Text.Json.Utf8JsonReader.GetString()
at System.Text.Json.Serialization.Converters.StringConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
--- End of inner exception stack trace ---
at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
at JsonDotnet5Repro.Program.Main(String[] args) in /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro/Program.cs:line 29
at JsonDotnet5Repro.Program.<Main>(String[] args)
PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro> 

@layomia layomia removed the untriaged New issue has not been triaged by the area owner label Sep 10, 2020
@layomia layomia added this to the 5.0.0 milestone Sep 10, 2020
@tmds
Copy link
Member

tmds commented Sep 11, 2020

It's possible this is fixed by #42089.

@am11
Copy link
Member

am11 commented Sep 11, 2020

With 125M of response.json with 1M elements in customers list, the memory API only reads 100K (result1.Customers.Count), while stream API returns null (no exception, we get nullref from the next line result2.Customers.Count). I am not sure if this is expected behavior from either of the APIs. 🤔

@frankbuckley
Copy link
Contributor Author

Here is a repro with fake data: https://github.com/frankbuckley/net5-json-repro

Running gives results similar to:

PS C:\dev\frankbuckley\net5-json-repro> dotnet run
0000: Reading Json from stream - deserialized 500 customer records.
0001: Reading Json from stream - deserialized 500 customer records.
0002: Reading Json from stream - deserialized 500 customer records.
0003: Reading Json from stream - deserialized 500 customer records.
0004: Reading Json from stream - deserialized 500 customer records.
0005: Reading Json from stream - deserialized 500 customer records.
0006: Reading Json from stream - deserialized 500 customer records.
0007: Reading Json from stream - deserialized 500 customer records.
0008: Reading Json from stream - deserialized 500 customer records.
0009: Reading Json from stream - deserialized 500 customer records.
0010: Reading Json from stream - deserialized 500 customer records.
Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.customers[66].addresses[3].province | LineNumber: 0 | BytePositionInLine: 147609.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'EndObject' as a string.
   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Program.Main(String[] args) in C:\dev\frankbuckley\net5-json-repro\Program.cs:line 47
   at Program.<Main>(String[] args)

To do this reliably, the .RuleFor(a => a.Province, f => f.Address.State().OrNull(f, 0.7f)) seems to be required (at least, I have not yet seen the error without it).

@devsko
Copy link
Contributor

devsko commented Sep 12, 2020

This happens, when the buffered data ends exactly while reading a null token.
When the next block of data is available, the state continuation mechanism stops (after entering the first frame) at this condition in JsonPropertyInfo<T>.ReadJsonAndSetMember() which I think should also include state.IsContinuation

237                if (!isNullToken || !IgnoreDefaultValuesOnRead || !Converter.CanBeNull)

Will try to fix it without regression or at least add a simple repro.

devsko added a commit to devsko/runtime that referenced this issue Sep 12, 2020
devsko added a commit to devsko/runtime that referenced this issue Sep 16, 2020
layomia pushed a commit that referenced this issue Sep 16, 2020
* Repro #42070

* formatting

* namespace

* Fix

* never forget the header

* More tests
- Test continuation at every position inside the tested object
- Many member with primitive and nullable types
- One more level of nested object
- All combinations of class/struct for tested and nested object
- tested and nested object with parametrized ctor for some properties

* Addressed feedback

* Test with original repro data from #42070

* custom converter to ensure the padding is written in front of the tested object

* rename

* test data moved to Strings.resx
@layomia
Copy link
Contributor

layomia commented Sep 16, 2020

Re-opening for consideration in .NET 5.

@layomia layomia reopened this Sep 16, 2020
layomia pushed a commit to layomia/dotnet_runtime that referenced this issue Sep 16, 2020
* Repro dotnet#42070

* formatting

* namespace

* Fix

* never forget the header

* More tests
- Test continuation at every position inside the tested object
- Many member with primitive and nullable types
- One more level of nested object
- All combinations of class/struct for tested and nested object
- tested and nested object with parametrized ctor for some properties

* Addressed feedback

* Test with original repro data from dotnet#42070

* custom converter to ensure the padding is written in front of the tested object

* rename

* test data moved to Strings.resx
layomia added a commit that referenced this issue Sep 18, 2020
* Repro #42070

* formatting

* namespace

* Fix

* never forget the header

* More tests
- Test continuation at every position inside the tested object
- Many member with primitive and nullable types
- One more level of nested object
- All combinations of class/struct for tested and nested object
- tested and nested object with parametrized ctor for some properties

* Addressed feedback

* Test with original repro data from #42070

* custom converter to ensure the padding is written in front of the tested object

* rename

* test data moved to Strings.resx

Co-authored-by: devsko <devsko@users.noreply.github.com>
@layomia
Copy link
Contributor

layomia commented Sep 18, 2020

Fixed for .NET 5 in #42359.

@layomia layomia closed this as completed Sep 18, 2020
layomia added a commit that referenced this issue Nov 2, 2020
* Repro #42070

* formatting

* namespace

* Fix

* never forget the header

* More tests
- Test continuation at every position inside the tested object
- Many member with primitive and nullable types
- One more level of nested object
- All combinations of class/struct for tested and nested object
- tested and nested object with parametrized ctor for some properties

* Addressed feedback

* refactoring

* Test with original repro data from #42070

* custom converter to ensure the padding is written in front of the tested object

* rename

* test data moved to Strings.resx

* Using test data from SR

* Generalize continuation tests for payloads of any length
Tweak the payload and expect `JsonException`

* merge

* Test deserialize with Utf8JsonReader and ReadOnlySequence

* Again with value typed nested object

* Add tests for splitted whitespaces

* Addressed feedback
Added dictionary test

* Validate line and position of failure in tweaked payloads
more tweaks

* Fixed comment

Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com>
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants