JSON columns throws Invalid token type: 'StartObject' exception with AsNoTrackingWithIdentityResolution()

[DieBunnyxxx]

I have issues using Json columns.
I'm getting this error via single or split query while trying to query entities with JsonColumns

[ERR] [Microsoft.EntityFrameworkCore.Query] An exception occurred while iterating over the results of a query for context type 
System.InvalidOperationException: Invalid token type: 'StartObject'.
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.IncludeJsonEntityCollection[TIncludingEntity,TIncludedCollectionElement](QueryContext queryContext, Object[] keyPropertyValues, JsonReaderData jsonReaderData, TIncludingEntity entity, Func`4 innerShaper, Action`1 getOrCreateCollectionObject, Action`2 fixup, Boolean trackingQuery)
   at lambda_method14418(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
System.InvalidOperationException: Invalid token type: 'StartObject'.
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.IncludeJsonEntityCollection[TIncludingEntity,TIncludedCollectionElement](QueryContext queryContext, Object[] keyPropertyValues, JsonReaderData jsonReaderData, TIncludingEntity entity, Func`4 innerShaper, Action`1 getOrCreateCollectionObject, Action`2 fixup, Boolean trackingQuery)
   at lambda_method14418(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Error occurs if multiple instances of same object is queried. Works with AsNoTracking() but error with AsNoTrackingWithIdentityResolution()

modelBuilder.Entity<Customer>().OwnsMany(c => c.ContactInfo, ownedNavigationBuilder =>
{
    ownedNavigationBuilder.ToJson();
});

await context.Set<CaseCustomer>().AsNoTrackingWithIdentityResolution().Include(cc => cc.Customer).Where(e => e.CaseId == 1).ToListAsync();

project to reproduce issue:
https://github.com/DieBunnyxxx/EFCoreJsonException/tree/master

Include provider and version information

EF Core version: 8.0.1
Database provider: Microsoft.EntityFrameworkCore.SqlServer
Target framework: NET 8.0

[ajcvickers]

Note for triage: still repros on latest daily; regression from EF7.

[maumar]

for queries with JSON (when we are streaming), we still need to go through the entire stream even if we have an entry in the change tracker already. In case of TrackAll we have special logic that stores entity from the change tracker into a variable and we store bool value indicating that we already have entity instance. We loop through entire stream and and the end we discard the object that we constructed from the stream and instead use the one from change tracker.

For NoTrackingWithIdentityResolution we don't do that. Instead, when we find that the object is in the change tracker already we short circuit. This leaves JSON stream in unexpected state, since the data (that is there and waiting to be read) is pending but we already move to processing next entity/whatnot.

[maumar]

There is a fundamental problem with using NTWIR and not projecting root entity. For Tracking queries we don't allow that (i.e. we throw), but currently for NTWIR we do. When using change tracker, the problem we can get ourselves into is that collection navigation results are being populated in the order they are being materialized in the query. When parent entity is present we process it in a separate code path (Include), and results of that processing always come first in the materializer code. So, "full" JSON collection projections always are discovered and materialized first and therefore all subsequent instances of those collections are populated in the correct ("original") order.

For no tracking we always materialize everything separately so the problem doesn't exist.

For NTWIR, we can have a situation right now where a collection element is projected first, and only later the full collection is projected. In that scenario, in the "full" collection, that entity discovered earlier would come first in the list. This is an issue for JSON, because collections are supposed to be ordered.

Example problematic query:

            ss => ss.Set<JsonEntityBasic>().Select(
                x => new
                {
                    x.Id,
                    Duplicate = x.OwnedReferenceRoot.OwnedCollectionBranch[1],
                    Original = x.OwnedReferenceRoot,
                }).AsNoTrackingWithIdentityResolution(),

in the Original, OwnedCollectionBranch will contain OwnedCollectionBranch[1], before the rest of the elements.

[maumar]

We can either do sophisticated ordering of which elements get materialized before which (i.e. full collections before individual element accesses), or block the scenario where you don't project root in NTWIR, just like we do for tracking query.
In long term the correct fix is to implement Ordered Collections: #9067, then the issue goes away.

[roji]

OTOH blocking this scenario (for now) doesn't seem too bad to me. But given that this does work correctly for tracked queries, is it simply an implementation difficulty for the NTWIR code path or something? I mean, if tracking queries can do this correctly without ordered collections, it seems odd that we wouldn't be able to do the same for NTWIT no?

In any case, we may really want to have a design discussion on whether we want to continue investing in owned entity JSON mapping - as opposed to complex types. Unless I'm mistaken, this wouldn't be a problem with complex types, since we don't do identity resolution on them anyway (i.e. we'd project different instances, just like in the no-tracking case?).

[maumar]

NTWIT works fine for the case where you also project parent entity (just like regular Tracking), after fix from #33101 is applied. Cases where we don't project the parent are blocked for Tracking queries, so there is no issue. If you removed the block, we would be hitting the same problem. NTWIT doesn't have the same block so the problem gets exposed.

[lenardchristopher]

Thanks for fixing this!

Looks like its not going to roll out until EF9. Any chance this will be fixed in EF8 considering it's a regression from EF7?

[OrgesKreka]

@maumar Hello,
I am facing the same issue, even though I am not using .AsNoTrackingWithIdentityResolution().

My models are like below:

public class Foo
{
	public Foo1? Foo1{get;set;}
}

public class Foo1
{
	public List<CustomField> CustomFields{get;set;}
}


public record CustomField
{
	public Guid CustomFieldIdentifier { get; set; }
	public decimal? Value { get; set; } = default(decimal?);
}

And the configuration is like this:

protected override void ConfigureOtherProperties(EntityTypeBuilder<Foo> builder)
{
	builder.ToTable(nameof(Foo), "Foo");

 
	builder.OwnsOne(x => x.Foo1, ownedNavigationBuilder =>
	{
                // this were not working.
		//ownedNavigationBuilder.OwnsMany(p => p.CustomFields);
		//ownedNavigationBuilder.Ignore(p => p.CustomFields);

		ownedNavigationBuilder.ToJson();
	});
}

Every time I try to query the list of Foo I get the exception:

Microsoft.EntityFrameworkCore.Query: Error: An exception occurred while iterating over the results of a query for context type 'FooContext'.
System.InvalidOperationException: Invalid token type: 'StartObject'.
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.IncludeJsonEntityCollection[TIncludingEntity,TIncludedCollectionElement](QueryContext queryContext, Object[] keyPropertyValues, JsonReaderData jsonReaderData, TIncludingEntity entity, Func`4 innerShaper, Action`1 getOrCreateCollectionObject, Action`2 fixup, Boolean trackingQuery)
   at lambda_method1131(Closure, QueryContext, DbDataReader, ResultContext, SplitQueryResultCoordinator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

I am using EFCore 8.0.7, with postgres as database.

Please can you tell me if there is any workaround into this?!

[maumar]

@OrgesKreka can you file a new issue for this? Ideally, please include the JSON data that has been used to reproduce this. This generally happens when the JSON is shaped in a way that is according to what EF thinks it should be (e.g. expecting collection, got a single object). So for start I'd take a look at the JSON itself, but impossible to say more without seeing a full repro.

[maumar]

@lenardchristopher apologies for late reply. We were debating whether to port the fix but ultimately decided against it. Reason is that the fix is quite complex, with a significant risk of introducing a regression (in fact, the initial fix did cause a regression which we luckily caught) and AsNoTrackingWithIdentityResolution is bit of a niche functionality. So this is a rare case where we won't be fixing a regression :( Sorry about that!