Redesign conversion between JSON objects and ASP.NET models #1091

bart-degreed · 2021-09-29T10:25:59Z

This PR rewrites the conversion between JSON objects (ResourceObject, RelationshipObject) to the IIdentifiable entity classes which are used in ASP.NET model binding, ie the parameters on controller action methods.

Fixes #1028.
Fixes #1024.
Fixes #1089.
Closes #661.

Note: Client serialization support was removed in #1075.

Breaking: Added JsonApiOptions.IncludeRequestBodyInErrors (false by default) to write request body in error meta when unable to read it. Before, we'd always append it to detail message, except for atomic:operations requests. This is off by default to hide sensitive data in production logs.
Breaking: Added JsonApiOptions.AllowUnknownFieldsInRequestBody (false by default) to fail on unknown attributes/relationships in the request body. Before, we'd just ignore them, which sometimes makes debugging harder. Turn this on for the old behavior, typically if your API needs to support multiple endpoint versions.
Breaking: Unified several similar errors and more precise value for error.source.pointer when unable to read the request body. Before, we'd only point to the affected atomic:operation index, but now we provide deep paths like /atomic:operations[2]/data/relationships/performers/data[3]/type (and also on resource/relationship endpoints).
Fixed: Log at Error instead of Info when processing an operation throws an unknown error.
Fixed the number of resource definition callbacks for sparse fieldsets.
Fixed: Missing top-level link in error- and operations response.
Fixed: ILinkBuilder.GetResourceLinks was also used to render relationship links.
Fixed injected IJsonApiRequest management: restore backup after processing operations, so you'll get the correct injected top-level state when not processing an operation, for example in IResourceDefinition.OnSerialize.
Removed RequestScopedServiceProvider and IFieldsToSerialize (no longer used)
Use immutable sets instead of lists for include expressions. The included order in response body is now driven by the declaration order in the IIdentifiable class and no longer based on the order in ?include= query string value. Note though that this is an implementation detail: the order is undefined in JSON:API spec and we may change it over time. Also note there are corner cases, such as circular references and multiple paths, where we don't preserve this ordering for performance.
Fixed: return Content-Length header in HEAD response.
Breaking: Reorganized JsonApiDotNetCore.Serialization namespace. To customize response serialization, inherit from JsonApiDotNetCore.Serialization.Response.ResponseModelAdapter.
Breaking: Fixed: call ResourceDefinition.OnApplyIncludes for all nested children, even when empty. This allows adding extra nested includes from code.
Breaking: Renamed ResourceContext to ResourceType and exposed it through relationship left/right. This enabled to significantly reduce resource graph lookups from the codebase and reduce the dependency on the resource graph from various types. Code that used to be Type type = relationship.RightType should be changed to: Type type = relationship.RightType.ClrType.
Except for LinkBuilder, the (de)serialization flow no longer depends on HttpContext, which makes it easier to use in our tests and benchmark code. Instead, we heavily rely on the information in IJsonApiRequest, which is easy and cheap to instantiate.

QUALITY CHECKLIST

Changes implemented in code
Complies with our contributing guidelines
Adapted tests
Documentation updated
Created issue to update Templates: {ISSUE_NUMBER}

…ce, but do write them when logging

…ead it (false by default)

…n request body (false by default)

The existing validation logic inside controllers, JsonApiReader and RequestDeserializer/BaseDeserializer has been consolidated into a set of adapters that delegate to each other, passing along the current state. The dependency on `HttpContext` has been removed and similar errors are now grouped, in preparation to unify them. Because in this commit we always track the location of errors and write them to `error.source.pointer`, the error messages can be unified and simplified in future commits. In this commit, I've gone through great lenghts to preserve the existing error messages in our tests as much as possible, while keeping all tests green. All error tests in ReadWrite and Operations now have assertions on the source pointer. Added tests for missing/invalid 'data' in resource requests. Removed outdated unit tests and added new one for handling attributes of various CLR types. Updated benchmark code. The synthetic BenchmarkDotNet reports it has become 3-7% slower (we're talking microseconds here) compared to the old code. But when running full requests with our pipeline-instrumentation turned on, the difference falls in the range of background noise and is thus unmeasurable. ``` BEFORE: Read request body ............................... 0:00:00:00.0000167 ... 1.08% } 167 + 434 = 601 JsonSerializer.Deserialize .................... 0:00:00:00.0000560 ... 3.62% + Deserializer.Build (single) ................... 0:00:00:00.0000434 ... 2.80% } AFTER: Read request body ............................... 0:00:00:00.0000511 ... 3.43% JsonSerializer.Deserialize .................... 0:00:00:00.0000537 ... 3.61% ```

…ody does not get added to the error response meta

…/array

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD

… when empty

…onship left/right. This enabled to reduce many resource graph lookups from the codebase.

…Builder for easier reuse.

bart-degreed · 2021-10-08T11:45:56Z

@maurei I wasn't sure if we need to port some unit tests from https://github.com/json-api-dotnet/JsonApiDotNetCore/blob/c933e8b56849c6cbe6791243668e3735d04491b0/test/UnitTests/Serialization/Server/IncludedResourceObjectBuilderTests.cs.

…merge.

…er before merge.

It now always emits related resources in relationship declaration order, even in deeply nested circular chains where a subsequent inclusion chain is deeper and adds more relationships to an already converted resource. Performance impact, summary: - In the endpoint test, this commit improves performance for rendering includes, while slightly decreasing it for the other two scenarios. All are still faster or the same compared to the master branch. - In BenchmarkDotNet, this commit slightly increases rendering time, compared to earlier commits in this PR, but it is still faster than the master branch. Measurement results for GET http://localhost:14140/api/v1/todoItems?include=owner,assignee,tags: Write response body ............................. 0:00:00:00.0010385 -> 0:00:00:00.0008060 ... 77% (was: 130%) Measurement results for GET http://localhost:14140/api/v1/todoItems?filter=and(startsWith(description,'T'),equals(priority,'Low'),not(equals(owner,null)),not(equals(assignee,null))): Write response body ............................. 0:00:00:00.0006601 -> 0:00:00:00.0005629 ... 85% (was: 70%) Measurement results for POST http://localhost:14140/api/v1/operations (10x add-resource): Write response body ............................. 0:00:00:00.0003432 -> 0:00:00:00.0003411 ... 99% (was: 95%) | Method | Mean | Error | StdDev | |---------------------------------- |---------:|--------:|--------:| | LegacySerializeOperationsResponse | 239.0 us | 3.17 us | 2.81 us | | SerializeOperationsResponse | 153.8 us | 0.72 us | 0.60 us | | (new) SerializeOperationsResponse | 168.6 us | 1.74 us | 1.63 us | | Method | Mean | Error | StdDev | |-------------------------------- |---------:|--------:|--------:| | LegacySerializeResourceResponse | 177.6 us | 0.56 us | 0.50 us | | SerializeResourceResponse | 101.3 us | 0.31 us | 0.29 us | | (new) SerializeResourceResponse | 123.7 us | 1.12 us | 1.05 us |

bart-degreed · 2021-10-27T10:16:38Z

For review and future reference, I'm adding the data and steps produced by ResponseModelAdapterTests below.

resource-model

Resources_in_deeply_nested_circular_chain_are_written_in_relationship_declaration_order

Resources_in_overlapping_deeply_nested_circular_chains_are_written_in_relationship_declaration_order

* Replaced suppression with fix * Fixed: do not duplicate the list of JSON:API errors in meta stack trace, but do write them when logging * Breaking: Added option to write request body in meta when unable to read it (false by default) * Breaking: Added option to allow unknown attribute/relationship keys in request body (false by default) * Fixed invalid test * Rewrite of reading the request body and converting it into models. The existing validation logic inside controllers, JsonApiReader and RequestDeserializer/BaseDeserializer has been consolidated into a set of adapters that delegate to each other, passing along the current state. The dependency on `HttpContext` has been removed and similar errors are now grouped, in preparation to unify them. Because in this commit we always track the location of errors and write them to `error.source.pointer`, the error messages can be unified and simplified in future commits. In this commit, I've gone through great lenghts to preserve the existing error messages in our tests as much as possible, while keeping all tests green. All error tests in ReadWrite and Operations now have assertions on the source pointer. Added tests for missing/invalid 'data' in resource requests. Removed outdated unit tests and added new one for handling attributes of various CLR types. Updated benchmark code. The synthetic BenchmarkDotNet reports it has become 3-7% slower (we're talking microseconds here) compared to the old code. But when running full requests with our pipeline-instrumentation turned on, the difference falls in the range of background noise and is thus unmeasurable. ``` BEFORE: Read request body ............................... 0:00:00:00.0000167 ... 1.08% } 167 + 434 = 601 JsonSerializer.Deserialize .................... 0:00:00:00.0000560 ... 3.62% + Deserializer.Build (single) ................... 0:00:00:00.0000434 ... 2.80% } AFTER: Read request body ............................... 0:00:00:00.0000511 ... 3.43% JsonSerializer.Deserialize .................... 0:00:00:00.0000537 ... 3.61% ``` * Updated error message for unknown attribute/relationship * Updated error message for unknown resource type * Unified error messages about the absense or presense of 'id' and 'lid' * Unified error messages about the absense of 'type' * Unified error messages about incompatible types * Revert the use of different exception, because this way the request body does not get added to the error response meta * Unified error messages about mismatches in 'id' and 'lid' values * Additional unification of error messages * Unified error messages for failed type conversion * Fix cibuild * Unified error messages about data presense and its value: null/object/array * Unified remaining error messages * Sealed types and reduced dependencies * Fixed broken test on linux * Adapter renames: - IOperationsDocumentAdapter -> IDocumentInOperationsRequestAdapter - IResourceDocumentAdapter -> IDocumentInResourceOrRelationshipRequestAdapter - IOperationResourceDataAdapter -> IResourceDataInOperationsRequestAdapter * Added missing assertions on request body in error meta * Refactorings: - Changed deserializer to accept Error(s) instead of Document (used to be ErrorDocument, but they were merged) - Write request body in error meta instead of top-level meta - Fixed: Log at Error instead of Info when processing an operation throws unknown error * Enhanced existing tests: Assert on resource type when `included` contains mixed types * Fixed the number of resource definition callbacks for sparse fieldsets * Refactor: removed OperationContainer.Kind because IJsonApiRequest.WriteOperation is now populated consistently; applied c# pattern usage * Added test to capture the current behavior to return data:null for void operations. The spec allows an empty object too, but I don't consider this important enough to add yet another JSON converter with all its overhead. * Improved tests for includes * Rewrite of rendering the response body from models - Added temporary bridge class to toggle between old/new code - Fixed: Missing top-level link in error- and operations response - Fixed: `ILinkBuilder.GetResourceLinks` was also used to render relationship links - IJsonApiRequest management: restore backup after processing operations - Refreshed serialization benchmarks Measurement results for GET http://localhost:14140/api/v1/todoItems?include=owner,assignee,tags: Write response body ............................ 0:00:00:00.0010385 -> 0:00:00:00.0013343 = 130% Measurement results for GET http://localhost:14140/api/v1/todoItems?filter=and(startsWith(description,'T'),equals(priority,'Low'),not(equals(owner,null)),not(equals(assignee,null))): Write response body ............................ 0:00:00:00.0006601 -> 0:00:00:00.0004624 = 70% Measurement results for POST http://localhost:14140/api/v1/operations (10x add-resource): Write response body ............................ 0:00:00:00.0003432 -> 0:00:00:00.0003289 = 95% | Method | Mean | Error | StdDev | |---------------------------------- |---------:|--------:|--------:| | SerializeOperationsResponse | 153.8 us | 0.72 us | 0.60 us | | LegacySerializeOperationsResponse | 239.0 us | 3.17 us | 2.81 us | | Method | Mean | Error | StdDev | |-------------------------------- |---------:|--------:|--------:| | SerializeResourceResponse | 101.3 us | 0.31 us | 0.29 us | | LegacySerializeResourceResponse | 177.6 us | 0.56 us | 0.50 us | * Avoid closure in hot code path to reduce allocations * Cleanup reader and writer * Removed old code * Fixed: crash in test serializer on assertion failure * Removed RequestScopedServiceProvider * Use sets for include expressions * Fixed: return Content-Length header in HEAD response https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD * Reorganized JADNC.Serialization namespace * Created custom exception for remaining errors * Fixed: call ResourceDefinition.OnApplyIncludes for all children, even when empty * Renamed ResourceContext to ResourceType and exposed it through relationship left/right. This enabled to reduce many resource graph lookups from the codebase. * Moved logic to build resource graph from DbContext into ResourceGraphBuilder for easier reuse. * Opened up ResponseModelAdapter for extensibility * Check off roadmap entry * Review feedback * Simplified existing tests * Added extra test for data:null in relationship * Added test for broken resource linkage * Ported existing unit tests and changed how included[] is built. It now always emits related resources in relationship declaration order, even in deeply nested circular chains where a subsequent inclusion chain is deeper and adds more relationships to an already converted resource. Performance impact, summary: - In the endpoint test, this commit improves performance for rendering includes, while slightly decreasing it for the other two scenarios. All are still faster or the same compared to the master branch. - In BenchmarkDotNet, this commit slightly increases rendering time, compared to earlier commits in this PR, but it is still faster than the master branch. Measurement results for GET http://localhost:14140/api/v1/todoItems?include=owner,assignee,tags: Write response body ............................. 0:00:00:00.0010385 -> 0:00:00:00.0008060 ... 77% (was: 130%) Measurement results for GET http://localhost:14140/api/v1/todoItems?filter=and(startsWith(description,'T'),equals(priority,'Low'),not(equals(owner,null)),not(equals(assignee,null))): Write response body ............................. 0:00:00:00.0006601 -> 0:00:00:00.0005629 ... 85% (was: 70%) Measurement results for POST http://localhost:14140/api/v1/operations (10x add-resource): Write response body ............................. 0:00:00:00.0003432 -> 0:00:00:00.0003411 ... 99% (was: 95%) | Method | Mean | Error | StdDev | |---------------------------------- |---------:|--------:|--------:| | LegacySerializeOperationsResponse | 239.0 us | 3.17 us | 2.81 us | | SerializeOperationsResponse | 153.8 us | 0.72 us | 0.60 us | | (new) SerializeOperationsResponse | 168.6 us | 1.74 us | 1.63 us | | Method | Mean | Error | StdDev | |-------------------------------- |---------:|--------:|--------:| | LegacySerializeResourceResponse | 177.6 us | 0.56 us | 0.50 us | | SerializeResourceResponse | 101.3 us | 0.31 us | 0.29 us | | (new) SerializeResourceResponse | 123.7 us | 1.12 us | 1.05 us | * Fixed cibuild

Replaced suppression with fix

8534c41

bart-degreed marked this pull request as draft September 29, 2021 10:26

bart-degreed force-pushed the serialization-refactorings branch 3 times, most recently from c4a6a53 to 525dac7 Compare September 29, 2021 12:58

Bart Koelman added 4 commits September 29, 2021 15:19

Fixed: do not duplicate the list of JSON:API errors in meta stack tra…

f502628

…ce, but do write them when logging

Breaking: Added option to write request body in meta when unable to r…

afe119d

…ead it (false by default)

Breaking: Added option to allow unknown attribute/relationship keys i…

2b47438

…n request body (false by default)

Fixed invalid test

2b1c32b

bart-degreed force-pushed the serialization-refactorings branch 3 times, most recently from 8d3dfce to 5f618d2 Compare September 29, 2021 20:29

Bart Koelman added 10 commits September 30, 2021 12:37

Updated error message for unknown attribute/relationship

d247ba2

Updated error message for unknown resource type

8dbafd2

Unified error messages about the absense or presense of 'id' and 'lid'

d243dc3

Unified error messages about the absense of 'type'

4c3bf71

Unified error messages about incompatible types

1700c00

Revert the use of different exception, because this way the request b…

d4549e8

…ody does not get added to the error response meta

Unified error messages about mismatches in 'id' and 'lid' values

c2e31d3

Additional unification of error messages

5ea961f

Unified error messages for failed type conversion

6244ef7

bart-degreed force-pushed the serialization-refactorings branch from f39c18b to 6244ef7 Compare September 30, 2021 10:37

Bart Koelman added 3 commits September 30, 2021 14:01

Fix cibuild

72f02b0

Unified error messages about data presense and its value: null/object…

fe10147

…/array

Unified remaining error messages

2322ac8

bart-degreed force-pushed the serialization-refactorings branch 2 times, most recently from c0d7915 to 804464d Compare September 30, 2021 16:28

Bart Koelman added 2 commits October 1, 2021 09:51

Sealed types and reduced dependencies

d254ac9

Fixed broken test on linux

0923c8b

Bart Koelman added 7 commits October 8, 2021 11:22

Removed RequestScopedServiceProvider

fbdcefb

Use sets for include expressions

7fcb58e

Fixed: return Content-Length header in HEAD response

f8d71f2

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD

Reorganized JADNC.Serialization namespace

ff763f1

Created custom exception for remaining errors

80fe2b2

Fixed: call ResourceDefinition.OnApplyIncludes for all children, even…

f6caf3a

… when empty

Renamed ResourceContext to ResourceType and exposed it through relati…

0efde1b

…onship left/right. This enabled to reduce many resource graph lookups from the codebase.

bart-degreed force-pushed the serialization-refactorings branch from 9d057f1 to 0efde1b Compare October 8, 2021 09:22

Bart Koelman added 2 commits October 8, 2021 12:42

Moved logic to build resource graph from DbContext into ResourceGraph…

f7e7da4

…Builder for easier reuse.

Opened up ResponseModelAdapter for extensibility

3d98057

bart-degreed marked this pull request as ready for review October 8, 2021 11:43

bart-degreed requested a review from maurei October 8, 2021 11:43

bart-degreed pushed a commit that referenced this pull request Oct 8, 2021

**TEMP** Squashed commits from #1091. Should rebase on master before …

687d905

…merge.

Check off roadmap entry

2e3659f

bart-degreed pushed a commit that referenced this pull request Oct 18, 2021

**TEMP** Squashed commits from #1091 and #1093. Should rebase on mast…

4bfe551

…er before merge.

Bart Koelman added 5 commits October 20, 2021 11:48

Review feedback

2c4e75c

Simplified existing tests

7158595

Added extra test for data:null in relationship

7c6684f

Added test for broken resource linkage

4a8bfdd

bart-degreed force-pushed the serialization-refactorings branch from 88e5552 to 0f41f85 Compare October 26, 2021 08:22

Fixed cibuild

5dfb6ea

maurei approved these changes Oct 27, 2021

View reviewed changes

maurei merged commit 758a191 into master Oct 27, 2021

maurei deleted the serialization-refactorings branch October 27, 2021 14:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redesign conversion between JSON objects and ASP.NET models #1091

Redesign conversion between JSON objects and ASP.NET models #1091

bart-degreed commented Sep 29, 2021 •

edited

Loading

bart-degreed commented Oct 8, 2021

bart-degreed commented Oct 27, 2021 •

edited

Loading

Redesign conversion between JSON objects and ASP.NET models #1091

Redesign conversion between JSON objects and ASP.NET models #1091

Conversation

bart-degreed commented Sep 29, 2021 • edited Loading

QUALITY CHECKLIST

bart-degreed commented Oct 8, 2021

bart-degreed commented Oct 27, 2021 • edited Loading

bart-degreed commented Sep 29, 2021 •

edited

Loading

bart-degreed commented Oct 27, 2021 •

edited

Loading