@odata.nextLink inconsistent with specification - continuation

[tsharp]

Continuation of the discussion in #880

The implementation details of the @odata.nextLink is inconsistent with the fundamental implementation details of $top and $skip. These should both be client controllable values such that a client may specify the number of items returned and the skipped items in the query; however, when $top is specified by the client @odata.nextLink property is not returned in the response.

In the example given in the PR to demonstrate the proper functionality of the $top and $skip properties: #880 (comment) the usage is to specify a $top value greater than the default page size of 5, and in the response it returns $top=2&$skip=5 which again is inconsistent with how $top and $skip should work.

If one were doing manual page calculations for a number of reasons including not using Entity Framework - then having precise control over each of the values is important.

For example - cosmos query:

SELECT f.id, f.address.city
FROM Families f
ORDER BY f.address.city
OFFSET 1 LIMIT 1

or SQL Query:

SELECT First Name + ' ' + Last Name 
FROM Employees 
ORDER BY First Name 
OFFSET 10 ROWS 
FETCH NEXT 5 ROWS ONLY;

You would most likely be replacing the OFFSET values with $skip and the FETCH NEXT or LIMIT values with $top. As such, when you specify a page smaller than the default number of items to return - common in reducing the items returned in UI/UX scenarios, you return a specific number of items for that page. The total possible number of items is variable depending on the view. With the current implementation details - a developer is left with 2 options to correct the issue - write a custom function to generate the proper @odata.nextLink or use an encoded $skiptoken. Rather - if there was an option to allow the developer to either enable this behavior, as given by a number of very public api's (Microsoft Graph) then it would align better with the expectation of that particular feature.

In the original issue - I had stated that this was a regression from previous functionality and to a point - this is the truth. We had implementations in .net 2.1 and 3.1 that had this very bit of functionality available and when moving to OData 7.x - something broke in the process and we had thought we solved the issue, but we had not fully corrected the issue until we discovered that the next link generation functionality was intentionally omitting the property.

Proposed Solution 1

We leave the PR as is and allow for variable $top values specified by the client.

Proposed Solution 2

Further examine the possible scenarios where this would not be appropriate - rather - have an opt-in situation where the developer must make an intentional decision to enable the feature. This may mean that the following is true:

1. A min and max $top value may be specified
2. The $skip value MUST be a multiple of $top
3. If the min or max value of $top is observed - an api exception / error is given.

References

https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptiontop

The $top system query option specifies a non-negative integer n that limits the number of items returned from a collection. The service returns the number of available items up to but not greater than the specified value n.

https://docs.oasis-open.org/odata/odata-json-format/v4.0/cs01/odata-json-format-v4.0-cs01.html#_Toc365464689

The odata.nextLink annotation indicates that a response is only a subset of the requested collection of entities or collection of entity references. It contains a URL that allows retrieving the next subset of the requested collection.

[tsharp]

@gathogojr

[habbes]

@tsharp thanks for opening this up for discussion.

If you want to control the page size from the client, maybe what you're looking for is the odata.maxpagesize preference header (see: http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358876) instead of the $top query option. Take a look at that and see whether it works for your use case.

When using $top the client, expresses the maximum number of entries they went to receive in the result, the server should not return more than that. When using @nextLink, the server is signaling that they are returning a subset of the requested collection, and the next subset can be fetched from the next link. When the $top value is less than the page size, what the server returns is the entire require collection requested by the client, therefore it should not include a next link, because it has already exhausted the entire collection requested by the client.

It is possible that some MS Graph APIs have in implementation that doesn't match the protocol in the spec. However, the current implementation of the library does match the spec and is well documented. I don't think it makes sense at this point to introduce this new feature in the library for this use case because it's not supported by the spec and it may introduce unexpected behaviour.

If the suggestion doesn't work for your use case, then I think it would be appropriate to consider customizing the next link or skip token handler since this is a custom requirement.

[tsharp]

<argument removed - it's pointless>

I suppose that I'll continue to use my own implementation and hope that it doesn't break again with the next upgrade.