HTTP API Selection, projection, and pagination

[schmichael]

Nomad's HTTP API lacks a holistic implementation of 3 common features for data retrieval APIs:

1. Selection - filtering which objects (rows) are returned (eg the existing /v1/jobs?prefix=foo filter)
2. Projection - filtering which fields (columns) are returned (eg being able to customize the fields in AllocationListStub
3. Pagination - returning subsets of a full dataset at a time to avoid materializing large datasets in memory

A complicating factor in the implementation of these features is whether or not they should be implemented at the RPC layer as well since nearly all HTTP API's are backed by an RPC call.

There are multiple implementation approaches:

1. Extend /v1/search endpoint: The search endpoint already does limited selection and projection. It could be expanded to support more query features.
2. Progressive enhancement: Adding new query params, headers, etc to existing APIs in a backward compatible way. Selection is already being implemented this way, but progress is slow and filters rarely if ever reuse methods or code.
3. /v2/... API that provide these features in a holistic way while maintaining backward compatibility by leaving /v1/... APIs in place.

This has been discussed numerous times in numerous places, but I couldn't find an Github issue directly tracking it. Close this as a duplicate if another issue is found!

[tgross]

An interesting problem that's popping up in the work for CSI volume creation workflows is implementing the ListVolumes RPC. The interface to the plugin is paged because it's pulling the list of externally-defined storage volumes, which can be quite large. So we'll probably be threading the MaxEntries/StartToken/NextToken concern all the way up to the Nomad HTTP API. WIP PR is #10185

cc @DingoEatingFuzz

[DingoEatingFuzz]

🤩 This is very exciting news.

[tgross]

Implementation for CSI has landed in main with a NextToken and PerPage field in QueryOptions See api/api.go#L68-L74. This gives us just the initial plumbing in the HTTP API, which works for these endpoints because they're really just proxies for third-party storage provider APIs. But for data that's internal to Nomad, we'll need to figure out how to page into our data store as well.

[tomelliff]

Is this closed with #12186?

[tgross]

Just took a quick look through the API docs and it looks like pagination and filters still aren't implemented for ACL policies, Namespaces, CSI Plugins, Quotas, Recommendations, and Scaling Policies. @schmichael do you think that the projections issue is sufficiently covered by filtering in the current implementation? If so, should we break out the pagination/filtering for each of these endpoints? (None of these has a ton of data even on large clusters, so I suspect we maybe just don't care about these endpoints?)

[schmichael]

Let's do it! I think we've hit the highest quality endpoints/features, so let's approach the rest on an as-needed basis.

Closing this and we can open more specific issues as needed.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.