[Rollup] Add new capabilities endpoint based on concrete rollup indices #30401
Conversation
This adds a `key` parameter to the GetRollupCaps API, which allows the user to specify how they would like the capabilities grouped at the top level. The default behavior is as before: grouping by index pattern. But if the user specifies `key: rollup_index`, it groups the capabilities by the concrete rollup index where the rollups are stored.
polyfractal
added
>enhancement
review
:StorageEngine/Rollup
|
Pinging @elastic/es-search-aggs |
These need to be 7.0 until the feature is backported, to keep bwc/mixed cluster tests passing.
|
Jenkins, run gradle build tests |
|
/cc @colings86 since Jim is off on holiday for a while :) |
|
|
||
| public Request(String indexPattern) { | ||
| public Request(String indexPattern, String key) { |
There was a problem hiding this comment.
This is a little tricky because if we are allow an index pattern to resolve to concrete indexes then we should allow IndicesOptions as well so the user can control how wildcard expansion etc. behaves. This also introduces a bit of weirdness because the GetRollupCapsAction is a cluster action (the name starts with cluster: which is correct for when you are specifying the index pattern option because you are effectively asking the job not the indexes but when you are asking for the indexes option the action should really be an indices action (the name should start with indices:). I wonder if this actually needs to be two different APIs? or whether there is a way to make this always an indices: action?
There was a problem hiding this comment.
Hm, maybe two endpoints is best.
There was a problem hiding this comment.
Durp, hit enter too soon.
Right now I don't think it's a problem, since the options are:
-
"Index pattern" specified in the job, which we are not actually resolving as index patterns. These are just checking string equality against the job's defined index pattern.
-
The new behavior introduced in this PR, which requests the rollup job's concrete index. This does not support patterns at the moment, only a single, explicit index name.
So the PR as it stands should be ok. But Kibana has said they would like to be able to specify patterns with the concrete rollup index (rollup_storage_*), which would circle back to the concerns you've raised.
|
@polyfractal could you add the target versions to this PR please? |
polyfractal
changed the title
|
Moved things over to a new endpoint, updated the OP description and changed the title. Still need to edit the docs. Edit: lotsa test failures, i think i broke something :) |
|
Jenkins, run gradle build tests |
|
Tests now passing after finding that assertion issue, think this is ready for another review |
…lastic#30401) This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint. - Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern. - One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
* es/master: Add Index UUID to `/_stats` Response (#31871) Painless: Move and Rename Several Methods in the lookup package (#32105) Bypass highlight query terms extraction on empty fields (#32090) Switch non-x-pack to new style requests (#32106) [Rollup] Add new capabilities endpoint for concrete rollup indices (#30401) Revert "[test] disable packaging tests for suse boxes" SQL: allow LEFT and RIGHT as function names (#32066) DOCS: put LIMIT 10 to the SQL query (#32065) [test] turn on host io cache for opensuse (#32053) Tweaked Elasticsearch Service links for SEO
This adds akeyquery string parameter to the GetRollupCaps API, which allows the user to specify how they would like the capabilities grouped at the top level.The default behavior is as before: grouping by index pattern. But if the user specifieskey: rollup_index, it groups the capabilities by the concrete rollup index where the rollups are stored. The rest of the response is unchanged from before (shows a list of all the jobs in that index, their config, etc)This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint; it's a subtle but distinct difference.
logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern./cc @jen-huang