Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Rollup] Add new capabilities endpoint based on concrete rollup indices #30401

Merged
merged 14 commits into from
Jul 16, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions x-pack/docs/en/rest-api/rollup/rollup-index-caps.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
[role="xpack"]
[[rollup-get-rollup-index-caps]]
=== Get Rollup Index Capabilities
++++
<titleabbrev>Get Rollup Index Caps</titleabbrev>
++++

experimental[]

This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored).
A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs.

This API will allow you to determine:

1. What jobs are stored in an index (or indices specified via a pattern)?
2. What target indices were rolled up, what fields were used in those rollups and what aggregations can be performed on each job?

==== Request

`GET {index}/_xpack/rollup/data`

//===== Description

==== Path Parameters

`index`::
(string) Index or index-pattern of concrete rollup indices to check for capabilities.



==== Request Body

There is no request body for the Get Jobs API.

==== Authorization

You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see
{xpack-ref}/security-privileges.html[Security Privileges].

==== Examples

Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job, which stores it's data in `sensor_rollup`:

[source,js]
--------------------------------------------------
PUT _xpack/rollup/job/sensor
{
"index_pattern": "sensor-*",
"rollup_index": "sensor_rollup",
"cron": "*/30 * * * * ?",
"page_size" :1000,
"groups" : {
"date_histogram": {
"field": "timestamp",
"interval": "1h",
"delay": "7d"
},
"terms": {
"fields": ["node"]
}
},
"metrics": [
{
"field": "temperature",
"metrics": ["min", "max", "sum"]
},
{
"field": "voltage",
"metrics": ["avg"]
}
]
}
--------------------------------------------------
// CONSOLE
// TEST[setup:sensor_index]

If at a later date, we'd like to determine what jobs and capabilities were stored in the `sensor_rollup` index, we can use the Get Rollup
Index API:

[source,js]
--------------------------------------------------
GET /sensor_rollup/_xpack/rollup/data
--------------------------------------------------
// CONSOLE
// TEST[continued]

Note how we are requesting the concrete rollup index name (`sensor_rollup`) as the first part of the URL.
This will yield the following response:

[source,js]
----
{
"sensor_rollup" : {
"rollup_jobs" : [
{
"job_id" : "sensor",
"rollup_index" : "sensor_rollup",
"index_pattern" : "sensor-*",
"fields" : {
"node" : [
{
"agg" : "terms"
}
],
"temperature" : [
{
"agg" : "min"
},
{
"agg" : "max"
},
{
"agg" : "sum"
}
],
"timestamp" : [
{
"agg" : "date_histogram",
"time_zone" : "UTC",
"interval" : "1h",
"delay": "7d"
}
],
"voltage" : [
{
"agg" : "avg"
}
]
}
}
]
}
}
----
// TESTRESPONSE


The response that is returned contains information that is similar to the original Rollup configuration, but formatted
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
the index pattern that the job was targeting.

Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.

Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
configurations available.


Like other APIs that interact with indices, you can specify index patterns instead of explicit indices:

[source,js]
--------------------------------------------------
GET /*_rollup/_xpack/rollup/data
--------------------------------------------------
// CONSOLE
// TEST[continued]

Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ public void writeTo(StreamOutput out) throws IOException {
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
builder.startObject();
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
entry.getValue().toXContent(builder, params);
entry.getValue().toXContent(builder, params);
}
builder.endObject();
return builder;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.core.rollup.action;


import org.elasticsearch.action.Action;
import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionRequestBuilder;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.action.ActionResponse;
import org.elasticsearch.action.IndicesRequest;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.client.ElasticsearchClient;
import org.elasticsearch.common.ParseField;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.io.stream.Writeable;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.ToXContentObject;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.xpack.core.rollup.RollupField;

import java.io.IOException;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import java.util.Objects;

public class GetRollupIndexCapsAction extends Action<GetRollupIndexCapsAction.Response> {

public static final GetRollupIndexCapsAction INSTANCE = new GetRollupIndexCapsAction();
public static final String NAME = "indices:data/read/xpack/rollup/get/index/caps";
public static final ParseField CONFIG = new ParseField("config");
public static final ParseField STATUS = new ParseField("status");
private static final ParseField INDICES_OPTIONS = new ParseField("indices_options");

private GetRollupIndexCapsAction() {
super(NAME);
}

@Override
public Response newResponse() {
return new Response();
}

public static class Request extends ActionRequest implements IndicesRequest.Replaceable, ToXContent {
private String[] indices;
private IndicesOptions options;

public Request(String[] indices) {
this(indices, IndicesOptions.STRICT_EXPAND_OPEN_FORBID_CLOSED);
}

public Request(String[] indices, IndicesOptions options) {
this.indices = indices;
this.options = options;
}

public Request() {}

@Override
public IndicesOptions indicesOptions() {
return options;
}

@Override
public String[] indices() {
return indices;
}

@Override
public IndicesRequest indices(String... indices) {
Objects.requireNonNull(indices, "indices must not be null");
for (String index : indices) {
Objects.requireNonNull(index, "index must not be null");
}
this.indices = indices;
return this;
}

@Override
public void readFrom(StreamInput in) throws IOException {
super.readFrom(in);
this.indices = in.readStringArray();
this.options = IndicesOptions.readIndicesOptions(in);
}

@Override
public void writeTo(StreamOutput out) throws IOException {
super.writeTo(out);
out.writeStringArray(indices);
options.writeIndicesOptions(out);
}

@Override
public ActionRequestValidationException validate() {
return null;
}

@Override
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
builder.array(RollupField.ID.getPreferredName(), indices);
builder.field(INDICES_OPTIONS.getPreferredName(), options);
return builder;
}

@Override
public int hashCode() {
return Objects.hash(Arrays.hashCode(indices), options);
}

@Override
public boolean equals(Object obj) {
if (obj == null) {
return false;
}
if (getClass() != obj.getClass()) {
return false;
}
Request other = (Request) obj;
return Arrays.equals(indices, other.indices)
&& Objects.equals(options, other.options);
}
}

public static class RequestBuilder extends ActionRequestBuilder<Request, Response> {

protected RequestBuilder(ElasticsearchClient client, GetRollupIndexCapsAction action) {
super(client, action, new Request());
}
}

public static class Response extends ActionResponse implements Writeable, ToXContentObject {

private Map<String, RollableIndexCaps> jobs = Collections.emptyMap();

public Response() {

}

public Response(Map<String, RollableIndexCaps> jobs) {
this.jobs = Objects.requireNonNull(jobs);
}

Response(StreamInput in) throws IOException {
jobs = in.readMap(StreamInput::readString, RollableIndexCaps::new);
}

public Map<String, RollableIndexCaps> getJobs() {
return jobs;
}

@Override
public void writeTo(StreamOutput out) throws IOException {
super.writeTo(out);
out.writeMap(jobs, StreamOutput::writeString, (out1, value) -> value.writeTo(out1));
}

@Override
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
builder.startObject();
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
entry.getValue().toXContent(builder, params);
}
builder.endObject();
return builder;
}

@Override
public int hashCode() {
return Objects.hash(jobs);
}

@Override
public boolean equals(Object obj) {
if (obj == null) {
return false;
}
if (getClass() != obj.getClass()) {
return false;
}
Response other = (Response) obj;
return Objects.equals(jobs, other.jobs);
}

@Override
public final String toString() {
return Strings.toString(this);
}
}
}
Loading