Skip to content

Commit

Permalink
Merge branch '6.x' into ccr-6.x
Browse files Browse the repository at this point in the history 
* 6.x:
  [DOCS] Fixes typos in security settings
  Add support for indexed shape routing in geo_shape query (#30760)
  [DOCS] Splits auditing.asciidoc into smaller files
  Painless: Types Section Clean Up (#30283)
  [test] java tests for archive packaging (#30734)
  Deprecate http.pipelining setting (#30786)
  [DOCS] Fix more edit URLs in Stack Overview (#30704)
  Use correct cluster state version for node fault detection (#30810)
  [DOCS] Fixes broken link for native realm
  [DOCS] Clarified audit.index.client.hosts (#30797)
  Change serialization version of doc-value fields.
  Add a `format` option to `docvalue_fields`. (#29639)
  [TEST] Don't expect acks when isolating nodes
  Fixes UpdateSettingsRequestStreamableTests mutate bug
  Revert "Add more yaml tests for get alias API (#29513)"
  Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled"
  Only allow x-pack metadata if all nodes are ready (#30743)
  Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled
  Use original settings on full-cluster restart (#30780)
  Only ack cluster state updates successfully applied on all nodes (#30672)
  Replace Request#setHeaders with addHeader (#30588)
  [TEST] remove endless wait in RestClientTests (#30776)
  QA: Add xpack tests to rolling upgrade (#30795)
  Add support for search templates to the high-level REST client. (#30473)
  Reduce CLI scripts to one-liners on Windows (#30772)
  Fold RestGetAllSettingsAction in RestGetSettingsAction (#30561)
  Add more yaml tests for get alias API (#29513)
  [Docs] Fix script-fields snippet execution (#30693)
  Convert FieldCapabilitiesResponse to a ToXContentObject. (#30182)
  Remove assert statements from field caps documentation. (#30601)
  Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (#30181)
  Add support for field capabilities to the high-level REST client. (#29664)
  [DOCS] Add SAML configuration information (#30548)
  [DOCS] Remove X-Pack references from SQL CLI (#30694)
  [Docs] Fix typo in circuit breaker docs (#29659)
  [Feature] Adding a char_group tokenizer (#24186)
  Increase the maximum number of filters that may be in the cache. (#30655)
  [Docs] Fix broken cross link in documentation
  Test: wait for netty threads in a JUnit ClassRule (#30763)
  [Security] Include an empty json object in an json array when FLS filters out all fields (#30709)
  [DOCS] fixed incorrect default
  [TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk
  Enable installing plugins from snapshots.elastic.co (#30765)
  Ignore empty completion input (#30713)
  Fix docs failure on language analyzers (#30722)
  [Docs] Fix inconsistencies in snapshot/restore doc (#30480)
  Add Delete Repository High Level REST API (#30666)
  Reduce CLI scripts to one-liners (#30759)
  • Loading branch information
@dnhatn
dnhatn committed May 24, 2018
2 parents f45a2eb + 7d63d3d commit 2aa0570
Show file tree
Hide file tree
Showing 266 changed files with 7,052 additions and 1,974 deletions.
28 changes: 24 additions & 4 deletions TESTING.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,9 @@ into it
vagrant ssh debian-9
--------------------------------------------

Now inside the VM, to run the https://github.com/sstephenson/bats[bats] packaging tests
Now inside the VM, start the packaging tests from the terminal. There are two packaging
test projects. The old ones are written with https://github.com/sstephenson/bats[bats]
and only run on linux. To run them do

--------------------------------------------
cd $PACKAGING_ARCHIVES
Expand All @@ -524,18 +526,36 @@ sudo bats $BATS_TESTS/*.bats
sudo bats $BATS_TESTS/20_tar_package.bats $BATS_TESTS/25_tar_plugins.bats
--------------------------------------------

To run the Java packaging tests, again inside the VM
The new packaging tests are written in Java and run on both linux and windows. On
linux (again, inside the VM)

--------------------------------------------
bash $PACKAGING_TESTS/run-tests.sh
# run the full suite
sudo bash $PACKAGING_TESTS/run-tests.sh
# run specific test cases
sudo bash $PACKAGING_TESTS/run-tests.sh \
org.elasticsearch.packaging.test.DefaultZipTests \
org.elasticsearch.packaging.test.OssZipTests
--------------------------------------------

or on Windows
or on Windows, from a terminal running as Administrator

--------------------------------------------
# run the full suite
powershell -File $Env:PACKAGING_TESTS/run-tests.ps1
# run specific test cases
powershell -File $Env:PACKAGING_TESTS/run-tests.ps1 `
org.elasticsearch.packaging.test.DefaultZipTests `
org.elasticsearch.packaging.test.OssZipTests
--------------------------------------------

Note that on Windows boxes when running from inside the GUI, you may have to log out and
back in to the `vagrant` user (password `vagrant`) for the environment variables that
locate the packaging tests and distributions to take effect, due to how vagrant provisions
Windows machines.

When you've made changes you want to test, keep the VM up and reload the tests and
distributions inside by running (on the host)

Expand Down
1 change: 1 addition & 0 deletions Vagrantfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ def linux_common(config,

config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
touch /etc/is_vagrant_vm
touch /is_vagrant_vm # for consistency between linux and windows
SHELL

# This prevents leftovers from previous tests using the
Expand Down
23 changes: 20 additions & 3 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ class VagrantTestPlugin implements Plugin<Project> {
static final List<String> DISTRIBUTIONS = unmodifiableList([
'archives:tar',
'archives:oss-tar',
'archives:zip',
'archives:oss-zip',
'packages:rpm',
'packages:oss-rpm',
'packages:deb',
Expand Down Expand Up @@ -242,13 +244,27 @@ class VagrantTestPlugin implements Plugin<Project> {
Task createLinuxRunnerScript = project.tasks.create('createLinuxRunnerScript', FileContentsTask) {
dependsOn copyPackagingTests
file "${testsDir}/run-tests.sh"
contents "java -cp \"\$PACKAGING_TESTS/*\" org.junit.runner.JUnitCore ${-> project.extensions.esvagrant.testClass}"
contents """\
if [ "\$#" -eq 0 ]; then
test_args=( "${-> project.extensions.esvagrant.testClass}" )
else
test_args=( "\$@" )
fi
java -cp "\$PACKAGING_TESTS/*" org.elasticsearch.packaging.VMTestRunner "\${test_args[@]}"
"""
}
Task createWindowsRunnerScript = project.tasks.create('createWindowsRunnerScript', FileContentsTask) {
dependsOn copyPackagingTests
file "${testsDir}/run-tests.ps1"
// the use of $args rather than param() here is deliberate because the syntax for array (multivalued) parameters is likely
// a little trappy for those unfamiliar with powershell
contents """\
java -cp "\$Env:PACKAGING_TESTS/*" org.junit.runner.JUnitCore ${-> project.extensions.esvagrant.testClass}
if (\$args.Count -eq 0) {
\$testArgs = @("${-> project.extensions.esvagrant.testClass}")
} else {
\$testArgs = \$args
}
java -cp "\$Env:PACKAGING_TESTS/*" org.elasticsearch.packaging.VMTestRunner @testArgs
exit \$LASTEXITCODE
"""
}
Expand Down Expand Up @@ -525,9 +541,10 @@ class VagrantTestPlugin implements Plugin<Project> {

if (LINUX_BOXES.contains(box)) {
javaPackagingTest.command = 'ssh'
javaPackagingTest.args = ['--command', 'bash "$PACKAGING_TESTS/run-tests.sh"']
javaPackagingTest.args = ['--command', 'sudo bash "$PACKAGING_TESTS/run-tests.sh"']
} else {
javaPackagingTest.command = 'winrm'
// winrm commands run as administrator
javaPackagingTest.args = ['--command', 'powershell -File "$Env:PACKAGING_TESTS/run-tests.ps1"']
}

Expand Down
1 change: 1 addition & 0 deletions client/rest-high-level/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ dependencies {
compile "org.elasticsearch.plugin:parent-join-client:${version}"
compile "org.elasticsearch.plugin:aggs-matrix-stats-client:${version}"
compile "org.elasticsearch.plugin:rank-eval-client:${version}"
compile "org.elasticsearch.plugin:lang-mustache-client:${version}"

testCompile "org.elasticsearch.client:test:${version}"
testCompile "org.elasticsearch.test:framework:${version}"
Expand Down
65 changes: 59 additions & 6 deletions client/rest-high-level/src/main/java/org/elasticsearch/client/RequestConverters.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import org.apache.lucene.util.BytesRef;
import org.elasticsearch.action.DocWriteRequest;
import org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksRequest;
import org.elasticsearch.action.admin.cluster.repositories.delete.DeleteRepositoryRequest;
import org.elasticsearch.action.admin.cluster.repositories.get.GetRepositoriesRequest;
import org.elasticsearch.action.admin.cluster.repositories.put.PutRepositoryRequest;
import org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest;
Expand All @@ -46,13 +47,14 @@
import org.elasticsearch.action.admin.indices.open.OpenIndexRequest;
import org.elasticsearch.action.admin.indices.refresh.RefreshRequest;
import org.elasticsearch.action.admin.indices.rollover.RolloverRequest;
import org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest;
import org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest;
import org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest;
import org.elasticsearch.action.admin.indices.shrink.ResizeRequest;
import org.elasticsearch.action.admin.indices.shrink.ResizeType;
import org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.fieldcaps.FieldCapabilitiesRequest;
import org.elasticsearch.action.get.GetRequest;
import org.elasticsearch.action.get.MultiGetRequest;
import org.elasticsearch.action.index.IndexRequest;
Expand Down Expand Up @@ -81,6 +83,7 @@
import org.elasticsearch.index.VersionType;
import org.elasticsearch.index.rankeval.RankEvalRequest;
import org.elasticsearch.rest.action.search.RestSearchAction;
import org.elasticsearch.script.mustache.SearchTemplateRequest;
import org.elasticsearch.search.fetch.subphase.FetchSourceContext;
import org.elasticsearch.tasks.TaskId;

Expand Down Expand Up @@ -469,6 +472,15 @@ static Request search(SearchRequest searchRequest) throws IOException {
Request request = new Request(HttpPost.METHOD_NAME, endpoint(searchRequest.indices(), searchRequest.types(), "_search"));

Params params = new Params(request);
addSearchRequestParams(params, searchRequest);

if (searchRequest.source() != null) {
request.setEntity(createEntity(searchRequest.source(), REQUEST_BODY_CONTENT_TYPE));
}
return request;
}

private static void addSearchRequestParams(Params params, SearchRequest searchRequest) {
params.putParam(RestSearchAction.TYPED_KEYS_PARAM, "true");
params.withRouting(searchRequest.routing());
params.withPreference(searchRequest.preference());
Expand All @@ -484,11 +496,6 @@ static Request search(SearchRequest searchRequest) throws IOException {
if (searchRequest.scroll() != null) {
params.putParam("scroll", searchRequest.scroll().keepAlive());
}

if (searchRequest.source() != null) {
request.setEntity(createEntity(searchRequest.source(), REQUEST_BODY_CONTENT_TYPE));
}
return request;
}

static Request searchScroll(SearchScrollRequest searchScrollRequest) throws IOException {
Expand Down Expand Up @@ -518,6 +525,24 @@ static Request multiSearch(MultiSearchRequest multiSearchRequest) throws IOExcep
return request;
}

static Request searchTemplate(SearchTemplateRequest searchTemplateRequest) throws IOException {
Request request;

if (searchTemplateRequest.isSimulate()) {
request = new Request(HttpGet.METHOD_NAME, "_render/template");
} else {
SearchRequest searchRequest = searchTemplateRequest.getRequest();
String endpoint = endpoint(searchRequest.indices(), searchRequest.types(), "_search/template");
request = new Request(HttpGet.METHOD_NAME, endpoint);

Params params = new Params(request);
addSearchRequestParams(params, searchRequest);
}

request.setEntity(createEntity(searchTemplateRequest, REQUEST_BODY_CONTENT_TYPE));
return request;
}

static Request existsAlias(GetAliasesRequest getAliasesRequest) {
if ((getAliasesRequest.indices() == null || getAliasesRequest.indices().length == 0) &&
(getAliasesRequest.aliases() == null || getAliasesRequest.aliases().length == 0)) {
Expand All @@ -534,6 +559,17 @@ static Request existsAlias(GetAliasesRequest getAliasesRequest) {
return request;
}

static Request fieldCaps(FieldCapabilitiesRequest fieldCapabilitiesRequest) {
String[] indices = fieldCapabilitiesRequest.indices();
Request request = new Request(HttpGet.METHOD_NAME, endpoint(indices, "_field_caps"));

Params params = new Params(request);
params.withFields(fieldCapabilitiesRequest.fields());
params.withIndicesOptions(fieldCapabilitiesRequest.indicesOptions());

return request;
}

static Request rankEval(RankEvalRequest rankEvalRequest) throws IOException {
Request request = new Request(HttpGet.METHOD_NAME, endpoint(rankEvalRequest.indices(), Strings.EMPTY_ARRAY, "_rank_eval"));

Expand Down Expand Up @@ -687,6 +723,16 @@ static Request createRepository(PutRepositoryRequest putRepositoryRequest) throw
return request;
}

static Request deleteRepository(DeleteRepositoryRequest deleteRepositoryRequest) {
String endpoint = new EndpointBuilder().addPathPartAsIs("_snapshot").addPathPart(deleteRepositoryRequest.name()).build();
Request request = new Request(HttpDelete.METHOD_NAME, endpoint);

Params parameters = new Params(request);
parameters.withMasterTimeout(deleteRepositoryRequest.masterNodeTimeout());
parameters.withTimeout(deleteRepositoryRequest.timeout());
return request;
}

static Request putTemplate(PutIndexTemplateRequest putIndexTemplateRequest) throws IOException {
String endpoint = new EndpointBuilder().addPathPartAsIs("_template").addPathPart(putIndexTemplateRequest.name()).build();
Request request = new Request(HttpPut.METHOD_NAME, endpoint);
Expand Down Expand Up @@ -795,6 +841,13 @@ Params withFetchSourceContext(FetchSourceContext fetchSourceContext) {
return this;
}

Params withFields(String[] fields) {
if (fields != null && fields.length > 0) {
return putParam("fields", String.join(",", fields));
}
return this;
}

Params withMasterTimeout(TimeValue masterTimeout) {
return putParam("master_timeout", masterTimeout);
}
Expand Down
68 changes: 64 additions & 4 deletions client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.action.admin.cluster.repositories.get.GetRepositoriesRequest;
import org.elasticsearch.action.admin.cluster.repositories.get.GetRepositoriesResponse;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.fieldcaps.FieldCapabilitiesRequest;
import org.elasticsearch.action.fieldcaps.FieldCapabilitiesResponse;
import org.elasticsearch.action.get.GetRequest;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.get.MultiGetRequest;
Expand Down Expand Up @@ -62,6 +62,8 @@
import org.elasticsearch.plugins.spi.NamedXContentProvider;
import org.elasticsearch.rest.BytesRestResponse;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.script.mustache.SearchTemplateRequest;
import org.elasticsearch.script.mustache.SearchTemplateResponse;
import org.elasticsearch.search.aggregations.Aggregation;
import org.elasticsearch.search.aggregations.bucket.adjacency.AdjacencyMatrixAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.adjacency.ParsedAdjacencyMatrix;
Expand Down Expand Up @@ -499,6 +501,32 @@ public final void clearScrollAsync(ClearScrollRequest clearScrollRequest,
listener, emptySet(), headers);
}

/**
* Executes a request using the Search Template API.
*
* See <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html">Search Template API
* on elastic.co</a>.
*/
public final SearchTemplateResponse searchTemplate(SearchTemplateRequest searchTemplateRequest,
Header... headers) throws IOException {
return performRequestAndParseEntity(searchTemplateRequest, RequestConverters::searchTemplate,
SearchTemplateResponse::fromXContent, emptySet(), headers);
}

/**
* Asynchronously executes a request using the Search Template API
*
* See <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html">Search Template API
* on elastic.co</a>.
*/
public final void searchTemplateAsync(SearchTemplateRequest searchTemplateRequest,
ActionListener<SearchTemplateResponse> listener,
Header... headers) {
performRequestAsyncAndParseEntity(searchTemplateRequest, RequestConverters::searchTemplate,
SearchTemplateResponse::fromXContent, listener, emptySet(), headers);
}


/**
* Executes a request using the Ranking Evaluation API.
*
Expand All @@ -521,6 +549,31 @@ public final void rankEvalAsync(RankEvalRequest rankEvalRequest, ActionListener<
emptySet(), headers);
}

/**
* Executes a request using the Field Capabilities API.
*
* See <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html">Field Capabilities API
* on elastic.co</a>.
*/
public final FieldCapabilitiesResponse fieldCaps(FieldCapabilitiesRequest fieldCapabilitiesRequest,
Header... headers) throws IOException {
return performRequestAndParseEntity(fieldCapabilitiesRequest, RequestConverters::fieldCaps,
FieldCapabilitiesResponse::fromXContent, emptySet(), headers);
}

/**
* Asynchronously executes a request using the Field Capabilities API.
*
* See <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html">Field Capabilities API
* on elastic.co</a>.
*/
public final void fieldCapsAsync(FieldCapabilitiesRequest fieldCapabilitiesRequest,
ActionListener<FieldCapabilitiesResponse> listener,
Header... headers) {
performRequestAsyncAndParseEntity(fieldCapabilitiesRequest, RequestConverters::fieldCaps,
FieldCapabilitiesResponse::fromXContent, listener, emptySet(), headers);
}

protected final <Req extends ActionRequest, Resp> Resp performRequestAndParseEntity(Req request,
CheckedFunction<Req, Request, IOException> requestConverter,
CheckedFunction<XContentParser, Resp, IOException> entityParser,
Expand All @@ -537,7 +590,7 @@ protected final <Req extends ActionRequest, Resp> Resp performRequest(Req reques
throw validationException;
}
Request req = requestConverter.apply(request);
req.setHeaders(headers);
addHeaders(req, headers);
Response response;
try {
response = client.performRequest(req);
Expand Down Expand Up @@ -587,12 +640,19 @@ protected final <Req extends ActionRequest, Resp> void performRequestAsync(Req r
listener.onFailure(e);
return;
}
req.setHeaders(headers);
addHeaders(req, headers);

ResponseListener responseListener = wrapResponseListener(responseConverter, listener, ignores);
client.performRequestAsync(req, responseListener);
}

private static void addHeaders(Request request, Header... headers) {
Objects.requireNonNull(headers, "headers cannot be null");
for (Header header : headers) {
request.addHeader(header.getName(), header.getValue());
}
}

final <Resp> ResponseListener wrapResponseListener(CheckedFunction<Response, Resp, IOException> responseConverter,
ActionListener<Resp> actionListener, Set<Integer> ignores) {
return new ResponseListener() {
Expand Down
Loading

0 comments on commit 2aa0570

Please sign in to comment.