Skip to content

Commit

Permalink
Merge branch 'main' into remote_connect
Browse files Browse the repository at this point in the history
Signed-off-by: Ankit Kala <ankikala@amazon.com>
  • Loading branch information
ankitkala committed Jun 13, 2023
2 parents 91fe701 + 4b4d84e commit 5ef6976
Show file tree
Hide file tree
Showing 111 changed files with 3,754 additions and 161 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @reta @anasalkouz @andrross @reta @Bukhtawar @CEHENKLE @dblock @gbbafna @setiah @kartg @kotwanikunal @mch2 @nknize @owaiskazi19 @Rishikesh1159 @ryanbogan @saratvemulapalli @shwetathareja @dreamer-89 @tlfeng @VachaShah @dbwiddis
* @reta @anasalkouz @andrross @reta @Bukhtawar @CEHENKLE @dblock @gbbafna @setiah @kartg @kotwanikunal @mch2 @nknize @owaiskazi19 @Rishikesh1159 @ryanbogan @saratvemulapalli @shwetathareja @dreamer-89 @tlfeng @VachaShah @dbwiddis @sachinpkale
10 changes: 9 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add events correlation engine plugin ([#6854](https://github.com/opensearch-project/OpenSearch/issues/6854))
- Add support for ignoring missing Javadoc on generated code using annotation ([#7604](https://github.com/opensearch-project/OpenSearch/pull/7604))
- Implement concurrent aggregations support without profile option ([#7514](https://github.com/opensearch-project/OpenSearch/pull/7514))
- Add dynamic index and cluster setting for concurrent segment search ([#7956](https://github.com/opensearch-project/OpenSearch/pull/7956))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand Down Expand Up @@ -46,6 +47,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add `com.github.luben:zstd-jni` version 1.5.5-3 ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996))
- OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344)
- Bump `com.amazonaws` 1.12.270 to `software.amazon.awssdk` 2.20.55 ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7933](https://github.com/opensearch-project/OpenSearch/pull/7933/))
- Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](https://github.com/opensearch-project/OpenSearch/pull/7928)))
- Bump `com.azure:azure-storage-blob` from 12.21.1 to 12.22.2 ([7930](https://github.com/opensearch-project/OpenSearch/pull/7930))

Expand All @@ -60,6 +62,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add ZSTD compression for snapshotting ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996))
- Change `com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` to `aws.ec2MetadataServiceEndpoint` ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Change `com.amazonaws.sdk.stsEndpointOverride` to `aws.stsEndpointOverride` ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Align range and default value for deletes_pct_allowed in merge policy ([#7730](https://github.com/opensearch-project/OpenSearch/pull/7730))

### Deprecated

Expand Down Expand Up @@ -90,13 +93,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

## [Unreleased 2.x]
### Added
- Add task cancellation monitoring service ([#7642](https://github.com/opensearch-project/OpenSearch/pull/7642))
- Add TokenManager Interface ([#7452](https://github.com/opensearch-project/OpenSearch/pull/7452))
- Add Remote store as a segment replication source ([#7653](https://github.com/opensearch-project/OpenSearch/pull/7653))
- Add descending order search optimization through reverse segment read. ([#7967](https://github.com/opensearch-project/OpenSearch/pull/7967))
- Make remote cluster connection setup in async ([#8038](https://github.com/opensearch-project/OpenSearch/pull/8038))

### Dependencies
- Bump `jackson` from 2.15.1 to 2.15.2 ([#7897](https://github.com/opensearch-project/OpenSearch/pull/7897))
- Bump `netty` from 4.1.91.Final to 4.1.93.Final ([#7901](https://github.com/opensearch-project/OpenSearch/pull/7901))
- Bump `com.netflix.nebula:gradle-extra-configurations-plugin` from 9.0.0 to 10.0.0 in /buildSrc ([#7068](https://github.com/opensearch-project/OpenSearch/pull/7068))
- Bump `com.google.guava:guava` from 32.0.0-jre to 32.0.1-jre (#8009)
- Bump `commons-io:commons-io` from 2.12.0 to 2.13.0 (#8014, #8013)

### Changed
- Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](https://github.com/opensearch-project/OpenSearch/pull/7836))
Expand All @@ -115,4 +123,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.8...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.8...2.x
1 change: 1 addition & 0 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
| Owais Kazi | [owaiskazi19](https://github.com/owaiskazi19) | Amazon |
| Rishikesh Pasham | [Rishikesh1159](https://github.com/Rishikesh1159) | Amazon |
| Ryan Bogan | [ryanbogan](https://github.com/ryanbogan) | Amazon |
| Sachin Kale | [sachinpkale](https://github.com/sachinpkale) | Amazon |
| Sarat Vemulapalli | [saratvemulapalli](https://github.com/saratvemulapalli) | Amazon |
| Shweta Thareja | [shwetathareja](https://github.com/shwetathareja) | Amazon |
| Suraj Singh | [dreamer-89](https://github.com/dreamer-89) | Amazon |
Expand Down
8 changes: 5 additions & 3 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,11 @@ dependencies {
api 'commons-codec:commons-codec:1.15'
api 'org.apache.commons:commons-compress:1.23.0'
api 'org.apache.ant:ant:1.10.13'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:9.0.0'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:10.0.0'
api 'com.netflix.nebula:nebula-publishing-plugin:20.3.0'
api 'com.netflix.nebula:gradle-info-plugin:12.1.3'
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.11.0'
api 'commons-io:commons-io:2.13.0'
api "net.java.dev.jna:jna:5.13.0"
api 'gradle.plugin.com.github.johnrengelman:shadow:8.0.0'
api 'org.jdom:jdom2:2.0.6.1'
Expand All @@ -118,7 +118,9 @@ dependencies {
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.2'
api 'com.networknt:json-schema-validator:1.0.81'
api 'com.networknt:json-schema-validator:1.0.83'
api 'org.jruby.jcodings:jcodings:1.0.58'
api 'org.jruby.joni:joni:2.1.48'
api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"

testFixturesApi "junit:junit:${props.getProperty('junit')}"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
*
* Modifications Copyright OpenSearch Contributors. See
* GitHub history for details.
*/

/*
* Copyright 2014-2016 Netflix, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opensearch.gradle.plugin

import org.gradle.api.Plugin
import org.gradle.api.Project
import org.gradle.api.publish.PublishingExtension
import org.gradle.api.publish.ivy.IvyPublication
import org.gradle.api.publish.maven.MavenPublication
import org.gradle.api.publish.plugins.PublishingPlugin

/**
* Including the support of `optional` dependencies: https://github.com/nebula-plugins/gradle-extra-configurations-plugin/blob/v9.0.0/src/main/groovy/nebula/plugin/extraconfigurations/OptionalBasePlugin.groovy
*/
class OptionalDependenciesPlugin implements Plugin<Project> {
static final String OPTIONAL_IDENTIFIER = 'optional'

@Override
void apply(Project project) {
enhanceProjectModel(project)
configureMavenPublishPlugin(project)
configureIvyPublishPlugin(project)
}

/**
* Enhances the Project domain object by adding
*
* a) a extra property List that holds optional dependencies
* b) a extra method that can be executed as parameter when declaring dependencies
*
* @param project Project
*/
private void enhanceProjectModel(Project project) {
project.ext.optionalDeps = []

project.ext.optional = { dep ->
project.ext.optionalDeps << dep
}
}

/**
* Configures Maven Publishing plugin to ensure that published dependencies receive the optional element.
*
* @param project Project
*/
private void configureMavenPublishPlugin(Project project) {
project.plugins.withType(PublishingPlugin) {
project.publishing {
publications {
project.extensions.findByType(PublishingExtension)?.publications?.withType(MavenPublication) { MavenPublication pub ->
pub.pom.withXml {
project.ext.optionalDeps.each { dep ->
def foundDep = asNode().dependencies.dependency.find {
it.groupId.text() == dep.group && it.artifactId.text() == dep.name
}

if (foundDep) {
if (foundDep.optional) {
foundDep.optional.value = 'true'
} else {
foundDep.appendNode(OPTIONAL_IDENTIFIER, 'true')
}
}
}
}
}
}
}
}
}

/**
* Configures Ivy Publishing plugin to ensure that published dependencies receive the correct conf attribute value.
*
* @param project Project
*/
private void configureIvyPublishPlugin(Project project) {
project.plugins.withType(PublishingPlugin) {
project.publishing {
publications {
project.extensions.findByType(PublishingExtension)?.publications?.withType(IvyPublication) { IvyPublication pub ->
pub.descriptor.withXml {
def rootNode = asNode()

// Add optional configuration if it doesn't exist yet
if (!rootNode.configurations.find { it.@name == OPTIONAL_IDENTIFIER }) {
rootNode.configurations[0].appendNode('conf', [name: OPTIONAL_IDENTIFIER, visibility: 'public'])
}

// Replace dependency "runtime->default" conf attribute value with "optional"
project.ext.optionalDeps.each { dep ->
def foundDep = rootNode.dependencies.dependency.find { it.@name == dep.name }
foundDep?.@conf = OPTIONAL_IDENTIFIER
}
}
}
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ public void validate(InputChanges inputChanges) throws IOException {
File jsonSchemaOnDisk = getJsonSchema();
getLogger().debug("JSON schema : [{}]", jsonSchemaOnDisk.getAbsolutePath());
SchemaValidatorsConfig config = new SchemaValidatorsConfig();
config.setEcma262Validator(true);
JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7);
JsonSchema jsonSchema = factory.getSchema(mapper.readTree(jsonSchemaOnDisk), config);
Map<File, Set<String>> errors = new LinkedHashMap<>();
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#
# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.
#
# Modifications Copyright OpenSearch Contributors. See
# GitHub history for details.
#

implementation-class=org.opensearch.gradle.plugin.OptionalDependenciesPlugin
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.gradle.plugin;

import org.apache.maven.model.Dependency;
import org.apache.maven.model.Model;
import org.apache.maven.model.io.xpp3.MavenXpp3Reader;
import org.codehaus.plexus.util.xml.pull.XmlPullParserException;
import org.gradle.testkit.runner.BuildResult;
import org.gradle.testkit.runner.GradleRunner;
import org.junit.After;
import org.junit.Before;
import org.junit.rules.TemporaryFolder;
import org.opensearch.gradle.test.GradleUnitTestCase;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Optional;

import static org.gradle.testkit.runner.TaskOutcome.SUCCESS;
import static org.hamcrest.CoreMatchers.is;

public class OptionalDependenciesPluginTests extends GradleUnitTestCase {
private TemporaryFolder projectDir;

@Before
public void setUp() throws Exception {
projectDir = new TemporaryFolder();
projectDir.create();
}

@After
public void tearDown() {
projectDir.delete();
}

public void testApply() throws FileNotFoundException, IOException, XmlPullParserException {
final File mavenRepoDir = new File(projectDir.getRoot(), "mavenrepo");

try (InputStream in = getClass().getClassLoader().getResourceAsStream("plugin/optional-dependencies.gradle")) {
try (OutputStream out = new FileOutputStream(projectDir.newFile("build.gradle"))) {
in.transferTo(out);
}
}

GradleRunner runner = GradleRunner.create()
.forwardOutput()
.withPluginClasspath()
.withArguments("publish", "-PrepoUrl=" + mavenRepoDir.toURI().toURL())
.withProjectDir(projectDir.getRoot());

BuildResult result = runner.build();
assertEquals(SUCCESS, result.task(":" + "publish").getOutcome());

final String name = projectDir.getRoot().getName();
assertDependency(mavenRepoDir, name);
}

private void assertDependency(File repoUrl, String name) throws FileNotFoundException, IOException, XmlPullParserException {
final File pom = new File(repoUrl, "org/custom/group/" + name + "/1.0.0/" + name + "-1.0.0.pom");
assertThat(pom.exists(), is(true));

final MavenXpp3Reader reader = new MavenXpp3Reader();
final Model model = reader.read(new FileReader(pom));

final Optional<Dependency> dependecyOpt = model.getDependencies()
.stream()
.filter(d -> d.getArtifactId().equals("commons-lang3"))
.findAny();

assertThat(dependecyOpt.isPresent(), is(true));
assertThat(dependecyOpt.get().isOptional(), is(true));
}
}
39 changes: 39 additions & 0 deletions buildSrc/src/test/resources/plugin/optional-dependencies.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*
* Modifications Copyright OpenSearch Contributors. See
* GitHub history for details.
*/

plugins {
id 'java'
id 'maven-publish'
id 'opensearch.optional-dependencies'
}
group = "org.custom.group"
version = '1.0.0'

repositories {
mavenCentral()
}

dependencies {
implementation 'org.apache.commons:commons-lang3:3.3.2', optional
}

publishing {
publications {
mavenJava(MavenPublication) {
from components.java
}
}
repositories {
maven {
url "${repoUrl}"
}
}
}
2 changes: 2 additions & 0 deletions buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,5 @@ jmh = 1.35

# compression
zstd = 1.5.5-3

jzlib = 1.1.3
7 changes: 7 additions & 0 deletions distribution/src/config/opensearch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,4 +130,11 @@ ${path.logs}
#
# Gates the search pipeline feature. This feature enables configurable processors
# for search requests and search responses, similar to ingest pipelines.
#
#opensearch.experimental.feature.search_pipeline.enabled: false
#
#
# Gates the concurrent segment search feature. This feature enables concurrent segment search in a separate
# index searcher threadpool.
#
#opensearch.experimental.feature.concurrent_segment_search.enabled: false
2 changes: 1 addition & 1 deletion distribution/tools/keystore-cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,5 @@ dependencies {
compileOnly project(":libs:opensearch-cli")
testImplementation project(":test:framework")
testImplementation 'com.google.jimfs:jimfs:1.2'
testRuntimeOnly 'com.google.guava:guava:32.0.0-jre'
testRuntimeOnly 'com.google.guava:guava:32.0.1-jre'
}
1 change: 0 additions & 1 deletion libs/cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
* under the License.
*/
apply plugin: 'opensearch.build'
apply plugin: 'com.netflix.nebula.optional-base'
apply plugin: 'opensearch.publish'

dependencies {
Expand Down
1 change: 0 additions & 1 deletion libs/core/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@

import org.opensearch.gradle.info.BuildParams

apply plugin: 'com.netflix.nebula.optional-base'
apply plugin: 'opensearch.publish'

archivesBaseName = 'opensearch-core'
Expand Down
Loading

0 comments on commit 5ef6976

Please sign in to comment.