Add support of SOCKS proxies for S3 repository #2160

willyborankin · 2022-02-17T21:47:08Z

Description

Add support of SOCKS proxies for S3 repository.

Instead of using s3.client.protocol as a proxy type a new property was added:

s3.client.proxy.type - which maps directly to the official AWS proxy protocol setting.
And support HTTP, HTTPS and SOCKS proxies.

This code is almost the same as for others PRs related to PROXY support and needs to be move to the common place/part.

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-02-17T21:48:03Z

Can one of the admins verify this patch?

willyborankin · 2022-02-17T21:50:07Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/ProxySettings.java

+import java.net.UnknownHostException;
+import java.util.Objects;
+
+public class ProxySettings {


This is the same solution as for others repositories. Can be extracted to a common part

Tracked in #2155

willyborankin · 2022-02-17T21:50:51Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java

+        );
+    }
+
+    static ProxySettings validateAndCreateProxySettings(final Settings settings, final String clientName, final Protocol awsProtocol) {


willyborankin · 2022-02-17T22:02:31Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3Service.java

@@ -208,6 +242,20 @@ static ClientConfiguration buildConfiguration(S3ClientSettings clientSettings) {
        return clientConfiguration;
    }

+    private static SSLConnectionSocketFactory createSocksSslConnectionSocketFactory(final InetSocketAddress address) {


this is exactly the same as how AWS SDK creates SSLConnectionSocketFactory. Since the plugin does not use additional settings which could affect its settings. IMHO it is more or less ok.

opensearch-ci-bot · 2022-02-17T22:25:01Z

✅ Gradle Check success c6f4902e490d6db2e8a35b898fec7bde3610d695
Log 2525

Reports 2525

opensearch-ci-bot · 2022-02-17T23:11:33Z

❌ Gradle Check failure 4745a5d745d569c986ab86713d7f05ac39f53762
Log 2530

Reports 2530

opensearch-ci-bot · 2022-02-21T10:03:24Z

✅ Gradle Check success 6178e331476bded1f46a6c54ad8e6dbf7b945931
Log 2632

Reports 2632

reta · 2022-02-21T19:06:27Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/ProxySettings.java

+                // the probability that it could be thrown is small, but how knows
+                throw new SettingsException("Couldn't convert to S3 protocol");
+            } else if (this == SOCKS) {
+                throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supportedS3");


Suggested change

throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supportedS3");

throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supported by S3");

This code convert ProxyType to AWS Protocol objects, whic does not support SOCKS. SOCKS implementation was done by extending SSLSocketFactory

Fix the typo in the message? supportedS3

reta · 2022-02-21T19:07:49Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/ProxySettings.java

+    public static enum ProxyType {
+        HTTP(Protocol.HTTP.name()),
+        HTTPS(Protocol.HTTPS.name()),
+        SOCKS("SOCKS"),


Do we actually need to introduce SOCKS proxy type? Shouldn't we treat it as any other unsupported value?

It is possible to support SOCKS. Socks woks on the Socket level. So extending SSLSocketFactory is ok IMHO

Nevermind, I got that, conversion to protocol is not supported for SOCKS

reta · 2022-02-21T19:08:47Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java

@@ -54,6 +60,8 @@
 */
 final class S3ClientSettings {

+    static final Logger LOGGER = LogManager.getLogger(S3ClientSettings.class);


private ?

reta · 2022-02-21T19:10:35Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java

@@ -106,7 +121,7 @@
    static final Setting.AffixSetting<Integer> PROXY_PORT_SETTING = Setting.affixKeySetting(
        PREFIX,
        "proxy.port",
-        key -> Setting.intSetting(key, 80, 0, 1 << 16, Property.NodeScope)
+        key -> Setting.intSetting(key, 0, 0, 1 << 16, Property.NodeScope)


Changing defaults will break existing deployment (if anyone relies on default value)

Very good point

reta · 2022-02-21T19:13:38Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java

+            // In this case, proxy type should be the same as a protocol,
+            // when proxy host and port have been set
+            proxyType = ProxySettings.ProxyType.valueOf(awsProtocol.name());
+            LOGGER.warn(


You may consider switching to DeprecationLogger instead:

private static final DeprecationLogger LOGGER = DeprecationLogger.getLogger(S3ClientSettings.class);

reta · 2022-02-21T19:17:45Z

plugins/repository-s3/src/test/java/org/opensearch/repositories/s3/AwsS3ServiceImplTests.java

 import java.util.Locale;
 import java.util.Map;

 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.hamcrest.Matchers.is;

+import static org.mockito.Mockito.*;


I think * imports are not desired

hm interesting why checkstyle passed. will fix.

reta · 2022-02-21T21:27:57Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3Service.java

+                        });
+                    }
+                    clientConfiguration.getApacheHttpClientConfig()
+                        .setSslSocketFactory(createSocksSslConnectionSocketFactory(clientSettings.proxySettings.getAddress()));


Discussed that with @willyborankin offline: the SslSocketFactory is only used for HTTPS connections, otherwise ApacheHttpClient will use SdkPlainSocketFactory and the trick with opening new Socket(new Proxy(Proxy.Type.SOCKS, address)) would not work. It is probably make sense to throw an exception in this case (for non-HTTPS endpoints)

opensearch-ci-bot · 2022-02-21T21:38:37Z

✅ Gradle Check success 70a29c005271095df97f27571b6592f46c0307cd
Log 2664

Reports 2664

opensearch-ci-bot · 2022-02-22T13:50:57Z

✅ Gradle Check success bdbafb55ee1a6fa9d44301f3286d77ebb0cd6c40
Log 2697

Reports 2697

dblock

Fix the typo and we're good to go?

dblock · 2022-03-01T15:38:58Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/ProxySettings.java

+                // the probability that it could be thrown is small, but how knows
+                throw new SettingsException("Couldn't convert to S3 protocol");
+            } else if (this == SOCKS) {
+                throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supportedS3");


Fix the typo in the message? supportedS3

Signed-off-by: Andrey Pleskach <ples@aiven.io>

opensearch-ci-bot · 2022-03-02T11:54:24Z

❌ Gradle Check failure f5ea951
Log 2926

Reports 2926

kartg · 2022-03-02T18:03:14Z

#1565

Refiring

kartg · 2022-03-02T18:03:35Z

start gradle check

opensearch-ci-bot · 2022-03-02T18:47:43Z

✅ Gradle Check success f5ea951
Log 2932

Reports 2932

Signed-off-by: Andrey Pleskach <ples@aiven.io> (cherry picked from commit f13b951)

Signed-off-by: Andrey Pleskach <ples@aiven.io> (cherry picked from commit f13b951) Co-authored-by: Andrey Pleskach <ples@aiven.io>

This includes the following: commit 9cfa395 Author: Kartik <gkart@amazon.com> Date: Thu Mar 10 10:12:17 2022 -0800 Remove the IndexCommitRef class (opensearch-project#2421) This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> commit c8d8009 Author: Andriy Redko <andriy.redko@aiven.io> Date: Thu Mar 10 11:46:08 2022 -0500 Fixing bwcVersions and bwc builds (opensearch-project#2430) - adding 1.4.0 into main bwcVersions Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit fb9e150 Author: Kartik <85275476+kartg@users.noreply.github.com> Date: Wed Mar 9 12:21:09 2022 -0800 Refactoring gated and ref-counted interfaces and their implementations (opensearch-project#2396) * Reducing duplication in plugins around ref-counted releasable classes Both AmazonEc2Reference and AmazonS3Reference duplicate the same logic - a subclass of AbstractRefCounted that also implements Releasable. This change centralizes this paradigm into a AbstractRefCountedReleasable class and supports both clients via generics. It also updates all fetching implementations to use the get() method instead of client() Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Introduce Reference classes for the Closeable and AutoCloseable interfaces These classes allow you to wrap a reference instance with an onClose runnable that is executed when close() is invoked. Two separate classes are needed because the close() signatures for the two interfaces are different. This change takes the first step to have implementing classes extend from these generic superclasses, before attempting to remove the subclasses entirely. The get() method is also replaced throughout the code base. Note that there is also a separate Releasable interface that has a similar access pattern, but is implemented separately. This is used in AbstractRefCountedReleasable introduced in a prior commit Signed-off-by: Kartik Ganesh <gkart@amazon.com> * More improvements and refactoring * Functionality around one-way gating is now moved to a dedicated class - OneWayGate. This replaces duplicate functionality throughout the code. * The two *Reference classes have been renamed to Gated* since that better represents their functionality * The AbstractRefCountedReleasable has been improved to no longer be abstract by accepting the shutdown hook. This removes the need for the inner class in ReleasableBytesReference, and further simplifies the plugin subclasses (these could probably be removed entirely). * Finally, unit tests have been added for some classes Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added tests for GatedCloseable Also updated the license information in GatedAutoCloseableTests Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixing license information in new files Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added unit tests for RefCountedReleasable Signed-off-by: Kartik Ganesh <gkart@amazon.com> commit 5a9a114 Author: Nick Knize <nknize@apache.org> Date: Wed Mar 9 12:50:05 2022 -0600 [Remove] TrimUnsafeCommit logic for legacy 6.x indexes (opensearch-project#2225) * [Remove] TrimUnsafeCommit logic for legacy 6.x indexes Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible with legacy 6.x indexes, the logic to trim these unsafe commits is safely removed. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix assertion typo Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * rebase and incorporate pr feedback Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 9c679cb Author: Andriy Redko <andriy.redko@aiven.io> Date: Tue Mar 8 18:42:32 2022 -0500 MapperService has to be passed in as null for EnginePlugins CodecService constructor (opensearch-project#2177) * MapperService has to be passed in as null for EnginePlugins CodecService constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Delayed CodecService instantiation up to the shard initialization Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Added logger (associated with shard) to CodecServiceConfig Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Refactored the EngineConfigFactory / IndexShard instantiation of the CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit a6a47e7 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Tue Mar 8 14:43:04 2022 -0800 Remove inclue_type_name parameter from rest api spec (opensearch-project#2410) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 044f536 Author: Daniel Doubrovkine (dB.) <dblock@dblock.org> Date: Tue Mar 8 14:48:51 2022 -0500 Set target and source compatibility to 11, required by Lucene 9. (opensearch-project#2407) * Set target and source compatibility to 11, required by Lucene 9. Signed-off-by: dblock <dblock@dblock.org> * Uncomment commented code in opensearch-project#2321 for killing child processes that uses JDK9+ ProcessInfo. Signed-off-by: dblock <dblock@dblock.org> * Set distribution checker target JDK compatibility to 11. Signed-off-by: dblock <dblock@dblock.org> * Supress processing warnings. Signed-off-by: dblock <dblock@dblock.org> commit c3712a5 Author: Nick Knize <nknize@apache.org> Date: Tue Mar 8 11:30:27 2022 -0600 [Remove] include_type_name from HLRC (opensearch-project#2397) Removes include_type_name from the high level reset client along with relevant deprecated methods in IndicesClient. All tests are updated to remove the parameter from the rest requests along with various toXContent methods that are no longer required. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 63c75d1 Author: Tianli Feng <ftianli@amazon.com> Date: Tue Mar 8 08:35:36 2022 -0800 Deprecate setting 'reindex.remote.whitelist' and introduce the alternative setting 'reindex.remote.allowlist' (opensearch-project#2221) * Add setting reindex.remote.allowlist, and deprecate setting reindex.remote.whitelist Signed-off-by: Tianli Feng <ftianli@amazon.com> * Add unit test for renaming the setting reindex.remote.allowlist Signed-off-by: Tianli Feng <ftianli@amazon.com> * Remove system.out.println() Signed-off-by: Tianli Feng <ftianli@amazon.com> * Adjust format by spotlessApply task Signed-off-by: Tianli Feng <ftianli@amazon.com> * Replace REMOTE_CLUSTER_WHITELIST with REMOTE_CLUSTER_ALLOWLIST Signed-off-by: Tianli Feng <ftianli@amazon.com> * Add a unit test to test final setting value when both settings have got a value Signed-off-by: Tianli Feng <ftianli@amazon.com> * Rename the unit test class name Signed-off-by: Tianli Feng <ftianli@amazon.com> * Remove the Access modifiers public from the constant REMOTE_CLUSTER_WHITELIST Signed-off-by: Tianli Feng <ftianli@amazon.com> * Initialize ReindexPlugin without using the @before method Signed-off-by: Tianli Feng <ftianli@amazon.com> * Rename 'unwhitelisted' to 'unallowlisted' in a yml file used for REST api testing. Signed-off-by: Tianli Feng <ftianli@amazon.com> commit 65debde Author: Andriy Redko <andriy.redko@aiven.io> Date: Tue Mar 8 11:30:48 2022 -0500 Update the BWC versions (post 1.x backport) (opensearch-project#2390) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit 919d180 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Mon Mar 7 12:43:05 2022 -0800 Remove type end-points from count action (opensearch-project#2379) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 1f0361a Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 10:28:17 2022 -0800 Bump asm-commons from 5.0.4 to 9.2 in /modules/lang-expression (opensearch-project#2385) * Bump asm-commons from 5.0.4 to 9.2 in /modules/lang-expression Bumps asm-commons from 5.0.4 to 9.2. --- updated-dependencies: - dependency-name: org.ow2.asm:asm-commons dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 09e16e3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 13:08:07 2022 -0500 Bump guava from 30.1.1-jre to 31.1-jre in /plugins/repository-azure (opensearch-project#2382) * Bump guava from 30.1.1-jre to 31.1-jre in /plugins/repository-azure Bumps [guava](https://github.com/google/guava) from 30.1.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit e1fd4b7 Author: Subhobrata Dey <sbcd90@gmail.com> Date: Mon Mar 7 08:51:49 2022 -0800 Add valuesField in PercentilesAggregationBuilder streamInput constructor (opensearch-project#2308) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> commit 4395ed5 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:38 2022 -0500 Bump guava in /distribution/tools/upgrade-cli (opensearch-project#2383) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 72c5d81 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:18 2022 -0500 Bump guava in /distribution/tools/keystore-cli (opensearch-project#2384) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 75e837d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:00 2022 -0500 Bump guava from 31.0.1-jre to 31.1-jre in /distribution/tools/plugin-cli (opensearch-project#2387) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 3e9031f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:47:37 2022 -0500 Bump gradle-extra-configurations-plugin from 3.0.3 to 7.0.0 in /buildSrc (opensearch-project#2386) Bumps [gradle-extra-configurations-plugin](https://github.com/nebula-plugins/gradle-extra-configurations-plugin) from 3.0.3 to 7.0.0. - [Release notes](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/releases) - [Changelog](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/blob/main/CHANGELOG.md) - [Commits](nebula-plugins/gradle-extra-configurations-plugin@v3.0.3...v7.0.0) --- updated-dependencies: - dependency-name: com.netflix.nebula:gradle-extra-configurations-plugin dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 9224537 Author: Owais Kazi <owaiskazi19@gmail.com> Date: Fri Mar 4 16:58:13 2022 -0800 Updated the url for docker distribution (opensearch-project#2325) Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> commit d5e58a2 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Mar 4 16:40:47 2022 -0800 [Remove] Type mappings from GeoShapeQueryBuilder (opensearch-project#2322) * Remove type end-points from GeoShapeBuilder Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix integration test failures Signed-off-by: Suraj Singh <surajrider@gmail.com> commit be64af2 Author: aponb <aponb@gmx.at> Date: Sat Mar 5 01:34:09 2022 +0100 Replace exclusionary words whitelist and blacklist in the places that won't impact backwards compatibility (opensearch-project#2178) * Replace the exclusionary word whitelist with allowlist, and blacklist with denylist, in code commet and internal variable/method/class/package name. Signed-off-by: Andreas <apre@gmx.at> commit ae52008 Author: Andriy Redko <andriy.redko@aiven.io> Date: Fri Mar 4 17:44:52 2022 -0500 Fixing the --release flag usage for javac (opensearch-project#2343) (opensearch-project#2352) * Fixing the --release flag usage for javac (opensearch-project#2343) * Fixing the --release flag usage for javac Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fixing the --html5 flag usage for javadoc Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fix java-version-checker source/target compatibility settings (opensearch-project#2354) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit 0cc2c9b Author: Nick Knize <nknize@apache.org> Date: Fri Mar 4 13:30:43 2022 -0600 [Remove] types from PutMappingRequest (opensearch-project#2335) Remove type support from putMappingRequest, dependencies, and all tests. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 729bc43 Author: Nick Knize <nknize@apache.org> Date: Fri Mar 4 10:52:24 2022 -0600 [Test-Failure] Mute TranslogPolicyIT (opensearch-project#2342) This test is slated for removal as it only applies to ancient indexes (Legacy 6.x). Muting test so bwc tests are consistent and no longer angry. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 5f90227 Author: Andriy Redko <andriy.redko@aiven.io> Date: Fri Mar 4 11:12:27 2022 -0500 Add '_name' field support to score functions and provide it back in explanation response (opensearch-project#2244) * Add '_name' field support to score functions and provide it back in explanation response Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Address code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit ae14259 Author: Daniel Doubrovkine (dB.) <dblock@dblock.org> Date: Thu Mar 3 15:34:53 2022 -0500 Restore Java 8 compatibility for build tools. (opensearch-project#2300) (opensearch-project#2321) * Restore Java 8 compatibility for build tools. Signed-off-by: dblock <dblock@dblock.org> * Make source code compatible with Java 8. Signed-off-by: dblock <dblock@dblock.org> commit cb57b92 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Mar 3 14:23:47 2022 -0500 Bump log4j-core in /buildSrc/src/testKit/thirdPartyAudit/sample_jars (opensearch-project#2281) Bumps log4j-core from 2.17.1 to 2.17.2. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit cdb42ad Author: Vacha Shah <vachshah@amazon.com> Date: Thu Mar 3 10:39:27 2022 -0800 Remove Github DCO action since DCO runs via Github App now (opensearch-project#2317) Signed-off-by: Vacha Shah <vachshah@amazon.com> commit f13b951 Author: Andrey Pleskach <ples@aiven.io> Date: Wed Mar 2 23:36:09 2022 +0100 Add support of SOCKS proxies for S3 repository (opensearch-project#2160) Signed-off-by: Andrey Pleskach <ples@aiven.io> commit 3d5aff4 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Wed Mar 2 13:21:00 2022 -0800 Remove type end-points from search and related APIs (opensearch-project#2263) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 897f4e7 Author: Nick Knize <nknize@apache.org> Date: Wed Mar 2 13:44:04 2022 -0600 [Remove] deprecated getMapping API from IndicesClient (opensearch-project#2262) Removes the deprecated types based get, getMapping, getAsync, and getMappingAsync methods from IndicesClient. It also removes extra nesting of mappings belong the deprecated type named object and removes the types based methods from the affected request classes. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 4b89410 Author: Breno Faria <30877433+br3no@users.noreply.github.com> Date: Wed Mar 2 19:52:38 2022 +0100 Reintroduce negative epoch_millis opensearch-project#1991 (opensearch-project#2232) * Reintroduce negative epoch_millis opensearch-project#1991 Fixes a regression introduced with Elasticsearch 7 regarding the date field type that removed support for negative timestamps with sub-second granularity. Thanks to Ryan Kophs (https://github.com/rkophs) for allowing me to use his previous work. Signed-off-by: Breno Faria <breno.faria@intrafind.de> * applying spotless fix Signed-off-by: Breno Faria <breno.faria@intrafind.de> * more conservative implementation of isSupportedBy Signed-off-by: Breno Faria <breno.faria@intrafind.de> * adding braces to control flow statement Signed-off-by: Breno Faria <breno.faria@intrafind.de> * spotless fix... Signed-off-by: Breno Faria <breno.faria@intrafind.de> Co-authored-by: Breno Faria <breno.faria@intrafind.de> commit 9e225dc Author: Peng Huo <penghuo@gmail.com> Date: Wed Mar 2 10:34:02 2022 -0800 Fix flaky test case - string profiler via global ordinals (opensearch-project#2226) forcemerge to one segment before executing aggregation query. Signed-off-by: Peng Huo <penghuo@gmail.com> commit c8a7606 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:18:28 2022 -0800 Bump commons-math3 from 3.2 to 3.6.1 in /benchmarks (opensearch-project#2282) Bumps commons-math3 from 3.2 to 3.6.1. --- updated-dependencies: - dependency-name: org.apache.commons:commons-math3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 1b8181c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:17:16 2022 -0800 Bump gson from 2.8.9 to 2.9.0 in /plugins/repository-hdfs (opensearch-project#2279) * Bump gson from 2.8.9 to 2.9.0 in /plugins/repository-hdfs Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.8.9...gson-parent-2.9.0) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 0df9845 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:16:56 2022 -0800 Bump morfologik-fsa from 2.1.1 to 2.1.8 in /plugins/analysis-ukrainian (opensearch-project#2278) * Bump morfologik-fsa from 2.1.1 to 2.1.8 in /plugins/analysis-ukrainian Bumps [morfologik-fsa](https://github.com/morfologik/morfologik-stemming) from 2.1.1 to 2.1.8. - [Release notes](https://github.com/morfologik/morfologik-stemming/releases) - [Changelog](https://github.com/morfologik/morfologik-stemming/blob/master/CHANGES.txt) - [Commits](morfologik/morfologik-stemming@2.1.1...2.1.8) --- updated-dependencies: - dependency-name: org.carrot2:morfologik-fsa dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 9780fc6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:16:34 2022 -0800 Bump bc-fips from 1.0.2.1 to 1.0.2.3 in /distribution/tools/plugin-cli (opensearch-project#2276) * Bump bc-fips from 1.0.2.1 to 1.0.2.3 in /distribution/tools/plugin-cli Bumps bc-fips from 1.0.2.1 to 1.0.2.3. --- updated-dependencies: - dependency-name: org.bouncycastle:bc-fips dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 4ef30f4 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:46 2022 -0800 Bump azure-storage-common from 12.14.0 to 12.14.3 in /plugins/repository-azure (opensearch-project#2274) * Bump azure-storage-common in /plugins/repository-azure Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.14.0 to 12.14.3. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.14.0...azure-storage-blob_12.14.3) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 5d0b015 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:33 2022 -0800 Bump jimfs from 1.1 to 1.2 in /distribution/tools/keystore-cli (opensearch-project#2272) Bumps [jimfs](https://github.com/google/jimfs) from 1.1 to 1.2. - [Release notes](https://github.com/google/jimfs/releases) - [Commits](google/jimfs@v1.1...v1.2) --- updated-dependencies: - dependency-name: com.google.jimfs:jimfs dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit f6264a9 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:18 2022 -0800 Bump spock-core from 2.0-groovy-3.0 to 2.1-groovy-3.0 in /buildSrc (opensearch-project#2270) Bumps spock-core from 2.0-groovy-3.0 to 2.1-groovy-3.0. --- updated-dependencies: - dependency-name: org.spockframework:spock-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 21f11ec Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:40:51 2022 -0800 Bump asm-tree from 5.0.4 to 9.2 in /modules/lang-expression (opensearch-project#2269) * Bump asm-tree from 5.0.4 to 9.2 in /modules/lang-expression Bumps asm-tree from 5.0.4 to 9.2. --- updated-dependencies: - dependency-name: org.ow2.asm:asm-tree dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 223d0b2 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Feb 25 13:00:25 2022 -0800 Remove type end-points from no-op bulk and search action (opensearch-project#2261) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 0bd7850 Author: Nick Knize <nknize@apache.org> Date: Fri Feb 25 13:35:48 2022 -0600 [Remove] remaining type usage in Client and AbstractClient (opensearch-project#2258) Removes type parameter from remaining prepareIndex in Client and AbstractClient. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 3a4c2f6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Feb 25 10:40:11 2022 -0800 Bump jimfs from 1.1 to 1.2 in /qa/evil-tests (opensearch-project#2130) * Bump jimfs from 1.1 to 1.2 in /qa/evil-tests Bumps [jimfs](https://github.com/google/jimfs) from 1.1 to 1.2. - [Release notes](https://github.com/google/jimfs/releases) - [Commits](google/jimfs@v1.1...v1.2) --- updated-dependencies: - dependency-name: com.google.jimfs:jimfs dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fixing failing precommit and check Signed-off-by: Vacha Shah <vachshah@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Vacha Shah <vachshah@amazon.com> commit 5b0da85 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Feb 25 10:18:41 2022 -0800 Remove type from validate query API (opensearch-project#2255) * Remove type mapping from RestValidateAction Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless check apply Signed-off-by: Suraj Singh <surajrider@gmail.com> * Include suggested review comment Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 494c7bc Author: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com> Date: Fri Feb 25 03:39:43 2022 +0000 Revert "Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url" (opensearch-project#2256) * Override default Distribution Download URL with custom Distribution URL Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Accidently made commit to main branch, this revives it.Override default Distribution Download URL with custom Distribution URL Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Revert Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> commit 8b48207 Author: Nick Knize <nknize@apache.org> Date: Thu Feb 24 21:20:03 2022 -0600 [Remove] Type from Client.prepare(Index,Delete,Update) (opensearch-project#2253) Removes the type parameter from Client.prepare(Index,Delete,Update) and everywhere it's used throughout the codebase except for prepareIndex(index, type, id) which is removed in a follow up. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Merge branch 'main' into feature/segment-replication Signed-off-by: Kartik Ganesh <gkart@amazon.com>

willyborankin commented Feb 17, 2022

View reviewed changes

willyborankin marked this pull request as ready for review February 17, 2022 22:03

willyborankin requested a review from a team as a code owner February 17, 2022 22:03

willyborankin changed the title ~~Add support of SOCKS proxies for S3 Azure repository~~ Add support of SOCKS proxies for S3 repository Feb 17, 2022

willyborankin force-pushed the s3-proxy-improvements branch from c6f4902 to 4745a5d Compare February 17, 2022 22:27

willyborankin force-pushed the s3-proxy-improvements branch from 4745a5d to 6178e33 Compare February 21, 2022 09:20

reta reviewed Feb 21, 2022

View reviewed changes

willyborankin force-pushed the s3-proxy-improvements branch from 6178e33 to 70a29c0 Compare February 21, 2022 20:57

reta reviewed Feb 21, 2022

View reviewed changes

willyborankin force-pushed the s3-proxy-improvements branch from 70a29c0 to bdbafb5 Compare February 22, 2022 13:13

reta approved these changes Feb 22, 2022

View reviewed changes

dblock requested changes Mar 1, 2022

View reviewed changes

Add support of SOCKS proxies for S3 repository

f5ea951

Signed-off-by: Andrey Pleskach <ples@aiven.io>

willyborankin force-pushed the s3-proxy-improvements branch from bdbafb5 to f5ea951 Compare March 2, 2022 11:21

willyborankin requested a review from dblock March 2, 2022 11:21

kartg approved these changes Mar 2, 2022

View reviewed changes

dblock approved these changes Mar 2, 2022

View reviewed changes

dblock merged commit f13b951 into opensearch-project:main Mar 2, 2022

dblock added the backport 1.x label Mar 2, 2022

opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 2, 2022

Add support of SOCKS proxies for S3 repository (#2160)

9706760

Signed-off-by: Andrey Pleskach <ples@aiven.io> (cherry picked from commit f13b951)

opensearch-trigger-bot bot mentioned this pull request Mar 2, 2022

[Backport 1.x] Add support of SOCKS proxies for S3 repository #2316

Merged

dblock pushed a commit that referenced this pull request Mar 3, 2022

Add support of SOCKS proxies for S3 repository (#2160) (#2316)

8a4adf8

Signed-off-by: Andrey Pleskach <ples@aiven.io> (cherry picked from commit f13b951) Co-authored-by: Andrey Pleskach <ples@aiven.io>

nice-redbull mentioned this pull request Jun 6, 2023

Fixing error: adding a new/forgotten parameter to the configuration for checking the config on startup in plugins/repository-s3 #7924

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of SOCKS proxies for S3 repository #2160

Add support of SOCKS proxies for S3 repository #2160

willyborankin commented Feb 17, 2022 •

edited

Loading

opensearch-ci-bot commented Feb 17, 2022

willyborankin Feb 17, 2022

kartg Mar 1, 2022

willyborankin Feb 17, 2022 •

edited

Loading

willyborankin Feb 17, 2022 •

edited

Loading

opensearch-ci-bot commented Feb 17, 2022

opensearch-ci-bot commented Feb 17, 2022

opensearch-ci-bot commented Feb 21, 2022

reta Feb 21, 2022 •

edited

Loading

willyborankin Feb 21, 2022

dblock Mar 1, 2022

reta Feb 21, 2022

willyborankin Feb 21, 2022

reta Feb 21, 2022

reta Feb 21, 2022

willyborankin Feb 21, 2022

reta Feb 21, 2022

willyborankin Feb 21, 2022

reta Feb 21, 2022

willyborankin Feb 21, 2022

reta Feb 21, 2022

willyborankin Feb 21, 2022

reta Feb 21, 2022 •

edited

Loading

opensearch-ci-bot commented Feb 21, 2022

opensearch-ci-bot commented Feb 22, 2022

dblock left a comment

dblock Mar 1, 2022

opensearch-ci-bot commented Mar 2, 2022

kartg commented Mar 2, 2022 •

edited

Loading

kartg commented Mar 2, 2022

opensearch-ci-bot commented Mar 2, 2022

	throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supportedS3");
	throw new SettingsException("Couldn't convert to S3 protocol. SOCKS is not supported by S3");

Add support of SOCKS proxies for S3 repository #2160

Add support of SOCKS proxies for S3 repository #2160

Conversation

willyborankin commented Feb 17, 2022 • edited Loading

Description

Issues Resolved

Check List

opensearch-ci-bot commented Feb 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willyborankin Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

willyborankin Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

opensearch-ci-bot commented Feb 17, 2022

opensearch-ci-bot commented Feb 17, 2022

opensearch-ci-bot commented Feb 21, 2022

reta Feb 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Feb 21, 2022 • edited Loading

Choose a reason for hiding this comment

opensearch-ci-bot commented Feb 21, 2022

opensearch-ci-bot commented Feb 22, 2022

dblock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Mar 2, 2022

kartg commented Mar 2, 2022 • edited Loading

kartg commented Mar 2, 2022

opensearch-ci-bot commented Mar 2, 2022

willyborankin commented Feb 17, 2022 •

edited

Loading

willyborankin Feb 17, 2022 •

edited

Loading

willyborankin Feb 17, 2022 •

edited

Loading

reta Feb 21, 2022 •

edited

Loading

reta Feb 21, 2022 •

edited

Loading

kartg commented Mar 2, 2022 •

edited

Loading