Merge branch 'main' into fix_dissect_bug

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

CHANGELOG.md

@@ -33,7 +33,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.google.code.gson:gson` from 2.10 to 2.10.1
 - Bump `com.maxmind.geoip2:geoip2` from 4.0.0 to 4.0.1
 - Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.16.11 to 0.16.12
-- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0
 - Bump `org.apache.commons:commons-configuration2` from 2.8.0 to 2.9.0
 - Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.3.0
 - Bump `io.opencensus:opencensus-api` from 0.18.0 to 0.31.1 ([#7291](https://github.com/opensearch-project/OpenSearch/pull/7291))
@@ -85,77 +84,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased 2.x]
 ### Added
-- Add coordinator level stats for search latency ([#8386](https://github.com/opensearch-project/OpenSearch/issues/8386))
-- Add metrics for thread_pool task wait time ([#9681](https://github.com/opensearch-project/OpenSearch/pull/9681))
-- Async blob read support for S3 plugin ([#9694](https://github.com/opensearch-project/OpenSearch/pull/9694))
-- [Telemetry-Otel] Added support for OtlpGrpcSpanExporter exporter ([#9666](https://github.com/opensearch-project/OpenSearch/pull/9666))
-- Async blob read support for encrypted containers ([#10131](https://github.com/opensearch-project/OpenSearch/pull/10131))
-- Add capability to restrict async durability mode for remote indexes ([#10189](https://github.com/opensearch-project/OpenSearch/pull/10189))
-- Add Doc Status Counter for Indexing Engine ([#4562](https://github.com/opensearch-project/OpenSearch/issues/4562))
-- Add unreferenced file cleanup count to merge stats ([#10204](https://github.com/opensearch-project/OpenSearch/pull/10204))
-- [Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository ([#9839](https://github.com/opensearch-project/OpenSearch/pull/9839))
-- Improve compressed request handling ([#10261](https://github.com/opensearch-project/OpenSearch/pull/10261))
 
 ### Dependencies
-- Bump `peter-evans/create-or-update-comment` from 2 to 3 ([#9575](https://github.com/opensearch-project/OpenSearch/pull/9575))
-- Bump `actions/checkout` from 2 to 4 ([#9968](https://github.com/opensearch-project/OpenSearch/pull/9968))
-- Bump OpenTelemetry from 1.26.0 to 1.30.1 ([#9950](https://github.com/opensearch-project/OpenSearch/pull/9950))
-- Bump `org.apache.commons:commons-compress` from 1.23.0 to 1.24.0 ([#9973, #9972](https://github.com/opensearch-project/OpenSearch/pull/9973, https://github.com/opensearch-project/OpenSearch/pull/9972))
-- Bump `com.google.cloud:google-cloud-core-http` from 2.21.1 to 2.23.0 ([#9971](https://github.com/opensearch-project/OpenSearch/pull/9971))
-- Bump `mockito` from 5.4.0 to 5.5.0 ([#10022](https://github.com/opensearch-project/OpenSearch/pull/10022))
-- Bump `bytebuddy` from 1.14.3 to 1.14.7 ([#10022](https://github.com/opensearch-project/OpenSearch/pull/10022))
-- Bump `com.zaxxer:SparseBitSet` from 1.2 to 1.3 ([#10098](https://github.com/opensearch-project/OpenSearch/pull/10098))
-- Bump `tibdex/github-app-token` from 1.5.0 to 2.1.0 ([#10125](https://github.com/opensearch-project/OpenSearch/pull/10125))
-- Bump `org.wiremock:wiremock-standalone` from 2.35.0 to 3.1.0 ([#9752](https://github.com/opensearch-project/OpenSearch/pull/9752))
-- Bump `com.google.http-client:google-http-client-jackson2` from 1.43.2 to 1.43.3 ([#10126](https://github.com/opensearch-project/OpenSearch/pull/10126))
-- Bump `org.xerial.snappy:snappy-java` from 1.1.10.3 to 1.1.10.5 ([#10206](https://github.com/opensearch-project/OpenSearch/pull/10206), [#10299](https://github.com/opensearch-project/OpenSearch/pull/10299))
-- Bump `com.google.api.grpc:proto-google-common-protos` from 2.10.0 to 2.25.1 ([#10208](https://github.com/opensearch-project/OpenSearch/pull/10208), [#10298](https://github.com/opensearch-project/OpenSearch/pull/10298))
-- Bump `codecov/codecov-action` from 2 to 3 ([#10209](https://github.com/opensearch-project/OpenSearch/pull/10209))
-- Bump `org.bouncycastle:bcpkix-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
-- Bump `org.bouncycastle:bcprov-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
-- Bump `org.bouncycastle:bcmail-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
-- Bump Lucene from 9.7.0 to 9.8.0 ([10276](https://github.com/opensearch-project/OpenSearch/pull/10276))
-- Bump `com.netflix.nebula.ospackage-base` from 11.4.0 to 11.5.0 ([#10295](https://github.com/opensearch-project/OpenSearch/pull/10295))
-- Bump asm from 9.5 to 9.6 ([#10302](https://github.com/opensearch-project/OpenSearch/pull/10302))
-- Bump netty from 4.1.97.Final to 4.1.99.Final ([#10303](https://github.com/opensearch-project/OpenSearch/pull/10303))
-- Bump `peter-evans/create-pull-request` from 3 to 5 ([#10301](https://github.com/opensearch-project/OpenSearch/pull/10301))
-- Bump `org.apache.avro:avro` from 1.11.2 to 1.11.3 ([#10210](https://github.com/opensearch-project/OpenSearch/pull/10210))
-- Bump `org.bouncycastle:bc-fips` from 1.0.2.3 to 1.0.2.4 ([#10297](https://github.com/opensearch-project/OpenSearch/pull/10297))
-- Bump `org.apache.zookeeper:zookeeper` from 3.9.0 to 3.9.1 ([#10506](https://github.com/opensearch-project/OpenSearch/pull/10506))
 
 ### Changed
-- Add instrumentation in rest and network layer. ([#9415](https://github.com/opensearch-project/OpenSearch/pull/9415))
-- Allow parameterization of tests with OpenSearchIntegTestCase.SuiteScopeTestCase annotation ([#9916](https://github.com/opensearch-project/OpenSearch/pull/9916))
-- Mute the query profile IT with concurrent execution ([#9840](https://github.com/opensearch-project/OpenSearch/pull/9840))
-- Force merge with `only_expunge_deletes` honors max segment size ([#10036](https://github.com/opensearch-project/OpenSearch/pull/10036))
-- Add instrumentation in transport service. ([#10042](https://github.com/opensearch-project/OpenSearch/pull/10042))
-- [Tracing Framework] Add support for SpanKind. ([#10122](https://github.com/opensearch-project/OpenSearch/pull/10122))
-- Pass parent filter to inner query in nested query ([#10246](https://github.com/opensearch-project/OpenSearch/pull/10246))
-- Disable concurrent segment search when terminate_after is used ([#10200](https://github.com/opensearch-project/OpenSearch/pull/10200))
-- Add instrumentation in Inbound Handler. ([#100143](https://github.com/opensearch-project/OpenSearch/pull/10143))
-- Enable remote segment upload backpressure by default ([#10356](https://github.com/opensearch-project/OpenSearch/pull/10356))
-- [Remote Store] Add support to reload repository metadata inplace ([#9569](https://github.com/opensearch-project/OpenSearch/pull/9569))
-- [Metrics Framework] Add Metrics framework. ([#10241](https://github.com/opensearch-project/OpenSearch/pull/10241))
-- Updating the separator for RemoteStoreLockManager since underscore is allowed in base64UUID url charset ([#10379](https://github.com/opensearch-project/OpenSearch/pull/10379))
 
 ### Deprecated
 
 ### Removed
-- Remove spurious SGID bit on directories ([#9447](https://github.com/opensearch-project/OpenSearch/pull/9447))
 
 ### Fixed
-- Fix ignore_missing parameter has no effect when using template snippet in rename ingest processor ([#9725](https://github.com/opensearch-project/OpenSearch/pull/9725))
-- Fix broken backward compatibility from 2.7 for IndexSorted field indices ([#10045](https://github.com/opensearch-project/OpenSearch/pull/10045))
-- Fix concurrent search NPE when track_total_hits, terminate_after and size=0 are used ([#10082](https://github.com/opensearch-project/OpenSearch/pull/10082))
-- Fix remove ingest processor handing ignore_missing parameter not correctly ([10089](https://github.com/opensearch-project/OpenSearch/pull/10089))
-- Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API ([#10101](https://github.com/opensearch-project/OpenSearch/pull/10101))
-- Fix circular dependency in Settings initialization ([10194](https://github.com/opensearch-project/OpenSearch/pull/10194))
-- Fix registration and initialization of multiple extensions ([10256](https://github.com/opensearch-project/OpenSearch/pull/10256))
-- Fix dissect ingest processor parsing empty brackets failed ([#9225](https://github.com/opensearch-project/OpenSearch/pull/9255))
-- Fix Segment Replication ShardLockObtainFailedException bug during index corruption ([10370](https://github.com/opensearch-project/OpenSearch/pull/10370))
-- Fix some test methods in SimulatePipelineRequestParsingTests never run and fix test failure ([#10496](https://github.com/opensearch-project/OpenSearch/pull/10496))
+- Fix failure in dissect ingest processor parsing empty brackets ([#9225](https://github.com/opensearch-project/OpenSearch/pull/9255))
 
 ### Security
 
 [Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
-[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
+[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.12...2.x

buildSrc/build.gradle

@@ -114,7 +114,7 @@ dependencies {
   api 'com.github.johnrengelman:shadow:8.1.1'
   api 'org.jdom:jdom2:2.0.6.1'
   api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
-  api 'de.thetaphi:forbiddenapis:3.5.1'
+  api 'de.thetaphi:forbiddenapis:3.6'
   api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.9.4'

buildSrc/version.properties

@@ -28,7 +28,7 @@ jakarta_annotation = 1.3.5
 # when updating the JNA version, also update the version in buildSrc/build.gradle
 jna               = 5.13.0
 
-netty             = 4.1.99.Final
+netty             = 4.1.100.Final
 joda              = 2.12.2
 
 # client dependencies

libs/core/src/main/java/org/opensearch/Version.java

@@ -96,7 +96,7 @@ public class Version implements Comparable<Version>, ToXContentFragment {
     public static final Version V_2_10_0 = new Version(2100099, org.apache.lucene.util.Version.LUCENE_9_7_0);
     public static final Version V_2_10_1 = new Version(2100199, org.apache.lucene.util.Version.LUCENE_9_7_0);
     public static final Version V_2_11_0 = new Version(2110099, org.apache.lucene.util.Version.LUCENE_9_7_0);
-    public static final Version V_2_12_0 = new Version(2120099, org.apache.lucene.util.Version.LUCENE_9_7_0);
+    public static final Version V_2_12_0 = new Version(2120099, org.apache.lucene.util.Version.LUCENE_9_8_0);
     public static final Version V_3_0_0 = new Version(3000099, org.apache.lucene.util.Version.LUCENE_9_8_0);
     public static final Version CURRENT = V_3_0_0;
 

modules/transport-netty4/licenses/netty-buffer-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+39b05d2d4027971bf99111a9be1d7035a116bb55

modules/transport-netty4/licenses/netty-codec-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+9c3c71e7cf3b8ce3bfc9fa52a524b9ca7ddf259c

modules/transport-netty4/licenses/netty-codec-http-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+992623e7d8f2d96e41faf1687bb963f5433e3517

modules/transport-netty4/licenses/netty-codec-http2-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+cbf1a430ea44dbdedbcde16b185cbb95f28d72c7

modules/transport-netty4/licenses/netty-common-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+847f942381145de23f21c836d05b0677474271d3

modules/transport-netty4/licenses/netty-handler-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+4c0acdb8bb73647ebb3847ac2d503d53d72c02b4

modules/transport-netty4/licenses/netty-resolver-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+fe62f9ccd41b8660d07639dbbab8ae1edd6f2720

modules/transport-netty4/licenses/netty-transport-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+6620fbfb47667a5eb6050e35c7b4c88000bcd77f

modules/transport-netty4/licenses/netty-transport-native-unix-common-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+648ff5571022dbfa6789122e3872477bbf67fa7b

modules/transport-netty4/src/main/java/org/opensearch/http/netty4/Netty4HttpChannel.java

@@ -40,6 +40,7 @@
 import org.opensearch.transport.netty4.Netty4TcpChannel;
 
 import java.net.InetSocketAddress;
+import java.util.Optional;
 
 import io.netty.channel.Channel;
 import io.netty.channel.ChannelPipeline;
@@ -98,6 +99,22 @@ public Channel getNettyChannel() {
         return channel;
     }
 
+    @SuppressWarnings("unchecked")
+    @Override
+    public <T> Optional<T> get(String name, Class<T> clazz) {
+        Object handler = getNettyChannel().pipeline().get(name);
+
+        if (handler == null && inboundPipeline() != null) {
+            handler = inboundPipeline().get(name);
+        }
+
+        if (handler != null && clazz.isInstance(handler) == true) {
+            return Optional.of((T) handler);
+        }
+
+        return Optional.empty();
+    }
+
     @Override
     public String toString() {
         return "Netty4HttpChannel{" + "localAddress=" + getLocalAddress() + ", remoteAddress=" + getRemoteAddress() + '}';

modules/transport-netty4/src/main/java/org/opensearch/transport/netty4/Netty4TcpChannel.java

@@ -41,6 +41,7 @@
 import org.opensearch.transport.TransportException;
 
 import java.net.InetSocketAddress;
+import java.util.Optional;
 
 import io.netty.channel.Channel;
 import io.netty.channel.ChannelFuture;
@@ -164,6 +165,18 @@ public void sendMessage(BytesReference reference, ActionListener<Void> listener)
         }
     }
 
+    @SuppressWarnings("unchecked")
+    @Override
+    public <T> Optional<T> get(String name, Class<T> clazz) {
+        final Object handler = getNettyChannel().pipeline().get(name);
+
+        if (handler != null && clazz.isInstance(handler) == true) {
+            return Optional.of((T) handler);
+        }
+
+        return Optional.empty();
+    }
+
     public Channel getNettyChannel() {
         return channel;
     }

plugins/ingest-attachment/build.gradle

@@ -57,7 +57,7 @@ dependencies {
   runtimeOnly "com.google.guava:guava:${versions.guava}"
   // Other dependencies
   api 'org.tukaani:xz:1.9'
-  api 'commons-io:commons-io:2.13.0'
+  api 'commons-io:commons-io:2.14.0'
   api "org.slf4j:slf4j-api:${versions.slf4j}"
 
   // character set detection

plugins/ingest-attachment/licenses/commons-io-2.14.0.jar.sha1

@@ -0,0 +1 @@
+a4c6e1f6c196339473cd2e1b037f0eb97c62755b

plugins/repository-azure/licenses/netty-codec-dns-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+663b1b7bf3ff0f12fde4df20c72d9e94584ebffa

plugins/repository-azure/licenses/netty-codec-http2-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+cbf1a430ea44dbdedbcde16b185cbb95f28d72c7

plugins/repository-azure/licenses/netty-codec-socks-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+a9fbf4d64b08abed542eefd5f7aed4807edca56f

plugins/repository-azure/licenses/netty-handler-proxy-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+af3cf676eed30184215426ecf0f0dde15555ea9c

plugins/repository-azure/licenses/netty-resolver-dns-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+62dbdf5f25eda75ea8456be1ed72b3fcb0d18774

plugins/repository-azure/licenses/netty-transport-native-unix-common-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+648ff5571022dbfa6789122e3872477bbf67fa7b

plugins/repository-hdfs/licenses/netty-all-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+5ef15a3ce29a792b7ad17438e5f84c617b3f2993

plugins/repository-s3/licenses/netty-buffer-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+39b05d2d4027971bf99111a9be1d7035a116bb55

plugins/repository-s3/licenses/netty-codec-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+9c3c71e7cf3b8ce3bfc9fa52a524b9ca7ddf259c

plugins/repository-s3/licenses/netty-codec-http-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+992623e7d8f2d96e41faf1687bb963f5433e3517

plugins/repository-s3/licenses/netty-codec-http2-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+cbf1a430ea44dbdedbcde16b185cbb95f28d72c7

plugins/repository-s3/licenses/netty-common-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+847f942381145de23f21c836d05b0677474271d3

plugins/repository-s3/licenses/netty-handler-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+4c0acdb8bb73647ebb3847ac2d503d53d72c02b4

plugins/repository-s3/licenses/netty-resolver-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+fe62f9ccd41b8660d07639dbbab8ae1edd6f2720

plugins/repository-s3/licenses/netty-transport-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+6620fbfb47667a5eb6050e35c7b4c88000bcd77f

plugins/repository-s3/licenses/netty-transport-classes-epoll-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+78489936ca1d91483e34a31d04a3b0812386eb39

plugins/repository-s3/licenses/netty-transport-native-unix-common-4.1.100.Final.jar.sha1

@@ -0,0 +1 @@
+648ff5571022dbfa6789122e3872477bbf67fa7b

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobContainer.java

@@ -113,13 +113,6 @@ class S3BlobContainer extends AbstractBlobContainer implements AsyncMultiStreamB
 
     private static final Logger logger = LogManager.getLogger(S3BlobContainer.class);
 
-    /**
-     * Maximum number of deletes in a {@link DeleteObjectsRequest}.
-     *
-     * @see <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html">S3 Documentation</a>.
-     */
-    private static final int MAX_BULK_DELETES = 1000;
-
     private final S3BlobStore blobStore;
     private final String keyPath;
 
@@ -339,12 +332,12 @@ private void doDeleteBlobs(List<String> blobNames, boolean relative) throws IOEx
             outstanding = new HashSet<>(blobNames);
         }
         try (AmazonS3Reference clientReference = blobStore.clientReference()) {
-            // S3 API only allows 1k blobs per delete so we split up the given blobs into requests of max. 1k deletes
+            // S3 API allows 1k blobs per delete so we split up the given blobs into requests of bulk size deletes
             final List<DeleteObjectsRequest> deleteRequests = new ArrayList<>();
             final List<String> partition = new ArrayList<>();
             for (String key : outstanding) {
                 partition.add(key);
-                if (partition.size() == MAX_BULK_DELETES) {
+                if (partition.size() == blobStore.getBulkDeletesSize()) {
                     deleteRequests.add(bulkDelete(blobStore.bucket(), partition));
                     partition.clear();
                 }