POC Dynamic threadpool #158
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* add `Strings#isDigits` API inspiration taken from [this SO answer][SO]. note that the stream is not parallelised to avoid the overhead of this as the method is intended to be called primarily with shorter strings where the time to set up would take longer than the actual check. [SO]: https://stackoverflow.com/a/35150400 Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> * add `phone` & `phone-search` analyzer + tokenizer this is largely based on [elasticsearch-phone] and internally uses [libphonenumber]. this intentionally only ports a subset of the features: only `phone` and `phone-search` are supported right now, `phone-email` can be added if/when there's a clear need for it. using `libphonenumber` is required since parsing phone numbers is a non-trivial task (even though it might seem trivial at first glance!), as can be seen in the list [falsehoods programmers believe about phone numbers][falsehoods]. this allows defining the region to be used when analysing a phone number. so far only the generic "unkown" region (`ZZ`) had been used which worked as long as international numbers were prefixed with `+` but did not work when using local numbers (e.g. a number stored as `+4158...` was not matched against a number entered as `004158...` or `058...`). example configuration for an index: ```json { "index": { "analysis": { "analyzer": { "phone": { "type": "phone" }, "phone-search": { "type": "phone-search" }, "phone-ch": { "type": "phone", "phone-region": "CH" }, "phone-search-ch": { "type": "phone-search", "phone-region": "CH" } } } } } ``` this creates four analyzers: `phone` and `phone-search` which do not explicitly specify a region and thus fall back to `ZZ` (unknown region, regional version of international dialing prefix (e.g. `00` instead of `+` in most of europe) will not be recognised) and `phone-ch` and `phone-search-ch` which will try to parse the phone number as a swiss phone number (thus e.g. `00` as a prefix is recognised as the international dialing prefix). note that the analyzer is (currently) not meant to find phone numbers in large text documents - instead it should be used on fields which contain just the phone number (though extra text will be ignored) and it collects the whole content of the field into a `String` in memory, making it unsuitable for large field values. this has been implemented in a new plugin which is however part of the central opensearch repository as it was deemed too big an overhead to have it in a separate repository but not important enough to bundle it directly in `analysis-common` (see the discussion on the issue and the PR for further details). note that the new plugin has been added to the exclude list of the javadoc check as this check is overzealous and also complains in many cases where it shouldn't (e.g. on overridden methods - which it should theoretically not do - or constructors which don't even exist). the check first needs to be improved before this exclusion could be removed. closes opensearch-project#11326 [elasticsearch-phone]: https://github.com/purecloudlabs/elasticsearch-phone [libphonenumber]: https://github.com/google/libphonenumber [falsehoods]: https://github.com/google/libphonenumber/blob/master/FALSEHOODS.md Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> --------- Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…ges as non-breaking (opensearch-project#16181) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…anslog (opensearch-project#16151) Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
…ct#16179) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
…pensearch-project#16194) * Fix warnings from SLF4J on startup when repository-s3 is installed Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix precommit Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>
…ch-project#16192) Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
* Adding _list/shards API Signed-off-by: Harsh Garg <gkharsh@amazon.com>
* Update Apache Lucene to 9.12.0 Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * change to IOContext READONCE in locations where the file is not expected to be read multiple times. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Use READ IOContext for all non Segment* files when copying node-node Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fixing more test failures Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Move Composite912Codec under org.opensearch.index.codec.composite.composite912 package Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Co-authored-by: Marc Handalian <marc.handalian@gmail.com>
…4993) * Avoid deep copy and other allocation improvements * Refactoring based on PR Comments and added JavaDocs * Added more comments * Added character for Triggering Jenkins build * Changes to cover collectZeroDocEntries method * Updated comment based on change in method's functionality * Added test to cover branches in collectZeroDocEntriesIfRequired * Rebased and resolved changelog conflict --------- Signed-off-by: expani <anijainc@amazon.com>
…ject#15925) * cancellation related Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * add better cancellation reason Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update DefaultTaskCancellationTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * refactor Signed-off-by: Kiran Prakash <awskiran@amazon.com> * refactor Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update DefaultTaskSelectionStrategy.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * refactor Signed-off-by: Kiran Prakash <awskiran@amazon.com> * refactor node level threshold Signed-off-by: Kiran Prakash <awskiran@amazon.com> * use query group task Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * code clean up and refactorings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add unit tests and fix existing ones Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * uncomment the test case Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * update CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add queryGroupService Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * refactor and add UTs for new constructs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadocs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove code clutter Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * change annotation version and task selection strategy Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * rename a util class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wrappers from resource type Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * apply spotless Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add rename changes Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * initial changes Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * refactor changes and logical bug fix Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add chanegs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * temp changes Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add UTs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add changelog Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add task completion listener hook Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add remaining pieces to make the feature functional Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * extend stats and fix bugs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix bugs and add logic to make SBP work with wlm Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix bugs and SBP ITs Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missed applyCluster state change Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * decouple queryGroupService and cancellationService Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * replace StateApplier with StateListener interface Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix precommit errors Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kiran Prakash <awskiran@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Co-authored-by: Kiran Prakash <awskiran@amazon.com>
* Segmented cache changes for TieredCache Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding change log Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Allow segment number to be power of two Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Moving common tiered cache IT methods to a common base class Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding disk took time IT test with multiple segment Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Correcting changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing invalid segment count variable name Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Introducing new settings for size for respective cache tier Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changing the default segmentCount logic Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing missing java doc issue Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>
…project#15249) --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
* Refactoring builder tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * adding date tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
…#16110) * Change successfulSearchShardIndices to Set<Index> Signed-off-by: David Zane <davizane@amazon.com> * Update CHANGELOG.md Signed-off-by: Ankit Jain <akjain@amazon.com> --------- Signed-off-by: David Zane <davizane@amazon.com> Signed-off-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
…azure (opensearch-project#16217) * Bump com.azure:azure-json in /plugins/repository-azure Bumps [com.azure:azure-json](https://github.com/Azure/azure-sdk-for-java) from 1.1.0 to 1.3.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@v1.1.0...v1.3.0) --- updated-dependencies: - dependency-name: com.azure:azure-json dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…roject#16228) Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
gbbafna
force-pushed
the
dynamic-threadpool
branch
2 times, most recently
from
71df521
to
8d75efd
Compare
…es (opensearch-project#16237) * Remove force to TLSv1.2 in gradle.properties Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add TLSv1.3 Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>
…rch-project#16240) Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>
…16238) * Fix Cache breaking change Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Empty commit to trigger build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
…omposite99Codec (opensearch-project#16227) * Refactoring builder tests * composite912 codec refactor changes --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
…plugins/repository-gcs (opensearch-project#16216) * Bump com.google.api-client:google-api-client in /plugins/repository-gcs Bumps [com.google.api-client:google-api-client](https://github.com/googleapis/google-api-java-client) from 2.2.0 to 2.7.0. - [Release notes](https://github.com/googleapis/google-api-java-client/releases) - [Changelog](https://github.com/googleapis/google-api-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-api-java-client@v2.2.0...v2.7.0) --- updated-dependencies: - dependency-name: com.google.api-client:google-api-client dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: gaobinlong <gbinlong@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…s-fixture (opensearch-project#16212) * Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.9.0 to 3.9.1. - [Release notes](https://github.com/square/okio/releases) - [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md) - [Commits](square/okio@parent-3.9.0...3.9.1) --- updated-dependencies: - dependency-name: com.squareup.okio:okio dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…opensearch-project#16213) * Bump io.grpc:grpc-api from 1.57.2 to 1.68.0 in /plugins/discovery-gce Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.57.2 to 1.68.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.57.2...v1.68.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> * Centralize grpc version management Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
…6144) Signed-off-by: Finn Carroll <carrofin@amazon.com>
…ch-project#16254) * The protobuf-java leaks through client library dependencies Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Address code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…StoreFetchMultiNodeMultiIndexesUsingBatchAction (opensearch-project#16093) Signed-off-by: kkewwei <kkewwei@163.com>
…h-project#16250) Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
…search-project#16248) Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
…15777) * changelog Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * add tests Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify uri Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * changelog Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * add tests Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify uri Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * revise Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * changelog Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * add tests Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify uri Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * git pull Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * rebase Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * encapsulate querygroupstats in wlmstats Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * fix UT Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
…roject#15955) * add IT for workload management CRUD APIs Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * modify based on comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * add IT for workload management CRUD APIs Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
* Reconfigure remote state thread pool count Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
…ch-project#16267) Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* fix wlm stats output Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * rename wlm stats vars Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix ut failure Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
…rch-project#16281) Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
gbbafna
force-pushed
the
dynamic-threadpool
branch
from
07c95a7
to
4bedb9c
Compare
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
… listener on it Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
gbbafna
force-pushed
the
dynamic-threadpool
branch
from
b161100
to
cc11f7e
Compare
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.