Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase throttling feature branch #4713

Conversation

dhwanilpatel
Copy link
Contributor

Description

Re base feature/master-task-throttling branch with latest main branch.

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

burck1 and others added 30 commits August 24, 2022 11:27
…bat manager" (opensearch-project#4289)

* [BUG] Update opensearch-service-x64.exe parameters to //ES for Execute Service. Update opensearch-service-mgr.exe parameters to //ES for Edit Service. Add code comments for the Apache Commons Daemon.

Signed-off-by: Alex Burck <me@alexburck.com>

* update changelog with pull request link

Signed-off-by: Alex Burck <me@alexburck.com>

Signed-off-by: Alex Burck <me@alexburck.com>
* Removing dead code in RecoveryTarget.

This code in RecoveryTarget is not invoked, all of these methods are implemented by the parent ReplicationTarget with the same behavior.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR Comments.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
* Update changelog contribution guide

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Fix reference to pull request

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…4133)

This commit prevents a replica shard from being failed in the case that
a replication action to a replica is canceled due to the primary shard
being closed.

Signed-off-by: Andrew Ross <andrross@amazon.com>

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…der from package 'org.opensearch.action.support.master.info' for subclasses (opensearch-project#4307)

* Restore using the class ClusterInfoRequest and ClusterInfoRequestBuilder from package 'org.opensearch.action.support.master.info' for subclasses

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Add changelog

Signed-off-by: Tianli Feng <ftianli@amazon.com>

Signed-off-by: Tianli Feng <ftianli@amazon.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
…nsearch-project#4305)

* [BUG] Create logs directory before running OpenSearch on Windows

Signed-off-by: Alex Burck <me@alexburck.com>

* update changlog pr link

Signed-off-by: Alex Burck <me@alexburck.com>

Signed-off-by: Alex Burck <me@alexburck.com>
…h-project#4240)

* Use RemoteSegmentStoreDirectory instead of RemoteDirectory

Signed-off-by: Sachin Kale <kalsac@amazon.com>
When publishing Zip POM the groupId value was hard-coded to `org.opensearch.plugin` value which worked fine for existing core plugins but is not convenient for other plugins (such as community plugins maintained in independent repositories).

This PR changes the sources of the ZIP publishing groupId value.

Specifically, there are two ways to set the value:
1) It is automatically inherited from the Gradle "project.group"
2) It can be manually specified in the ZIP publication POM object

This PR also brings a major rework of tests in PublishTests class. Individual testing scenarios are driven by "real" gradle building scripts (utilizing `java-gradle-plugin` gradle plugin).

Closes opensearch-project#3692

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>
…ss in testReplicationOnDone test (opensearch-project#4314)

* [Segment Replication] testReplicationOnDone Add timeout to allow time for verify call

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update changelog

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add change log entry

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
The package `org.yaml:snakeyaml` before version 1.31 are vulnerable to Denial of Service (DoS) due missing to nested depth limitation for collections.

Details at https://nvd.nist.gov/vuln/detail/CVE-2022-25857

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Fixed label for dependabot PR helper

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Update autocommit workflow for dependabot changelog

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add version config for dependabot changelog helper

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Support for HTTP/2 (server-side)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Added HTTP/1.1 channel configuration

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Update pul request URL in CHANGELOG.md

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…" (opensearch-project#4361)

This reverts commit c28221e.

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…n. (opensearch-project#4225)

* Segment Replication.  Fix Cancellation of replication events.

This PR updates segment replication paths to correctly cancel replication events on the primary and replica.
In the source service, any ongoing event for a primary that is sending to a replica that shuts down or is promoted as a new primary are cancelled.
In the target service, any ongoing event for a replica that is promoted as a new primary or is fetching from a primary that shuts down.
It wires up SegmentReplicationSourceService as an IndexEventListener so that it can respond to events and cancel any ongoing transfer state.
This change also includes some test cleanup for segment replication to rely on actual components over mocks.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Fix to not start/stop SegmentReplicationSourceService as a lifecycle component with feature flag off.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Update logic to properly mark SegmentReplicationTarget as cancelled when cancel initiated by primary.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Minor updates from self review.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add missing changelog entry.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix token usage for changelog helper

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add conditional check for dependabot steps

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add dependency section

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Bug fixes for dependabot changelog verifier

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Update the changelog

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…vel client (opensearch-project#4064)

* Create and delete PIT search rest layer changes

Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…search-project#4354)

* Bump xmlbeans from 5.1.0 to 5.1.1 in /plugins/ingest-attachment

Bumps xmlbeans from 5.1.0 to 5.1.1.

---
updated-dependencies:
- dependency-name: org.apache.xmlbeans:xmlbeans
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…nts (opensearch-project#4352)

Overload `generateHistoryOnReplica` to be able to generate only a specific `Engine.Operation.TYPE` operations as required by the `testUpdateSegments` test

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Added bwc version 2.2.2

* Add changelog

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>
… replica promotion (opensearch-project#4365)

* [Segment Replication] Bump segment infos counter before commit during replica promotion

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add changelog entry

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>
…d primary on onNewCheckpoint on replica (opensearch-project#4363)

* [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add changelog entry

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments 2

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Test failures

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>
yeohbraddy and others added 21 commits October 4, 2022 10:12
… Jenkins build now waits for pre-commit checks to pass before attempting (opensearch-project#4660)

* Ignore all malformed objects when ignore_malformed is true (opensearch-project#4494)

Fixes a bug to not fail the entire document when "ignore_malformed" is set to true. Allowing the valid fields 
to be indexed and ignore only the malformed fields.

Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>

* Adding javadoc check to the precommi\t GitHub Action

Signed-off-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>

* gradle check now only runs after successful gradle precommit check

Signed-off-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>

* Added to CHANGELOG

Signed-off-by: Braddy Yeoh yeohbraddy@gmail.com
Signed-off-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>

Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>
Signed-off-by: Braddy Yeoh yeohbraddy@gmail.com
Co-authored-by: Hauck <67768441+hauck-jvsh@users.noreply.github.com>
Co-authored-by: Braddy Yeoh <braddy.yeoh@ucdconnect.ie>
)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…ct#4679)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* First draft for Windows precommit

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Adding changelog

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
* First draft for Windows precommit

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Adding changelog

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Adding precommit for macos

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Adding Changelog

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
…t#4683)

This test fails pretty reliably if I run it on repeat. I believe the
problem is that the test assumes the function will take longer than 2ms,
which is likely not a valid assumption in all cases. Fortunately, I can
pass in a zero duration which is guaranteed to timeout even if the
system clock does not advance at all.

Also moved the assertions out of the callback into the main test method,
otherwise the assertion error messages would get buried and the test
report would just show a timeout error.

Signed-off-by: Andrew Ross <andrross@amazon.com>

Signed-off-by: Andrew Ross <andrross@amazon.com>
…elyRunningAsRoot() (opensearch-project#4656)

* Bump jna from 5.11.0 to 5.12.1

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Include Windows OS in check for definitelyRunningAsRoot()

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Combine try/catch blocks

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Someone sniped my PR number between predicting and pushing

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Update SHAs

Signed-off-by: Daniel Widdis <widdis@gmail.com>

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Recommission API service level changes

Signed-off-by: pranikum <109206473+pranikum@users.noreply.github.com>
* Weighted round-robin scheduling policy for shard coordination traffic routing

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
…ject#4688)

My previous fix introduced a new race condition by making the assertions
before waiting on the latch.

Signed-off-by: Andrew Ross <andrross@amazon.com>

Signed-off-by: Andrew Ross <andrross@amazon.com>
Fixes bwc failure caused by commit
6993ac9.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…tailed_error disabled (opensearch-project#4669)

* Fixed misunderstanding message 'No OpenSearchException found' when detailed_error disabled

Signed-off-by: Xue Zhou <xuezhou@amazon.com>
…pensearch-project#4572)

* Update access modifier to support extensibility

Change access modifier from default to protected.
This will help to build new geo based aggregation
outside OpenSearch, by keeping GeoGrid Classes as base class.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Updated CHANGELOG

Added PR details to CHANGELOG.md

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Rename InternalGeoGridBucket to BaseGeoGridBucket

Update class names, references and comments.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Rename InternalGeoGrid to BaseGeoGrid

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Make GridBucket classes package-private

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Remove Internal prefix from Geo Grid classes

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update constructor and class access modifier

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update access modifier based on usage

Made classes package private if it is not used
outside the package.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
…oject#2768)

* [Remove] LegacyESVersion.V_7_0_* and V_7_1_* constants

Removes all usages of LegacyESVersion.V_7_0_ and LegacyESVersion.V_7_1 version
constants.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Rebase from main

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix serialization issue with build flavor removal

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* remove stale bwc test

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* rebase and update

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* cleanup

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix failing mapper test

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…xx tasks (opensearch-project#4696)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…rch-project#4066)

* Bump gson from 2.9.0 to 2.9.1 in /test/fixtures/hdfs-fixture

Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
* Always auto release the flood stage block

Removes support for using a system property to disable the automatic release of
the write block applied when a node exceeds the flood-stage watermark.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update IAE message

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Controlling discovery for decommissioned nodes

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
…oject#4702)

Removes all usages of LegacyESVersion.V_7_2_ and LegacyESVersion.V_7_3 version
constants along with related ancient APIs.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…oject#4704)

Removes all usages of LegacyESVersion.V_7_4_ and LegacyESVersion.V_7_5
version constants along with ancient API logic.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2022

Gradle Check (Jenkins) Run Completed with:

…8-10 for feature branch rebase

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2022

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

Codecov Report

Merging #4713 (7cdffe2) into feature/master-task-throttling (e231136) will increase coverage by 0.12%.
The diff coverage is 71.05%.

@@                         Coverage Diff                          @@
##             feature/master-task-throttling    #4713      +/-   ##
====================================================================
+ Coverage                             70.57%   70.69%   +0.12%     
- Complexity                            57297    57604     +307     
====================================================================
  Files                                  4621     4665      +44     
  Lines                                275568   276614    +1046     
  Branches                              40325    40297      -28     
====================================================================
+ Hits                                 194472   195550    +1078     
+ Misses                                64826    64771      -55     
- Partials                              16270    16293      +23     
Impacted Files Coverage Δ
...main/java/org/opensearch/gradle/PublishPlugin.java 43.43% <0.00%> (-0.45%) ⬇️
.../java/org/opensearch/client/RequestConverters.java 82.02% <0.00%> (-3.81%) ⬇️
...ava/org/opensearch/client/RestHighLevelClient.java 41.75% <0.00%> (-0.66%) ⬇️
...va/org/opensearch/common/settings/KeyStoreCli.java 0.00% <ø> (ø)
...rc/main/java/org/opensearch/plugins/PluginCli.java 0.00% <ø> (ø)
...main/java/org/opensearch/plugins/PluginHelper.java 100.00% <ø> (+9.09%) ⬆️
...va/org/opensearch/geometry/GeometryCollection.java 92.85% <0.00%> (-3.44%) ⬇️
...earch/analysis/common/NGramTokenFilterFactory.java 75.00% <0.00%> (+10.71%) ⬆️
...gregations/bucket/geogrid/BucketPriorityQueue.java 75.00% <ø> (ø)
...arch/aggregations/bucket/geogrid/CellIdSource.java 69.23% <ø> (ø)
... and 727 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@dhwanilpatel dhwanilpatel marked this pull request as ready for review October 8, 2022 11:39
@dhwanilpatel dhwanilpatel requested review from a team and reta as code owners October 8, 2022 11:39
@Bukhtawar Bukhtawar merged commit 1212afd into opensearch-project:feature/master-task-throttling Oct 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.