Added search backpressure stats API #4932

ketanv3 · 2022-10-26T08:53:17Z

Description

Added search backpressure stats to the existing node/stats API to describe:

the number of cancellations (currently for SearchShardTask only)
the current state of TaskResourceUsageTracker

Sample request: GET /_nodes/stats/search_backpressure?human

Sample response:

{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "runTask",
    "nodes": {
        "T7aqO6zaQX-lt8XBWBYLsA": {
            "timestamp": 1667409521070,
            "name": "runTask-0",
            "transport_address": "127.0.0.1:9300",
            "host": "127.0.0.1",
            "ip": "127.0.0.1:9300",
            "roles": [
                "cluster_manager",
                "data",
                "ingest",
                "remote_cluster_client"
            ],
            "attributes": {
                "testattr": "test",
                "shard_indexing_pressure_enabled": "true"
            },
            "search_backpressure": {
                "search_shard_task": {
                    "resource_tracker_stats": {
                        "heap_usage_tracker": {
                            "cancellation_count": 34,
                            "current_max": "1.1mb",
                            "current_max_bytes": 1203272,
                            "current_avg": "683.8kb",
                            "current_avg_bytes": 700267,
                            "rolling_avg": "1.1mb",
                            "rolling_avg_bytes": 1156270
                        },
                        "cpu_usage_tracker": {
                            "cancellation_count": 318,
                            "current_max": "731.3ms",
                            "current_max_millis": 731,
                            "current_avg": "303.6ms",
                            "current_avg_millis": 303
                        },
                        "elapsed_time_tracker": {
                            "cancellation_count": 310,
                            "current_max": "1.3s",
                            "current_max_millis": 1305,
                            "current_avg": "649.3ms",
                            "current_avg_millis": 649
                        }
                    },
                    "cancellation_stats": {
                        "cancellation_count": 318,
                        "cancellation_limit_reached_count": 97,
                        "last_cancelled_task": {
                            "cpu_usage": "759.8ms",
                            "cpu_usage_millis": 759,
                            "heap_usage": "1.1mb",
                            "heap_usage_bytes": 1211240,
                            "elapsed_time": "1.2s",
                            "elapsed_time_millis": 1207
                        }
                    }
                },
                "mode": "enforced"
            }
        }
    }
}

Issues Resolved

#1181

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma ketan9495@gmail.com

github-actions · 2022-10-26T09:24:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5663/
CommitID: e8e5118

nssuresh2007

LGTM!

github-actions · 2022-10-31T17:38:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5986/
CommitID: 89549f1
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-10-31T19:07:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5992/
CommitID: 46d138d

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

github-actions · 2022-11-02T09:41:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6195/
CommitID: b17b52b

github-actions · 2022-11-02T10:17:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6196/
CommitID: cc2e682

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

github-actions · 2022-11-02T12:07:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6197/
CommitID: 8c0decb
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Bukhtawar

"cancellation_stats": {
                        "cancellation_count": 0,
                        "cancellation_limit_reached_count": 0,
                        "last_cancelled_task": null
                    }

Can we have individual trackers have this info?

ketanv3 · 2022-11-02T14:47:05Z

Can we have individual trackers have this info?

These cancellation stats are at a shard level. Only having them at a tracker level won't be very meaningful as tasks may get cancelled for multiple reasons, essentially double-counting the cancellation count.

cancellation_limit_reached_count is not influenced by trackers, so it's better to keep it separate.

github-actions · 2022-11-02T17:26:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6231/
CommitID: 66791e1
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureState.java

server/src/main/java/org/opensearch/search/backpressure/settings/SearchBackpressureMode.java

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

server/src/main/java/org/opensearch/search/backpressure/stats/CancelledTaskStats.java

server/src/main/java/org/opensearch/search/backpressure/stats/SearchBackpressureStats.java

server/src/main/java/org/opensearch/search/backpressure/stats/SearchShardTaskStats.java

Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <ketan9495@gmail.com>

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

github-actions · 2022-11-03T07:16:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6278/
CommitID: 7602cac
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

github-actions · 2022-11-03T08:06:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6282/
CommitID: 2f73821
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ketanv3 · 2022-11-03T08:36:03Z

Note on Gradle checks:

The ./gradlew ':qa:mixed-cluster:v2.5.0#mixedClusterTest' task is expected to fail as backward-compatibility checks have been lowered from 3.0.0 to 2.4.0 in this PR. This will be resolved once the backport PR (#5039) also gets merged.

Have separately verified with 3.0.0 that everything passes.

./gradlew check
...
BUILD SUCCESSFUL in 20m 29s
2557 actionable tasks: 584 executed, 1973 up-to-date

psychbot

Looks good to me!

Bukhtawar · 2022-11-03T12:46:14Z

Overriding merge due to Jenkins failure opensearch-project/opensearch-ci#222

ketanv3 marked this pull request as ready for review October 26, 2022 09:53

ketanv3 requested review from a team and reta as code owners October 26, 2022 09:53

nssuresh2007 approved these changes Oct 27, 2022

View reviewed changes

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from e8e5118 to 89549f1 Compare October 31, 2022 17:20

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from 89549f1 to 46d138d Compare October 31, 2022 18:34

Bukhtawar reviewed Nov 2, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java Show resolved Hide resolved

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from 46d138d to b17b52b Compare November 2, 2022 09:09

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from b17b52b to cc2e682 Compare November 2, 2022 09:43

Bukhtawar reviewed Nov 2, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Nov 2, 2022

View reviewed changes

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java Show resolved Hide resolved

ketanv3 mentioned this pull request Nov 2, 2022

[Backport 2.x] In-flight cancellation of SearchShardTask based on resource consumption #5039

Merged

6 tasks

Bukhtawar reviewed Nov 2, 2022

View reviewed changes

psychbot reviewed Nov 2, 2022

View reviewed changes

ketanv3 added 3 commits November 3, 2022 12:15

Added search backpressure stats API

d96cb97

Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <ketan9495@gmail.com>

Simplified enabled/enforced flags with search backpressure mode

9d37186

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

Add support from Version.V_2_4_0 onwards

93b77f7

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from 66791e1 to 7602cac Compare November 3, 2022 07:15

Removed CancelledTaskStats for last cancelled task

2f73821

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

ketanv3 force-pushed the feature/inflight-cancellation-stats branch from 7602cac to 2f73821 Compare November 3, 2022 08:04

psychbot approved these changes Nov 3, 2022

View reviewed changes

Bukhtawar approved these changes Nov 3, 2022

View reviewed changes

Bukhtawar merged commit 7c8b031 into opensearch-project:main Nov 3, 2022

dhwanilpatel mentioned this pull request Jan 16, 2023

Add cluster manager task throttling stats in nodes stats API #5790

Merged

6 tasks

dbwiddis mentioned this pull request Oct 3, 2023

[FEATURE] Add support for indexing pressure stats opensearch-project/opensearch-java#453

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added search backpressure stats API #4932

Added search backpressure stats API #4932

ketanv3 commented Oct 26, 2022 •

edited

Loading

github-actions bot commented Oct 26, 2022

nssuresh2007 left a comment

github-actions bot commented Oct 31, 2022

github-actions bot commented Oct 31, 2022

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

Bukhtawar left a comment

ketanv3 commented Nov 2, 2022 •

edited

Loading

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 3, 2022

github-actions bot commented Nov 3, 2022

ketanv3 commented Nov 3, 2022

psychbot left a comment

Bukhtawar commented Nov 3, 2022

Added search backpressure stats API #4932

Added search backpressure stats API #4932

Conversation

ketanv3 commented Oct 26, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Oct 26, 2022

Gradle Check (Jenkins) Run Completed with:

nssuresh2007 left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 31, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 31, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

Bukhtawar left a comment

Choose a reason for hiding this comment

ketanv3 commented Nov 2, 2022 • edited Loading

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

ketanv3 commented Nov 3, 2022

psychbot left a comment

Choose a reason for hiding this comment

Bukhtawar commented Nov 3, 2022

ketanv3 commented Oct 26, 2022 •

edited

Loading

ketanv3 commented Nov 2, 2022 •

edited

Loading