consume query level cpu and memory usage in query insights #13739

ansjcy · 2024-05-17T22:39:06Z

Description

This is a follow up on https://github.com/opensearch-project/OpenSearch/pull/13172/files#diff-916410426108bbd0b06952c34490ca05c7a2a78c1c10f7af28e88ead23a70f24 to consume query level cpu and memory usage in query insights

Related Issues

#12399

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
API changes companion pull request created.
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-05-17T22:47:01Z

❌ Gradle check result for 80c4615: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-30T22:44:47Z

❌ Gradle check result for 02bb39e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...sights/src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

...va/org/opensearch/plugin/insights/rules/transport/top_queries/TransportTopQueriesAction.java

deshsidd · 2024-06-03T22:18:36Z

Looks good to me overall. Some minor comments and changelog also needed.

jainankitk

Minor comments to improve the readability of the code!

...sights/src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

plugins/query-insights/src/main/java/org/opensearch/plugin/insights/rules/model/MetricType.java

...sights/src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

github-actions · 2024-06-04T23:12:44Z

❌ Gradle check result for 14294b1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-07T18:38:05Z

❌ Gradle check result for 2d19224: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

deshsidd · 2024-06-07T19:09:53Z

Thanks for the refactoring! LGTM

github-actions · 2024-06-07T19:48:55Z

❌ Gradle check result for fa2c127: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-07T20:11:32Z

❌ Gradle check result for 46669df:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-07T20:11:56Z

❌ Gradle check result for fa2c127:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Chenyang Ji <cyji@amazon.com>

github-actions · 2024-06-10T17:30:10Z

✅ Gradle check result for 0ccd62d: SUCCESS

codecov · 2024-06-10T17:31:42Z

Codecov Report

Attention: Patch coverage is 67.30769% with 34 lines in your changes missing coverage. Please review.

Project coverage is 71.66%. Comparing base (b15cb0c) to head (0ccd62d).
Report is 385 commits behind head on main.

Files	Patch %	Lines
.../insights/core/listener/QueryInsightsListener.java	69.23%	10 Missing and 2 partials ⚠️
...in/insights/core/service/QueryInsightsService.java	50.00%	8 Missing and 4 partials ⚠️
...ansport/top_queries/TransportTopQueriesAction.java	20.00%	7 Missing and 1 partial ⚠️
...lugin/insights/core/service/TopQueriesService.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13739      +/-   ##
============================================
+ Coverage     71.42%   71.66%   +0.24%     
- Complexity    59978    61670    +1692     
============================================
  Files          4985     5091     +106     
  Lines        282275   289543    +7268     
  Branches      40946    41864     +918     
============================================
+ Hits         201603   207495    +5892     
- Misses        63999    64967     +968     
- Partials      16673    17081     +408

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

opensearch-trigger-bot · 2024-06-10T18:48:27Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13739-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 04a417a7328ef7d4c501508703898ff2b477f302
# Push it to GitHub
git push --set-upstream origin backport/backport-13739-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13739-to-2.x.

…h-project#13739) * consume query level cpu and memory usage in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * handle failed requests metrics in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * refactor the code to make it more maintainable Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit 04a417a)

…s tracking (#14085) * Query-level resource usages tracking (#13172) * Query-level resource usages tracking Signed-off-by: Chenyang Ji <cyji@amazon.com> * Moving TaskResourceTrackingService to clusterService Signed-off-by: Chenyang Ji <cyji@amazon.com> * use shard response header to piggyback task resource usages Signed-off-by: Chenyang Ji <cyji@amazon.com> * split changes for query insights plugin Signed-off-by: Chenyang Ji <cyji@amazon.com> * improve the supplier logic and other misc items Signed-off-by: Chenyang Ji <cyji@amazon.com> * track resource usage for failed requests Signed-off-by: Chenyang Ji <cyji@amazon.com> * move resource usages interactions into TaskResourceTrackingService Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit 3d1fa98) * fix concurrent modification issue in thread context (#14084) Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit c8f0b6d) * consume query level cpu and memory usage in query insights (#13739) * consume query level cpu and memory usage in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * handle failed requests metrics in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * refactor the code to make it more maintainable Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit 04a417a) * fix japicmp check for threadContext Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit b403fdc)

…s tracking (opensearch-project#14085) * Query-level resource usages tracking (opensearch-project#13172) * Query-level resource usages tracking Signed-off-by: Chenyang Ji <cyji@amazon.com> * Moving TaskResourceTrackingService to clusterService Signed-off-by: Chenyang Ji <cyji@amazon.com> * use shard response header to piggyback task resource usages Signed-off-by: Chenyang Ji <cyji@amazon.com> * split changes for query insights plugin Signed-off-by: Chenyang Ji <cyji@amazon.com> * improve the supplier logic and other misc items Signed-off-by: Chenyang Ji <cyji@amazon.com> * track resource usage for failed requests Signed-off-by: Chenyang Ji <cyji@amazon.com> * move resource usages interactions into TaskResourceTrackingService Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit 3d1fa98) * fix concurrent modification issue in thread context (opensearch-project#14084) Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit c8f0b6d) * consume query level cpu and memory usage in query insights (opensearch-project#13739) * consume query level cpu and memory usage in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * handle failed requests metrics in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * refactor the code to make it more maintainable Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit 04a417a) * fix japicmp check for threadContext Signed-off-by: Chenyang Ji <cyji@amazon.com> (cherry picked from commit b403fdc) Signed-off-by: kkewwei <kkewwei@163.com>

…h-project#13739) * consume query level cpu and memory usage in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * handle failed requests metrics in query insights Signed-off-by: Chenyang Ji <cyji@amazon.com> * refactor the code to make it more maintainable Signed-off-by: Chenyang Ji <cyji@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com>

ansjcy requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners May 17, 2024 22:39

ansjcy mentioned this pull request May 17, 2024

[META] Generic Query Insights Framework #11522

Open

32 tasks

ansjcy force-pushed the consume-resource-usages-in-query-insights branch from 80c4615 to 02bb39e Compare May 30, 2024 22:32

deshsidd reviewed Jun 3, 2024

View reviewed changes

jainankitk reviewed Jun 4, 2024

View reviewed changes

ansjcy force-pushed the consume-resource-usages-in-query-insights branch from 02bb39e to 14294b1 Compare June 4, 2024 22:58

ansjcy force-pushed the consume-resource-usages-in-query-insights branch from 2d19224 to fa2c127 Compare June 7, 2024 18:36

deshsidd approved these changes Jun 7, 2024

View reviewed changes

ansjcy force-pushed the consume-resource-usages-in-query-insights branch from fa2c127 to 46669df Compare June 7, 2024 19:55

ansjcy mentioned this pull request Jun 8, 2024

[Backport 2.x] Core and Plugin changes for query-level resource usages tracking #14085

Merged

3 tasks

ansjcy added 3 commits June 10, 2024 09:36

consume query level cpu and memory usage in query insights

8c310dd

Signed-off-by: Chenyang Ji <cyji@amazon.com>

handle failed requests metrics in query insights

8d08e8a

Signed-off-by: Chenyang Ji <cyji@amazon.com>

refactor the code to make it more maintainable

0ccd62d

Signed-off-by: Chenyang Ji <cyji@amazon.com>

ansjcy force-pushed the consume-resource-usages-in-query-insights branch from 46669df to 0ccd62d Compare June 10, 2024 16:37

jed326 approved these changes Jun 10, 2024

View reviewed changes

jed326 merged commit 04a417a into opensearch-project:main Jun 10, 2024
29 of 30 checks passed

opensearch-trigger-bot bot added the backport-failed label Jun 10, 2024

This was referenced Jun 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for AwarenessAttributeDecommissionIT #14290

Open

[AUTOCUT] Gradle Check Flaky Test Report for TieredSpilloverCacheStatsIT #14300

Open

[AUTOCUT] Gradle Check Flaky Test Report for RelocationIT #14309

Open

finnegancarroll mentioned this pull request Jun 14, 2024

Remove composeBuild fixture tasks when docker support not found #14357

Merged

3 tasks

BrewTestBot mentioned this pull request Jun 26, 2024

opensearch 2.15.0 Homebrew/homebrew-core#175637

Merged

jainankitk mentioned this pull request Jul 16, 2024

[BUG] Setting validation missing during node startup #14777

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

consume query level cpu and memory usage in query insights #13739

consume query level cpu and memory usage in query insights #13739

ansjcy commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024

github-actions bot commented May 30, 2024

deshsidd commented Jun 3, 2024

jainankitk left a comment

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 7, 2024

deshsidd commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 10, 2024

codecov bot commented Jun 10, 2024

opensearch-trigger-bot bot commented Jun 10, 2024

consume query level cpu and memory usage in query insights #13739

consume query level cpu and memory usage in query insights #13739

Conversation

ansjcy commented May 17, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 17, 2024

github-actions bot commented May 30, 2024

deshsidd commented Jun 3, 2024

jainankitk left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 7, 2024

deshsidd commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

github-actions bot commented Jun 10, 2024

codecov bot commented Jun 10, 2024

Codecov Report

opensearch-trigger-bot bot commented Jun 10, 2024

ansjcy commented May 17, 2024 •

edited

Loading