Include file cache stats in node stats api response #6333

adnapibar · 2023-02-16T07:10:42Z

Description

Follow up to the #5641. This change exposes the file cache stats as part the node stats API response. The file cache was added as part of searchable snapshot in #5641. ~~In addition this also adds an index event listener to clean up cache entries when a remote store index is deleted.~~

Issues Resolved

Resolves #5982

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-02-16T07:39:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11208/
CommitID: 881de81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

github-actions · 2023-02-17T01:27:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

URL: https://build.ci.opensearch.org/job/gradle-check/11270/
CommitID: 9780770
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-02-17T01:29:16Z

Codecov Report

Merging #6333 (c594d2a) into main (d439244) will increase coverage by 0.03%.
The diff coverage is 55.43%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6333      +/-   ##
============================================
+ Coverage     70.71%   70.74%   +0.03%     
- Complexity    59011    59049      +38     
============================================
  Files          4800     4801       +1     
  Lines        282712   282798      +86     
  Branches      40763    40767       +4     
============================================
+ Hits         199907   200073     +166     
+ Misses        66427    66303     -124     
- Partials      16378    16422      +44

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	`69.56% <ø> (ø)`
...src/main/java/org/opensearch/node/NodeService.java	`69.84% <0.00%> (-1.13%)`	⬇️
...search/cluster/MockInternalClusterInfoService.java	`0.00% <0.00%> (ø)`
.../java/org/opensearch/test/InternalTestCluster.java	`57.72% <ø> (-0.81%)`	⬇️
...in/java/org/opensearch/indices/IndicesService.java	`60.52% <7.14%> (-3.04%)`	⬇️
...rch/action/admin/cluster/node/stats/NodeStats.java	`52.43% <33.33%> (-1.11%)`	⬇️
server/src/main/java/org/opensearch/node/Node.java	`83.64% <50.00%> (ø)`
...h/index/store/remote/filecache/FileCacheStats.java	`71.18% <71.18%> (ø)`
...on/admin/cluster/node/stats/NodesStatsRequest.java	`93.15% <100.00%> (+0.09%)`	⬆️
.../cluster/node/stats/TransportNodesStatsAction.java	`100.00% <100.00%> (ø)`
... and 501 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

kotwanikunal · 2023-02-17T01:49:39Z

server/src/main/java/org/opensearch/index/store/remote/filecache/FileCacheCleaner.java

+    }
+
+    private Path getShardPath(ShardId shardId) throws IOException {
+        final Path[] paths = nodeEnvironment.availableShardPaths(shardId);


We can potentially use the change from https://github.com/opensearch-project/OpenSearch/pull/6350/files#diff-64eed71f060448867b2456f621ebd72a43d52cd20db9080a82308978a0930a5a.

Moving this to another PR as per Andrew's comment below, will address in there.

andrross · 2023-02-17T18:27:50Z

In addition this also adds an index event listener to clean up cache entries when a remote store index is deleted.

Is this related to the stats API at all? If not, can we split it out as a separate PR and commit? It'll be easier to review and make the commit history a bit clearer.

andrross · 2023-02-17T22:44:42Z

We'll want to make sure that the new file_cache section of that stats API isn't present if the feature flag is not enabled.

adnapibar · 2023-02-23T16:44:42Z

In addition this also adds an index event listener to clean up cache entries when a remote store index is deleted.

Is this related to the stats API at all? If not, can we split it out as a separate PR and commit? It'll be easier to review and make the commit history a bit clearer.

We can, in that case we have to merge the stats API code changes first before raising a PR for the cleaner as it depends on the former.

kotwanikunal · 2023-02-23T17:46:08Z

We'll want to make sure that the new file_cache section of that stats API isn't present if the feature flag is not enabled.

+1 on this. LGTM otherwise.

github-actions · 2023-02-23T20:29:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11657/
CommitID: c594d2a

andrross · 2023-02-23T21:32:15Z

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

@@ -171,6 +176,11 @@ public NodeStats(StreamInput in) throws IOException {
        } else {
            weightedRoutingStats = null;
        }
+        if (in.getVersion().onOrAfter(Version.V_3_0_0)) {


I think we'll want to gate these writes behind the feature flag as well. We want to be able to make backwards-incompatible changes to this feature (like maybe remove this or change the structure) and not impact any instances that have not enabled the experimental feature flag. The only way to do that (I think) is to make sure we don't write anything over the wire unless the feature flag is turned on.

This change exposes the file cache stats as part the node stats API response. Signed-off-by: Rabi Panda <adnapibar@gmail.com>

github-actions · 2023-02-23T22:35:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/11666/
CommitID: 16f9eb4
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-02-23T23:01:55Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6333-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0f3b8705a982ac8cce43d57e235cc4e13fb3ae6f
# Push it to GitHub
git push --set-upstream origin backport/backport-6333-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6333-to-2.x.

kotwanikunal · 2023-02-23T23:14:38Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6333-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0f3b8705a982ac8cce43d57e235cc4e13fb3ae6f
# Push it to GitHub
git push --set-upstream origin backport/backport-6333-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6333-to-2.x.

@adnapibar Looks like it needs a manual backport 🙂

) This change exposes the file cache stats as part the node stats API response. Signed-off-by: Rabi Panda <adnapibar@gmail.com> (cherry picked from commit 0f3b870)

kotwanikunal · 2023-02-25T00:02:11Z

Backport: #6485

* Include file cache stats in node stats response (#6333) This change exposes the file cache stats as part the node stats API response. Signed-off-by: Rabi Panda <adnapibar@gmail.com> (cherry picked from commit 0f3b870) * Add cache reservation logic (#6350) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit eb78246) --------- Co-authored-by: Rabi Panda <adnapibar@gmail.com>

anasalkouz linked an issue Feb 16, 2023 that may be closed by this pull request

[Searchable Snapshot] Add listener to Index events to update local file system cache entries #5981

Closed

adnapibar added the skip-changelog label Feb 17, 2023

adnapibar marked this pull request as ready for review February 17, 2023 00:08

adnapibar requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners February 17, 2023 00:08

adnapibar commented Feb 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java Outdated Show resolved Hide resolved

kotwanikunal reviewed Feb 17, 2023

View reviewed changes

kotwanikunal approved these changes Feb 23, 2023

View reviewed changes

adnapibar removed a link to an issue Feb 23, 2023

[Searchable Snapshot] Add listener to Index events to update local file system cache entries #5981

Closed

andrross reviewed Feb 23, 2023

View reviewed changes

Include file cache stats in node stats response

16f9eb4

This change exposes the file cache stats as part the node stats API response. Signed-off-by: Rabi Panda <adnapibar@gmail.com>

andrross mentioned this pull request Feb 23, 2023

[BUG] org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness flaky #3603

Open

andrross approved these changes Feb 23, 2023

View reviewed changes

andrross added the backport 2.x Backport to 2.x branch label Feb 23, 2023

andrross merged commit 0f3b870 into opensearch-project:main Feb 23, 2023

kotwanikunal mentioned this pull request Feb 24, 2023

[Backport 2.x] Add file cache stats, reservation logic #6485

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include file cache stats in node stats api response #6333

Include file cache stats in node stats api response #6333

adnapibar commented Feb 16, 2023 •

edited

Loading

github-actions bot commented Feb 16, 2023

github-actions bot commented Feb 17, 2023

codecov-commenter commented Feb 17, 2023 •

edited

Loading

kotwanikunal Feb 17, 2023

adnapibar Feb 23, 2023

andrross commented Feb 17, 2023

andrross commented Feb 17, 2023

adnapibar commented Feb 23, 2023

kotwanikunal commented Feb 23, 2023

github-actions bot commented Feb 23, 2023

andrross Feb 23, 2023

github-actions bot commented Feb 23, 2023

opensearch-trigger-bot bot commented Feb 23, 2023

kotwanikunal commented Feb 23, 2023

kotwanikunal commented Feb 25, 2023

Include file cache stats in node stats api response #6333

Include file cache stats in node stats api response #6333

Conversation

adnapibar commented Feb 16, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Feb 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 17, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Feb 17, 2023 • edited Loading

Codecov Report

kotwanikunal Feb 17, 2023

Choose a reason for hiding this comment

adnapibar Feb 23, 2023

Choose a reason for hiding this comment

andrross commented Feb 17, 2023

andrross commented Feb 17, 2023

adnapibar commented Feb 23, 2023

kotwanikunal commented Feb 23, 2023

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

andrross Feb 23, 2023

Choose a reason for hiding this comment

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Feb 23, 2023

kotwanikunal commented Feb 23, 2023

kotwanikunal commented Feb 25, 2023

adnapibar commented Feb 16, 2023 •

edited

Loading

codecov-commenter commented Feb 17, 2023 •

edited

Loading