POC Coordinator node stats on search request #1

buddharajusahil · 2023-06-15T17:44:15Z

This change adds and tracks node coordinator stats in the SearchStats object that is returned when prompted for node stats.

This does not include testing, just the basic functionality.

github-actions · 2023-06-15T17:45:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 585f33c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

sgup432 · 2023-06-16T07:22:44Z

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

@@ -56,6 +58,8 @@
 */
 public class SearchStats implements Writeable, ToXContentFragment {


We also need to make change in writeTo(StreamOutput out), Stats(StreamInput in). As these methods are used by nodes to pass stats between each other.

server/src/main/java/org/opensearch/action/search/SearchCoordinatorStats.java

sgup432 · 2023-06-16T09:20:26Z

server/src/main/java/org/opensearch/action/search/SearchRequestOperationsListener.java

+                try {
+                    queryPhaseEnd = System.nanoTime();
+                    queryTotal = TimeUnit.NANOSECONDS.toMillis(queryPhaseEnd - queryPhaseStart);
+                    listener.onQueryPhaseEnd(context);


Just pass the took time in onQueryPhaseEnd. Inside stats you will implement the same way like in shard level listener which is thread safe.
Same for below.

sgup432 · 2023-06-16T09:28:16Z

server/src/main/java/org/opensearch/action/search/SearchRequestOperationsListener.java

+    void onExpandSearchPhaseStart(SearchPhaseContext context);
+    void onExpandSearchPhaseFailure(SearchPhaseContext context);
+    void onExpandSearchPhaseEnd(SearchPhaseContext context);
+    void addQueryTotal(long queryTotal);


As mentioned earlier, don't need this. Just use phaseEnd and pass took time.
Similar for others.

server/src/main/java/org/opensearch/action/CoordinatorStats.java

server/src/main/java/org/opensearch/action/admin/indices/stats/CommonStats.java

github-actions · 2023-06-16T22:19:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 6296ff0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-17T00:31:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: f92f0c8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…tenerExecutor class inside of AbstractSearchAsync

github-actions · 2023-06-17T18:12:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 2a5de27
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-17T18:26:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: c73bd57
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java

server/src/main/java/org/opensearch/action/search/SearchCoordinatorStats.java

server/src/main/java/org/opensearch/action/search/SearchRequestOperationsListener.java

server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java

server/src/main/java/org/opensearch/action/search/SearchCoordinatorStats.java

github-actions · 2023-06-19T20:28:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 2aed8d0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…ecutor

github-actions · 2023-06-19T21:36:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 77e941e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-20T16:21:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 03d3902
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-20T21:08:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: eb501ba
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-20T22:21:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 5ed6c14
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-20T22:54:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: f712928
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…henFetchAsyncAction

github-actions · 2023-06-22T22:13:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 65fe65f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-22T23:44:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 8b13f34
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-23T00:10:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: af445a0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-23T19:04:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 365db4f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-23T21:32:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 54950bd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/test/java/org/opensearch/action/search/AbstractSearchAsyncActionTests.java

…rStats

github-actions · 2023-06-26T22:45:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: b6ab3ea
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-26T23:03:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 222059b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-27T16:13:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 1bae299
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-27T18:09:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 32a7adf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-27T23:47:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: ff8d974
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-29T17:26:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 8b3ba56
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-29T18:27:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 4e8e4d2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

.idea/runConfigurations/Debug_OpenSearch.xml

server/src/main/java/org/opensearch/action/CoordinatorStats.java

server/src/main/java/org/opensearch/action/admin/indices/stats/CommonStats.java

server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java

server/src/main/java/org/opensearch/common/metrics/CounterMetric.java

server/src/main/java/org/opensearch/common/metrics/MeanMetric.java

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

server/src/test/java/org/opensearch/action/search/AbstractSearchAsyncActionTests.java

Signed-off-by: sahil <61558528+buddharajusahil@users.noreply.github.com>

github-actions · 2023-06-30T18:56:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 50c6be3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

POC Coordinator node stats on search request

585f33c

sgup432 reviewed Jun 16, 2023

View reviewed changes

Fixed simple issues

6296ff0

Fixed bug in last commit

f92f0c8

Changed onPhaseEnd methods to have tookTime passed in by adding a Lis…

2a5de27

…tenerExecutor class inside of AbstractSearchAsync

Made small change to SearchOperationListenerExecutor class

c73bd57

sgup432 reviewed Jun 19, 2023

View reviewed changes

fixed minor issues

2aed8d0

modified SearchPhase abstract class, removed SearchRequestOperationEx…

77e941e

…ecutor

Added SearchDFS and CanMatch phase metrics

03d3902

Added SearchRequestOperationsListener UTs

eb501ba

Added Search Coordinator Stats UTs

5ed6c14

Added Coordinator Stat Testing to SearchStatsTests

f712928

Added tests for canMatch and Dfs query, added multi thread for QueryT…

65fe65f

…henFetchAsyncAction

Wrote multi thread testing for DFS query phase

8b13f34

Added fetch multithreading testing

af445a0

refactored multi thread tests

365db4f

sahil buddharaju added 2 commits June 23, 2023 14:19

Major refactoring changes in unit tests

85024eb

Applied formatting

54950bd