OpenSearch crashes on closed client connection before search reply when total ops higher compared to expected

[reta]

Signed-off-by: Andriy Redko andriy.redko@aiven.io

Description

In some circumstances, when running a search operation, if the client TCP connection is closed before the search operation is completed, opensearch crashes with:

[2022-07-20T17:51:36,145][ERROR][org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler] fatal error in thread [opensearch[mycluster-elasticsearch-3][search][T#8]], exiting
java.lang.AssertionError: unexpected higher total ops [15] compared to expected [14]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:465) ~[opensearch-1.3.4.jar:1.3.4]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$2(AbstractSearchAsyncAction.java:303) ~[opensearch-1.3.4.jar:1.3.4]

Issues Resolved

Closes #3962

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/1492/
• CommitID: addd0a8

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.64%. Comparing base (fbe93d4) to head (addd0a8).
Report is 2980 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4143      +/-   ##
============================================
+ Coverage     70.55%   70.64%   +0.08%     
- Complexity    57013    57093      +80     
============================================
  Files          4603     4603              
  Lines        274478   274482       +4     
  Branches      40201    40201              
============================================
+ Hits         193658   193905     +247     
+ Misses        64511    64336     -175     
+ Partials      16309    16241      -68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbonf]

will this be backported to 1.3 please?

[reta]

will this be backported to 1.3 please?

Yes, on it

[reta]

@dbonf would you be able to give it a try on 1.3.5-SNAPSHOT? thank you!

[dbonf]

@reta sure, do I need to compile it or is it already available somewhere?

[reta]

@reta sure, do I need to compile it or is it already available somewhere?

Good question, I think if you could compile (once #4148 is in), would be awesome, I am not 100% sure we publish snapshots

[dbonf]

@reta I managed to build from reta:backport/backport-4143-to-1.3 and the tests are promising, no crashes