[Backport 2.x] onShardResult and onShardFailure are executed on one shard causes opensearch jvm crashed

[kkewwei]

Backport from #12158

[github-actions]

❌ Gradle check result for 1cda2b3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change e449b92

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

✅ Gradle check result for e449b92: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.13%. Comparing base (0dd892c) to head (e449b92).
Report is 6 commits behind head on 2.x.

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #12612      +/-   ##
============================================
- Coverage     71.28%   71.13%   -0.15%     
+ Complexity    60145    60031     -114     
============================================
  Files          4957     4957              
  Lines        282799   282803       +4     
  Branches      41409    41410       +1     
============================================
- Hits         201591   201181     -410     
- Misses        64189    64625     +436     
+ Partials      17019    16997      -22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.