[Search Pipelines] Add request-scoped state shared between processors #9405

msfroh · 2023-08-16T23:29:43Z

To handle cases where multiple search pipeline processors need to share information, we will allocate a Map<String, Object> for the lifetime of the request and pass it to each processor to get/set values.

Description

To handle cases where multiple search pipeline processors need to share information, we will allocate a Map<String, Object> for the lifetime of the request and pass it to each processor to get/set values.

Related Issues

#6722

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-16T23:43:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22945/
CommitID: 39d7d25
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-17T00:09:20Z

Compatibility status:

Checks if related components are compatible with change f3957ea

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-17T00:41:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

URL: https://build.ci.opensearch.org/job/gradle-check/22946/
CommitID: 4d0d93e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-08-17T00:42:51Z

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (f7f3500) 71.37% compared to head (9ce4ca8) 71.28%.

Files	Patch %	Lines
...rch/pipeline/common/CollapseResponseProcessor.java	71.42%	7 Missing and 5 partials ⚠️
...ch/pipeline/common/helpers/SearchResponseUtil.java	69.69%	8 Missing and 2 partials ⚠️
...earch/search/pipeline/common/SearchRequestMap.java	80.00%	6 Missing and 3 partials ⚠️
...search/pipeline/common/ScriptRequestProcessor.java	40.00%	6 Missing ⚠️
...ch/pipeline/common/OversampleRequestProcessor.java	73.68%	3 Missing and 2 partials ⚠️
...pipeline/common/TruncateHitsResponseProcessor.java	80.76%	3 Missing and 2 partials ⚠️
...rg/opensearch/search/pipeline/common/BasicMap.java	90.90%	2 Missing ⚠️
...eline/common/SearchPipelineCommonModulePlugin.java	0.00%	1 Missing ⚠️
...h/search/pipeline/common/helpers/ContextUtils.java	75.00%	0 Missing and 1 partial ⚠️
...earch/pipeline/StatefulSearchRequestProcessor.java	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9405      +/-   ##
============================================
- Coverage     71.37%   71.28%   -0.09%     
+ Complexity    59113    59030      -83     
============================================
  Files          4893     4903      +10     
  Lines        277831   277979     +148     
  Branches      40367    40382      +15     
============================================
- Hits         198288   198146     -142     
- Misses        63042    63329     +287     
- Partials      16501    16504       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

...e-common/src/main/java/org/opensearch/search/pipeline/common/helpers/SearchResponseUtil.java

austintlee · 2023-08-21T20:07:38Z

@msfroh This looks really good. Let me know if there's anything I can do to help get this over the line. I can help with writing unit tests, e.g.

opensearch-trigger-bot · 2023-08-23T01:22:27Z

Compatibility status:

Checks if related components are compatible with change 60d272b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-23T01:37:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23250/
CommitID: 923b12d

github-actions · 2023-08-25T23:34:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23534/
CommitID: b995556
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-26T00:08:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23535/
CommitID: a9423d5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-26T05:46:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      3 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}
      1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"false"}}
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase

URL: https://build.ci.opensearch.org/job/gradle-check/23539/
CommitID: 3dff828
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

...eline-common/src/main/java/org/opensearch/search/pipeline/common/ScriptRequestProcessor.java

github-actions · 2023-12-05T00:28:19Z

❌ Gradle check result for 7b049b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

To handle cases where multiple search pipeline processors need to share information, we will allocate a Map<String, Object> for the lifetime of the request and pass it to each processor to get/set values. Signed-off-by: Michael Froh <froh@amazon.com>

Added "context_prefix" convention to scope variables to avoid collisions. Let script processor have access to the request context. Added more unit tests. Signed-off-by: Michael Froh <froh@amazon.com>

After realizing that we just need to keep the first hit for each group (since results are already sorted by the sort criteria), I think CollapseResponseProcessor might be worth including. Combining it with the oversample + truncate processors, it can provide a workaround for the lack of support for collapse + rescore. Signed-off-by: Michael Froh <froh@amazon.com>

Signed-off-by: Michael Froh <froh@amazon.com>

* Use default size in OversampleRequestProcessor if not specified. * Pass context to SearchPhaseResultsProcessor too. * Wrap context in its own class, in case we decide to add other fields to it in future. Signed-off-by: Michael Froh <froh@amazon.com>

Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-12-05T03:23:19Z

❕ Gradle check result for 9ce4ca8: UNSTABLE

TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…#9405) To handle cases where multiple search pipeline processors need to share information, we will allocate a context holder for the lifetime of the request and pass it to each processor to get/set values. To explain this behavior and benefit from it, this change also introduces three new processors: 1. The "oversample" request processor that increases "size", storing the original size in the context. 2. The "truncate" response processor that discards results after some number, by default using the original size before oversampling. 3. The "collapse" response processor offers similar behavior to a collapse query, discarding results that have a field value in common with a higher-scoring result. Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit c204585) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

austintlee · 2023-12-05T18:02:48Z

Awesome!

navneet1v · 2023-12-05T18:50:43Z

This is awesome. an awaited feature. :D

…#9405) (#11473) To handle cases where multiple search pipeline processors need to share information, we will allocate a context holder for the lifetime of the request and pass it to each processor to get/set values. To explain this behavior and benefit from it, this change also introduces three new processors: 1. The "oversample" request processor that increases "size", storing the original size in the context. 2. The "truncate" response processor that discards results after some number, by default using the original size before oversampling. 3. The "collapse" response processor offers similar behavior to a collapse query, discarding results that have a field value in common with a higher-scoring result. (cherry picked from commit c204585) Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#9405) To handle cases where multiple search pipeline processors need to share information, we will allocate a context holder for the lifetime of the request and pass it to each processor to get/set values. To explain this behavior and benefit from it, this change also introduces three new processors: 1. The "oversample" request processor that increases "size", storing the original size in the context. 2. The "truncate" response processor that discards results after some number, by default using the original size before oversampling. 3. The "collapse" response processor offers similar behavior to a collapse query, discarding results that have a field value in common with a higher-scoring result. Signed-off-by: Michael Froh <froh@amazon.com>

…opensearch-project#9405) To handle cases where multiple search pipeline processors need to share information, we will allocate a context holder for the lifetime of the request and pass it to each processor to get/set values. To explain this behavior and benefit from it, this change also introduces three new processors: 1. The "oversample" request processor that increases "size", storing the original size in the context. 2. The "truncate" response processor that discards results after some number, by default using the original size before oversampling. 3. The "collapse" response processor offers similar behavior to a collapse query, discarding results that have a field value in common with a higher-scoring result. Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

msfroh mentioned this pull request Aug 16, 2023

[Search Pipelines] Share state between multiple processors #6722

Closed

msfroh force-pushed the pipeline_state branch from 39d7d25 to 4d0d93e Compare August 17, 2023 00:01

austintlee reviewed Aug 21, 2023

View reviewed changes

...e-common/src/main/java/org/opensearch/search/pipeline/common/helpers/SearchResponseUtil.java Outdated Show resolved Hide resolved

msfroh force-pushed the pipeline_state branch from b995556 to a9423d5 Compare August 25, 2023 23:32

msfroh force-pushed the pipeline_state branch from a9423d5 to 3dff828 Compare August 26, 2023 04:49

msfroh marked this pull request as ready for review August 26, 2023 05:56

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19 and Rishikesh1159 as code owners August 26, 2023 05:56

reta reviewed Nov 20, 2023

View reviewed changes

...eline-common/src/main/java/org/opensearch/search/pipeline/common/ScriptRequestProcessor.java Show resolved Hide resolved

msfroh force-pushed the pipeline_state branch from bef6d68 to 7b049b0 Compare December 4, 2023 23:58

msfroh added 10 commits December 5, 2023 02:36

More work on stateful pipeline processors

5073026

Added "context_prefix" convention to scope variables to avoid collisions. Let script processor have access to the request context. Added more unit tests. Signed-off-by: Michael Froh <froh@amazon.com>

Add changelog entry

f627c68

Signed-off-by: Michael Froh <froh@amazon.com>

Suggested changes from @noCharger

c0637dc

Signed-off-by: Michael Froh <froh@amazon.com>

Add missing Javadoc and extra InternalSearchResponse case

dc22f61

Signed-off-by: Michael Froh <froh@amazon.com>

Encapsulate map in PipelinedRequestContext

545cc0a

Signed-off-by: Michael Froh <froh@amazon.com>

Rename PipelineRequestContext -> PipelineProcessingContext

a7427f9

Signed-off-by: Michael Froh <froh@amazon.com>

Move BasicMap and SearchRequestMap to parent package

9ce4ca8

Signed-off-by: Michael Froh <froh@amazon.com>

msfroh force-pushed the pipeline_state branch from 7b049b0 to 9ce4ca8 Compare December 5, 2023 02:36

reta approved these changes Dec 5, 2023

View reviewed changes

msfroh merged commit c204585 into opensearch-project:main Dec 5, 2023
29 checks passed

reta added backport 2.x Backport to 2.x branch v3.0.0 Issues and PRs related to version 3.0.0 labels Dec 5, 2023

opensearch-trigger-bot bot mentioned this pull request Dec 5, 2023

[Backport 2.x] [Search Pipelines] Add request-scoped state shared between processors #11473

Merged

msfroh mentioned this pull request Dec 14, 2023

Add documentation for collapse, oversample, truncate_hits processors opensearch-project/documentation-website#5881

Merged

1 task

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Search Pipelines] Add request-scoped state shared between processors #9405

[Search Pipelines] Add request-scoped state shared between processors #9405

msfroh commented Aug 16, 2023 •

edited

Loading

github-actions bot commented Aug 16, 2023

opensearch-trigger-bot bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

codecov bot commented Aug 17, 2023 •

edited

Loading

austintlee commented Aug 21, 2023

opensearch-trigger-bot bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 26, 2023

github-actions bot commented Aug 26, 2023

github-actions bot commented Dec 5, 2023

github-actions bot commented Dec 5, 2023

austintlee commented Dec 5, 2023

navneet1v commented Dec 5, 2023

[Search Pipelines] Add request-scoped state shared between processors #9405

[Search Pipelines] Add request-scoped state shared between processors #9405

Conversation

msfroh commented Aug 16, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 17, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 17, 2023 • edited Loading

Codecov Report

austintlee commented Aug 21, 2023

opensearch-trigger-bot bot commented Aug 23, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 5, 2023

github-actions bot commented Dec 5, 2023

austintlee commented Dec 5, 2023

navneet1v commented Dec 5, 2023

msfroh commented Aug 16, 2023 •

edited

Loading

codecov bot commented Aug 17, 2023 •

edited

Loading