Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG)

[austintlee]

Description

Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG)

Issues Resolved

#1150

Check List

• [x ] New functionality includes testing.
  • [x ] All tests pass
• [x ] New functionality has been documented.
  • [x ] New functionality has javadoc added
• [x ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ylwu-amzn]

Thanks for the quick action. Github CI failed, is this PR ready for review?

[austintlee]

Oh I didn't spot the issue with the code coverage when I built locally. It's definitely ready for review so I would really appreciate it if you could take a look and provide feedback especially around the placement of the code components and how this fits into ml-commons. I'll address code coverage in the meantime. Thanks in advance!

[austintlee]

731 tests completed, 1 failed, 58 skipped
Tests with failures:
 - org.opensearch.ml.action.prediction.PredictionITTests.testPredictionWithDataFrame_FitRCF

> Task :opensearch-ml-plugin:test FAILED

This seems unrelated to my changes.

[ylwu-amzn]

This PR will add search pipeline processor to ml-commons. I know @navneet1v built some PoC for search pipeline processor in neural-search plugin. I think that's a better place to hold these processors as they are more related with search part. @navneet1v Do you have any suggestion on this? Can you also help review this PR? I think there are some overlap with your PoC of summarization and QA processor.

[austintlee]

This PR will add search pipeline processor to ml-commons. I know @navneet1v built some PoC for search pipeline processor in neural-search plugin. I think that's a better place to hold these processors as they are more related with search part. @navneet1v Do you have any suggestion on this? Can you also help review this PR? I think there are some overlap with your PoC of summarization and QA processor.

RAG is not tied to neural (knn) searches, though.

[navneet1v]

@austintlee I understand RAG is not tied up to K-NN. But from a use-case and cohesiveness of the features is what @ylwu-amzn want to say its better suited to host this feature in Neural Search plugin.

The way I think about ML-Commons as a plugin is providing core components/features to do ML related workloads and Neural Search plugin uses the capabilities of ML Plugin to improve Search example is in Semantic Search we used ML predict API to convert text to embeddings during ingestion and Query time.

Now on your code as you are adding a processors which is mainly going to be used in Search(as you are using a Search Pipeline) I think we should move this feature to Neural Plugin. I am open to suggestion.

@ylwu-amzn , @austintlee Please add your thoughts. If we want to continue this discussion on RFC I am happy to do that.

[austintlee]

@navneet1v I disagree.

The example of text embeddings makes perfect sense for the neural search plugin (NSP). But the scope and use cases for RAG extend beyond what I think we want for the NSP. I don't think we disagree on this point. I definitely see where you are coming from and yes, RAG is closely associated with semantic search, but I just do not think that conceptually, it belongs in the NSP.

Using the same analogy you made, I can also make the case that just as Search Pipeline in OpenSearch core comes with default search processors (filter, script, etc), we can have RAG search processors that come with ML core (ml-commons). We are really debating how "core" this component is in the grand scheme of things. If you have not, please take a look at the discussion we had in #1150 where we suggest to put this functionality in ml-commons and the outcome of that discussion is the "feature/conversation" feature branch and this PR and #1196 .

Now on your code as you are adding a processors which is mainly going to be used in Search(as you are using a Search Pipeline) I think we should move this feature to Neural Plugin.

Again, "search" is really what we are all about. That's the core. Everything else exists to make search better, no?

[navneet1v]

@navneet1v I disagree.

The example of text embeddings makes perfect sense for the neural search plugin (NSP). But the scope and use cases for RAG extend beyond what I think we want for the NSP. I don't think we disagree on this point. I definitely see where you are coming from and yes, RAG is closely associated with semantic search, but I just do not think that conceptually, it belongs in the NSP.

Using the same analogy you made, I can also make the case that just as Search Pipeline in OpenSearch core comes with default search processors (filter, script, etc), we can have RAG search processors that come with ML core (ml-commons). We are really debating how "core" this component is in the grand scheme of things. If you have not, please take a look at the discussion we had in #1150 where we suggest to put this functionality in ml-commons and the outcome of that discussion is the "feature/conversation" feature branch and this PR and #1196 .

Now on your code as you are adding a processors which is mainly going to be used in Search(as you are using a Search Pipeline) I think we should move this feature to Neural Plugin.

Again, "search" is really what we are all about. That's the core. Everything else exists to make search better, no?

I was not aware of the big scope of changes that are happening. I have to read this in more details and get back to you.

[navneet1v]

@navneet1v I disagree.

The example of text embeddings makes perfect sense for the neural search plugin (NSP). But the scope and use cases for RAG extend beyond what I think we want for the NSP. I don't think we disagree on this point. I definitely see where you are coming from and yes, RAG is closely associated with semantic search, but I just do not think that conceptually, it belongs in the NSP.

Using the same analogy you made, I can also make the case that just as Search Pipeline in OpenSearch core comes with default search processors (filter, script, etc), we can have RAG search processors that come with ML core (ml-commons). We are really debating how "core" this component is in the grand scheme of things. If you have not, please take a look at the discussion we had in #1150 where we suggest to put this functionality in ml-commons and the outcome of that discussion is the "feature/conversation" feature branch and this PR and #1196 .

Now on your code as you are adding a processors which is mainly going to be used in Search(as you are using a Search Pipeline) I think we should move this feature to Neural Plugin.

Again, "search" is really what we are all about. That's the core. Everything else exists to make search better, no?

I read the proposal and will directly add comments on the RFC.

[austintlee]

@navneet1v @jngz-es @ylwu-amzn Could we get some traction on this PR please? Thanks!

[saratvemulapalli]

@austintlee I'll spend sometime today and tomorrow to review this one and #1196

[austintlee]

@saratvemulapalli @ylwu-amzn @navneet1v

I would really appreciate some feedback on this PR. I have more PRs that depend on this. Thank you.

[ylwu-amzn]

@austintlee , we just backported some bug fix from 2.9 branch to 2.x branch. Can you help sync 2.x branch to feature/conversation branch and rebase your own forked branch and refresh your PR?
I think the invalid json issue you mentioned #1215 (comment) will be addressed with latest 2.x code

[austintlee]

@austintlee , we just backported some bug fix from 2.9 branch to 2.x branch. Can you help sync 2.x branch to feature/conversation branch and rebase your own forked branch and refresh your PR? I think the invalid json issue you mentioned #1215 (comment) will be addressed with latest 2.x code

Henry took care of the sync. I will rebase and update this PR.

[austintlee]

In the latest commit, I removed use of MachineLearningClient (from opensearch-ml-client) in opensearch-ml-search-processors. With this dependency removed, all build issues with publishToMavenLocal and jarHell are gone. Thanks @reta for your help.

@saratvemulapalli

[codecov]

Codecov Report

Merging #1195 (1dbd99d) into feature/conversation (3319123) will decrease coverage by 0.19%.
Report is 1 commits behind head on feature/conversation.
The diff coverage is 73.86%.

❗ Current head 1dbd99d differs from pull request most recent head 13a5cfb. Consider uploading reports for the commit 13a5cfb to get more accurate results

@@                    Coverage Diff                     @@
##             feature/conversation    #1195      +/-   ##
==========================================================
- Coverage                   79.04%   78.86%   -0.19%     
- Complexity                   2148     2190      +42     
==========================================================
  Files                         172      186      +14     
  Lines                        8672     8870     +198     
  Branches                      870      885      +15     
==========================================================
+ Hits                         6855     6995     +140     
- Misses                       1422     1466      +44     
- Partials                      395      409      +14     

Flag	Coverage Δ
ml-commons	78.86% <73.86%> (-0.19%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ing/generative/GenerativeQAProcessorConstants.java	0.00% <0.00%> (ø)
...answering/generative/GenerativeSearchResponse.java	25.00% <25.00%> (ø)
...ng/generative/ext/GenerativeQAParamExtBuilder.java	35.29% <35.29%> (ø)
...es/questionanswering/generative/llm/LlmIOUtil.java	50.00% <50.00%> (ø)
...nswering/generative/ext/GenerativeQAParamUtil.java	66.66% <66.66%> (ø)
...swering/generative/ext/GenerativeQAParameters.java	70.00% <70.00%> (ø)
...rg/opensearch/ml/plugin/MachineLearningPlugin.java	97.34% <71.42%> (-1.73%)	⬇️
...nerative/client/MachineLearningInternalClient.java	72.41% <72.41%> (ø)
...ring/generative/GenerativeQAResponseProcessor.java	82.50% <82.50%> (ø)
...ering/generative/GenerativeQARequestProcessor.java	87.50% <87.50%> (ø)
... and 5 more

... and 3 files with indirect coverage changes

[ylwu-amzn]

Force merge first to unblock you. We may need another round of review and get two approvals when merge to 2.x.