Conversational Memory for GenAI Apps

[HenryL27]

Description

Implements conversational memory API using OpenSearch index

Potential organizational tweaks - feedback requested:

• move Actions from conversational/action/ to plugin/action/: keeping all the actions together (ofc the rest actions go to the rest directory)
• rename ConversationalMemoryHandler to ConversationalMemory: to draw a clearer distinction between memory in the abstract and the physical memory - might be more obvious that multiple backends could be here
• move conversational/index/ to conversational/storage/index: again, to allow room for additional backend CM storage options (DDB, Redis, etc)
• rename ConversationMeta to Conversation: is the Meta part redundant?

Issues Resolved

#1150

Design

We provide an API for handling conversational memory. First, I will define conversational memory. Then I will provide a description of the transport-layer implementation. Finally, a description of the storage-layer implementation.

Conversational Memory common/.../conversational/

The purpose of conversational memory is to store context of conversations had between an end-user and a LLM. Chatbots need to remember what has been said in a conversation prior to any new interactions; otherwise their statements will be completely ungrounded - for example, they might have no idea what a pronoun is referring to and be unable to reason effectively.

We consider two objects: a Conversation and an Interaction. A conversation is, essentially, a list of interactions. An interaction is a pair of messages, an "input" from the end-user and a "response" from the LLM. We add a bunch of additional data to these objects, to enable better analytics, explainability, etc.

---

Conversations also know about:

• when they began and when the last time they were hit, time-wise
• how long they are
• who they belong to

They also have an optional name field, so that an end-user can easily pick up a conversation from a moth or so ago.

---

Interactions, in addition the the input-response pair, know about:

• when they were created (this references when it was added to the conversation - i.e. completed - not when the interaction was initiated)
• what conversation they belong to
• what prompt was used (maybe rename to prompt template? use when there are processing steps between the end-user and the LLM generation)
• what agent was used (a conversation could conceivably interact with multiple agents that do different things)
• and an arbitrary 'metadata' field that apps can use to write whatever additional data they want.

Transport Layer conversational-memory/.../action/

We implement 5 OpenSearch Actions for handling conversations: CreateConversation, CreateInteraction, GetConversations, GetInteractions, and DeleteConversation. Each Action follows the (as far as I can tell) standard OpenSearch pattern of [Action, Request, Response, TransportAction, RestAction].

CreateConversation: Given an optional name for the conversation, create a new conversation object and return the id. If the security plugin is enabled and the user is a secure user (not admin) then the username also gets written into the conversation object.

GetConversations: Given nothing, return the list of conversations that this user has access to. If no security or admin user, this is the list of all conversations, sorted by recency of the last time they were added to. Note that the conversations returned do not include the interactions that go with. This is paginated - by default we return 10 conversations but this can be bumped. If there are more conversations than the 10 returned we also return a pagination token for the next page (which at the moment is just a "from" position integer because of how OS does pagination in search requests)

GetInteractions: Given a conversation id (retrieved either from CreateConversation or GetConversation), return all the interactions that belong to that conversation, sorted most recent to oldest. This is also paginated in the same way as GetConversations. If the user does not have access to the conversation they are requesting, we don't return any interactions.

CreateInteraction: Write an interaction into memory. This takes as input everything the Interaction needs to know about: input, response, prompt, conversation id, agent, metadata. The timestamp is generated here. Also, update the corresponding conversation object - bump the number of interactions, update the most recent hit time. Again, if the user has no access to the conversation they're trying to add to, we disallow this. Return the id of the new interaction.

DeleteConversation: Given a conversation id, delete it and all of its interactions from memory. Don't if the user doesn't have access.

The "TransportAction" versions of these actions use a ConversationalMemoryHandler, which is a storage-layer interface that performs all of these operations. I've implemented an OpenSearch version of ConversationalMemoryHandler, but it should be possible to write implementations for other databases as well.

Storage Layer conversational-memory/.../index/

We use OpenSearch as the storage for conversational memory. It seemed natural, since we're in OpenSearch. We use 2 indices: one for the high-level conversation objects (which we call "ConversationMeta") and one for interactions. The bulk of the work here is translating the Transport Layer operations into OpenSearch client operations.

CreateConversation: this just becomes an index request, creating a new "ConversationMeta" object in the .plugins-ml-conversation-meta index. If security is enabled, we read the user info from the ThreadContext.

GetConversations: depending on whether there is security involved, this is a Search request with either a TermQuery (matching user from ThreadContext to user in indexed documents) or a MatchAllQuery. Also, use OpenSearch to do the sorting too.

GetInteractions: first check if the user has access to the conversation by Getting the ConversationMeta object from the .plugins-ml-conversation-meta index, then (if yes) execute a TermQuery search request looking for Interactions with the correct conversation id. Use OS to sort by time.

CreateInteraction: First, check that the user has access to the conversation with a get request. Then, (if yes) update the ConversationMeta object with the new time and a bump to the conversation length. Also, index a new Interaction object.

DeleteConversation: First, check that the user has access to the conversation with a get request. Then, (if yes), delete the ConversationMeta from its index. Also, gather all of the interactions in that conversation with consecutive getInteractions calls, and finally Bulk delete them all.

Index Schemae

ConversationMeta Index .plugins-ml-conversation-meta

{
    "_meta": {
        "schema_version": 1
    },
    "properties": {
        "name": {"type": "keyword"},
        "create_time": {"type": "date", "format": "strict_date_time||epoch_millis"},
        "user": {"type": "keyword"}
    }
}

Interactions Index .plugins-ml-conversation-interactions

{
    "_meta": {
        "schema_version": 1
    },
    "properties": {
        "conversation_id": {"type": "keyword"},
        "create_time": {"type": "date", "format": "strict_date_time||epoch_millis"},
        "input": {"type": "text"},
        "prompt_template": {"type": "text"},
        "response": {"type": "text"},
        "origin": {"type": "text"},
        "additional_info": {"type": "text"}
    }
}

Security

For now, we keep conversations at the private resource-level security mode, i.e. only the owner and admin have access to a conversation. At the conversation level, we write the username of the user who created it (pulling from the variable in thread context created by the security plugin). See the storage layer details for how this interacts with the various actions we're introducing. There are a lot of hard problems when you start trying to bring conversations to a restricted access level - unlike models, the user can directly add information to a conversation, some of which may be private. So for now we leave all conversations at private access level.

Security Concerns

Theoretically, the user can write whatever they like into this storage. Credit card info, social security numbers, personal medical information, etc. Hence the blanket "private mode" over all conversations. One potential issue is that the admin's username is null - their conversations will not have a 'user' field. We list conversations by a TermQuery (over the user) or MatchAllQuery depending on who's doing the asking, but if the TermQuery returns documents that don't have that field then the admin's conversationMetas are public.

When deleting a conversation, it is possible for the conversationMeta to be deleted but not all of the interactions. In this case, we deny anyone access to the orphaned interactions, but they stick around forever (or at least until the admin goes and manually cleans them up). If you can automate that particular fail case then you can probably cause some problems.

I'll also add that the logic for deleting all the interactions of a conversation is fairly complicated, so I'd like someone to look it over.

Sample Rest API calls

> curl localhost:9200/_plugins/_ml/memory/conversation
{"conversations":[]}
> curl -X POST localhost:9200/_plugins/_ml/memory/conversation -H "Content-Type: application/json" -d '{"name": "some name"}'
{"conversation_id":"W3omQooBf26FN_SBEFXz"}
> curl localhost:9200/_plugins/_ml/memory/conversation                                                                       
{"conversations":[{"conversation_id":"W3omQooBf26FN_SBEFXz","create_time":"2023-08-29T16:35:05.586393202Z","name":"some name"}]}
> curl -X POST localhost:9200/_plugins/_ml/memory/conversation
{"conversation_id":"XHonQooBf26FN_SBAVXp"}
> curl localhost:9200/_plugins/_ml/memory/conversation        
{"conversations":[{"conversation_id":"XHonQooBf26FN_SBAVXp","create_time":"2023-08-29T16:36:07.271016443Z","name":""},{"conversation_id":"W3omQooBf26FN_SBEFXz","create_time":"2023-08-29T16:35:05.586393202Z","name":"some name"}]}
> curl -X POST localhost:9200/_plugins/_ml/memory/conversation/XHonQooBf26FN_SBAVXp -H "Content-Type: application/json" -d '{"input": "question", "prompt_template": "template about bees or something", "response": "answer to the question", "origin": "botanical index pipeline", "additional_info": "{\"documents\": [doc1, doc2,...]}"}'
{"interaction_id":"XXoqQooBf26FN_SBglWE"}
> curl -X POST localhost:9200/_plugins/_ml/memory/conversation/XHonQooBf26FN_SBAVXp -H "Content-Type: application/json" -d '{"input": "follow-up question", "prompt_template": "template about flowers this time", "response": "answer to the new question", "origin": "botanical index pipeline", "additional_info": "{\"documents\": [doc1, doc2,...]}"}'
{"interaction_id":"XnorQooBf26FN_SBcFXT"}
> curl localhost:9200/_plugins/_ml/memory/conversation/XHonQooBf26FN_SBAVXp
{"interactions":[{"conversation_id":"XHonQooBf26FN_SBAVXp","interaction_id":"XnorQooBf26FN_SBcFXT","create_time":"2023-08-29T16:40:57.805989175Z","input":"follow-up question","prompt_template":"template about flowers this time","response":"answer to the new question","origin":"botanical index pipeline","additional_info":"{\"documents\": [doc1, doc2,...]}"},{"conversation_id":"XHonQooBf26FN_SBAVXp","interaction_id":"XXoqQooBf26FN_SBglWE","create_time":"2023-08-29T16:39:56.681481718Z","input":"question","prompt_template":"template about bees or something","response":"answer to the question","origin":"botanical index pipeline","additional_info":"{\"documents\": [doc1, doc2,...]}"}]}
> curl -X DELETE localhost:9200/_plugins/_ml/memory/conversation/XHonQooBf26FN_SBAVXp
{"success":true}
> curl localhost:9200/_plugins/_ml/memory/conversation/XHonQooBf26FN_SBAVXp          
{"interactions":[]}
> curl localhost:9200/_plugins/_ml/memory/conversation
{"conversations":[{"conversation_id":"W3omQooBf26FN_SBEFXz","create_time":"2023-08-29T16:35:05.586393202Z","name":"some name"}]}

Future work (not this PR)

• singular GetConversation API (and I guess GetInteraction)
• role-based access control
• _search endpoint

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ylwu-amzn]

Thanks for contributing. Are you going to keep copyright as Aryn?

[HenryL27]

just changed em to

/*
 * Copyright 2023 Aryn
 * Copyright OpenSearch Contributors
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

[HenryL27]

@ylwu-amzn I'm not sure what the right thing to do is here, since I'm not sure I can call myself an OpenSearch contributor just yet. This was @mashah 's recommendation

[ylwu-amzn]

@HenryL27 , I think you definitely are OpenSearch contributor. I will wait to check if any other guys from community has concerns or not.

I know @reta contributed a lot to OpenSearch. Do you have any concern for adding Copyright 2023 Aryn from community perspective?

[reta]

@HenryL27 you are certainly contributor (DCO) which is covered by this generic statement:

 * Copyright OpenSearch Contributors

We do not include individual contributors (individuals or organizations) afaik

[austintlee]

Let's discuss the license header separately. I understand technically, it's part of the PR, but the more important thing here is the code. Do we have reviewers identified for this?

[ylwu-amzn]

@austintlee Agree, we can go ahead and review code first. @jngz-es is reviewing now. I will also review the code.

[hyandell]

Hi there, +1 from the AWS Open Source Program Office - many explicit copyright statements in each file incur a lot of tech debt.

OpenSearch is following the style of other projects, like the Apache Software Foundation's, to summarize copyrights belonging to the project contributors rather than list them in each source file. Source control provides the information on who has touched what should that become required. Happy to discuss with the Aryn open source leaders/program office if needed.

[mashah]

Henri --

There are two scenarios to consider:

1. Company X has a feature or capability that they developed internally and would like to contribute it back to the OpenSearch community.

2. Contributors from company X are updating existing files, incrementally improving existing features, or collaborating with multiple contributors across organizations on new features.

In scenario #1, we believe it's beneficial to keep the lineage of the work. It incentivizes companies to contribute back to the community that which they've already invested in and solely own. Lineage of contribution is kept intact and easily distinguished.

In scenario #2, we agree listing all the companies and their copyright would be messy.

Our opinion is that for conversational memory, we are in scenario #1. I also believe that we've followed this model with Elastic and Open Source Connections and acknowledged contributions as such in OpenSearch.

[mashah]

Also,

I'm not sure that I understand the "tech debt" statement. Copyright statements are not code and incur no overhead.

[HenryL27]

example interaction using this:

hmlin@Henrys-MBP OSContainer % curl -X POST localhost:9200/_plugins/ml/conversational/memory
{"conversationId":"deGW4YkByJOQvDkDZ3Bm"}
hmlin@Henrys-MBP OSContainer % curl localhost:9200/_plugins/ml/conversational/memory
{"conversations":[{"conversationId":"deGW4YkByJOQvDkDZ3Bm","createTime":"2023-08-10T22:34:37.788196552Z","lastInteractionTime":"2023-08-10T22:34:37.78820501Z","numInteractions":0,"name":"","user":null}]}
hmlin@Henrys-MBP OSContainer % curl -X POST localhost:9200/_plugins/ml/conversational/memory/deGW4YkByJOQvDkDZ3Bm -H "Content-Type: application/json" -d '{"input": "test input 0", "prompt": "answer the question: [QUESTION]", "response": "test response 0", "agent": "MEEE!!!", "attributes": "{\"some\":\"data\"}" }'
{"interactionId":"duGa4YkByJOQvDkD9XCp"}
hmlin@Henrys-MBP OSContainer % curl -X POST localhost:9200/_plugins/ml/conversational/memory/deGW4YkByJOQvDkDZ3Bm -H "Content-Type: application/json" -d '{"input": "test input 1", "prompt": "answer the question: [QUESTION]", "response": "test response 1", "agent": "MEEE!!!", "attributes": "{\"some\":\"other data\"}" }'
{"interactionId":"d-Gb4YkByJOQvDkDWHCy"}
hmlin@Henrys-MBP OSContainer % curl localhost:9200/_plugins/ml/conversational/memory
{"conversations":[{"conversationId":"deGW4YkByJOQvDkDZ3Bm","createTime":"2023-08-10T22:34:37.788196552Z","lastInteractionTime":"2023-08-10T22:40:01.710334635Z","numInteractions":2,"name":"","user":null}]}
hmlin@Henrys-MBP OSContainer % curl localhost:9200/_plugins/ml/conversational/memory/deGW4YkByJOQvDkDZ3Bm
{"interactions":[{"conversationId":"deGW4YkByJOQvDkDZ3Bm","interactionId":"d-Gb4YkByJOQvDkDWHCy","timestamp":"2023-08-10T22:40:01.710334635Z","input":"test input 1","prompt":"answer the question: [QUESTION]","response":"test response 1","agent":"MEEE!!!","attributes":"{\"some\":\"other data\"}"},{"conversationId":"deGW4YkByJOQvDkDZ3Bm","interactionId":"duGa4YkByJOQvDkD9XCp","timestamp":"2023-08-10T22:39:36.236766221Z","input":"test input 0","prompt":"answer the question: [QUESTION]","response":"test response 0","agent":"MEEE!!!","attributes":"{\"some\":\"data\"}"}]}
hmlin@Henrys-MBP OSContainer % curl -X DELETE localhost:9200/_plugins/ml/conversational/memory/deGW4YkByJOQvDkDZ3Bm
{"success":true}
hmlin@Henrys-MBP OSContainer % curl localhost:9200/_plugins/ml/conversational/memory/deGW4YkByJOQvDkDZ3Bm             
{"interactions":[]}
hmlin@Henrys-MBP OSContainer % curl localhost:9200/_plugins/ml/conversational/memory                                  
{"conversations":[]}

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (feature/conversation@7adb3d0). Click here to learn what that means.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##             feature/conversation    #1196   +/-   ##
=======================================================
  Coverage                        ?   79.04%           
  Complexity                      ?     2149           
=======================================================
  Files                           ?      172           
  Lines                           ?     8672           
  Branches                        ?      870           
=======================================================
  Hits                            ?     6855           
  Misses                          ?     1422           
  Partials                        ?      395           

Flag	Coverage Δ
ml-commons	79.04% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jngz-es]

Comparing with another pr #1195, can we keep the same style for constructors, get/set members etc. to use lombok?

[ylwu-amzn]

@HenryL27 , @austintlee Considering this is experimental feature , and your team plan to release in 2.10 which is very tight. I don't have enough time to review this PR. Approve this PR first to unblock you. For GA release, we may need another round of review.

[austintlee]

@HenryL27 , @austintlee Considering this is experimental feature , and your team plan to release in 2.10 which is very tight. I don't have enough time to review this PR. Approve this PR first to unblock you. For GA release, we may need another round of review.

Ack. Can you please merge this PR? Thanks.