Storing Chat History along with Image or Image Reference using LangChain JS

[hemantchaudhari2006]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const memory = new BufferMemory({
      chatHistory: new DynamoDBChatMessageHistory({
          conversationId,
          userId
      }),
      memoryKey: 'chat_history',
      returnMessages: true
  });

const llm = new BedrockChat({
      model: selectedModelId,
      temperature: 0.0,
      top_p: 0.9,
      streaming: true
  });

  const humanMessages = [];
  humanMessages.push({
      type: 'text',
      text: '{input}'
  });

  if (params.hasOwnProperty('Images') && Array.isArray(params['Images'])) {
      for (const imageIndex in params['Images']) {
          const base64String = await this.getImageBase64Data(userId + '/' + params['Images'][imageIndex]);
          humanMessages.push({
              type: 'image_url',
              image_url: {
                  url: `data:image/jpeg;base64,${base64String}`
              }
          });
      }
  }

  const promptTemplate = ChatPromptTemplate.fromMessages([
      new SystemMessage(instruction),
      new MessagesPlaceholder({ variableName: 'chat_history', optional: true }),
      new HumanMessage({ content: humanMessages })
  ]);

  const llmChain = new ConversationChain({
      memory,
      prompt: promptTemplate,
      llm
  });

  await llmChain.stream(
      {
          input: params['Text']
      },
      {
          callbacks: [
              {
                  handleLLMNewToken: async (data) => {
                      console.log(data);
                  }
              }
          ]
      }
  );

Description

I am using below example from the link to store chat history in dynamodb using langchain JS.
https://js.langchain.com/docs/integrations/memory/dynamodb

It is working fine, but I found that it does not store anything about image which I attached with one of my question. It just stores text, not image url or reference.

Actually, I am storing my images in S3 and collecting data from there to convert to base64 as mentioned above. Chat is giving correct answer as well for the image. The only problem is, it does not store anything for image in dynamodb, only stores text (my question). I want to have image reference with the question in the database, so that I can load chat history later on along with image data.

System Info

yarn info langchain
platform lambda
Node version 18

[dosubot[bot]]

Hey @hemantchaudhari2006! 👋 I'm here to help you out with any questions or issues you have. Let's work together to solve this problem!

To store image references alongside chat messages in DynamoDB using LangChain JS, you can utilize the additional_kwargs field in the DynamoDBChatMessageHistory class. This field is designed to store additional data in JSON format, which can include image URLs or references.

Here's how you can modify your code to include image URLs in the chat history:

1. When creating a StoredMessage, add the image URL to the additional_kwargs field. For example:

const message: StoredMessage = {
  type: "chat",
  data: {
    content: "Here is an image",
    role: "user",
    additional_kwargs: {
      imageUrl: `data:image/jpeg;base64,${base64String}`
    }
  }
};

2. Ensure that the createDynamoDBSerializedChatMessage method in DynamoDBChatMessageHistory correctly serializes this additional data. The method already handles additional_kwargs, so you should be able to store the image URL without further modifications to the method itself.

By following these steps, you can store image references in DynamoDB alongside your chat messages, allowing you to retrieve and display both text and images when loading chat history ^[1].

To continue talking to Dosu, mention @dosu.

[hemantchaudhari2006]

Thanks. But I have tried many ways to pass my image reference in additional_kwargs field, but it does not work. It always stores empty.

addMessage of DynamoDBChatMessageHistory is being called dynamically, so we cannot pass additional_kwargs there.