How to filter metadata with OpenSearch similaritySearch

[npthao1312]

Checked other resources

• I added a very descriptive title to this question.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the LangChain documentation with the integrated search.

Commit to Help

• I commit to help with one of those options 👆

Example Code

await vectorStore.similaritySearch(userPrompt, 1, {
    "term": {
              "metadata.source.keyword":
                "wikipedia.txt",
            },

  });

Description

For example, I want to filter/pre-filter documents with similaritySearch based on filename. This is the way of filter in OpenSearch "term": {
"metadata.source.keyword":
"wikipedia.txt",
}. But it seems like Langchain ignore this or I'm doing it wrong. Please help me.

System Info

Node.js v18.16.0
yarn version v1.22.19

[dosubot[bot]]

To filter documents by filename in your similarity search with LangChain and OpenSearch, you need to use the filter parameter correctly in your query. The key in the filter object should match the metadata field name where filenames are stored in your documents. Here's how you can construct and use the filter with an example filename:

// Define the filename you want to filter by
const filenameFilter = "example_filename.txt";

// Construct the filter object
const filter = {
  filename: filenameFilter // Ensure this key matches the actual metadata field name for filenames
};

// Use the filter in the similaritySearchVectorWithScore method
const queryVector = [/* your query vector here */];
const k = 10; // Number of similar documents to return
const filteredResults = await vectorStore.similaritySearchVectorWithScore(queryVector, k, filter);

Make sure filenameFilter is replaced with the filename you're targeting, and the key in the filter object (filename in this example) accurately reflects the metadata field name for filenames in your documents. This approach ensures the similarity search is performed only on documents with the specified filename.

It's important to note that the filtering capabilities have specific requirements regarding the metadata structure and types, and there are limitations on filtering based on array elements or complex data types. The filtering mechanism uses JSON serialization for comparison, so ensure your filter values are compatible. Additionally, consider the performance implications of filtering on deeply nested or large JSON objects.

If you encounter any issues or have further questions, feel free to ask for more assistance.

Details

Sources

• libs/langchain-community/src/vectorstores/opensearch.ts
• libs/langchain-community/src/vectorstores/singlestore.ts
• libs/langchain-community/src/vectorstores/redis.ts
• libs/langchain-community/src/vectorstores/hanavector.ts
• libs/langchain-community/src/vectorstores/tests/vectara.int.test.ts
• examples/src/indexes/vector_stores/hnswlib_filter.ts
• libs/langchain-community/src/vectorstores/azure_aisearch.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[npthao1312]

@dosu yes, I understand there is a similaritySearchVectorWithScore, but I was looking for the similaritySearch only

[dosubot[bot]]

To filter documents by filename in your similaritySearch with LangChain and OpenSearch, you should ensure that the filename is stored in the document's metadata and use a filter object that targets this specific filename field. Here's a concise way to apply a filter based on the filename:

1. Confirm that the filename is indexed in the document's metadata.
2. Construct a filter object that specifies the filename you want to filter by. For example:

const filenameFilter = {
  "term": {
    "metadata.filename.keyword": "example_filename.txt"
  }
};

3. Pass this filter object to the similaritySearch method. Ensure that the filter object's structure matches the expected format for the search method you are using.

Here's an example call to similaritySearch with a filename filter:

const results = await vectorStore.similaritySearch(userPrompt, 1, filenameFilter);

Replace "example_filename.txt" with the actual filename you're targeting. This approach ensures that the similarity search is performed only on documents with the specified filename, leveraging the metadata stored in your OpenSearch documents.

Details

Sources

• Inconsistent Behaviour with PrismaVectoreStore Filters.
• libs/langchain-community/src/vectorstores/opensearch.ts
• libs/langchain-community/src/vectorstores/redis.ts
• libs/langchain-community/src/vectorstores/vectara.ts
• examples/src/indexes/vector_stores/xata_metadata.ts
• libs/langchain-community/src/vectorstores/tests/vectara.int.test.ts
• libs/langchain-community/src/vectorstores/tests/prisma.test.ts
• examples/src/indexes/vector_stores/hnswlib_filter.ts
• libs/langchain-redis/src/tests/vectorstores.int.test.ts
• langchain/src/retrievers/self_query/vectara.ts
• libs/langchain-community/src/vectorstores/prisma.ts
• libs/langchain-community/src/vectorstores/tests/hnswlib.test.ts
• libs/langchain-community/src/vectorstores/tests/rockset.int.test.ts
• libs/langchain-community/src/vectorstores/tests/turbopuffer.int.test.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[stevedaskam]

@dosu This does not work for me. If I add documents and then filter with the code below, I get no documents back. What should the filter format be?

const docs = [
      new Document({
        metadata: { user: "U12345-1234" },
        pageContent: "Amazon Simple Storage Service (Amazon S3) is a cloud storage service that allows users to store and retrieve data of any amount from anywhere.",
      }),
      new Document({
        metadata: { user: "U12345-5678" },
        pageContent: "Amazon Elastic Compute Cloud (Amazon EC2) provides on-demand, scalable computing capacity in the Amazon Web Services (AWS) Cloud.",
      }),
    ];

const vectorStore = await OpenSearchVectorStore.fromDocuments(docs, embeddings, {
        client: client,
        indexName: 'my-index',
        service: 'aoss'
      });

const query = "What is S3?";
const k = 5; // Number of results to retrieve
const filter = {
  user: "U12345-1234"
};

const results = await vectorStore.similaritySearch(query, k, filter);

[stevedaskam]

@npthao1312 Did you get the filtering to work?

[KwangGan]

Any update on this ?