Mistral AI Rate limit

[AndiLeni]

Hello,

I was following the docs and tried to index some documents with LanceDB and MistralAI.
However, I run into Request size limit exceeded errors when I create the embeddings.
The text source file is rather large, it has approximately 35000 lines.

The code I tried is this one:

import { ChatMistralAI, MistralAIEmbeddings } from "@langchain/mistralai";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { LanceDB } from "@langchain/community/vectorstores/lancedb";
import fs from "node:fs/promises";
import path from "node:path";
import { connect } from "vectordb";

const loader = new TextLoader("./text/c.md");

var docs = await loader.load();

const splitter = RecursiveCharacterTextSplitter.fromLanguage("markdown", {
    chunkSize: 1000,
    chunkOverlap: 200,
});
const documents = await splitter.splitDocuments(docs);

const embeddings = new MistralAIEmbeddings({
    model: "mistral-embed",
    apiKey: "xxx",
});

const dir = path.join(process.cwd(), "lancedb-permanent");
await fs.mkdir(dir, { recursive: true });

const db = await connect(dir);

const table = await db.createTable("vectors", [
    {
        vector: Array(1024),
        text: "sample",
        source: "sample",
        loc: {
            lines: {
                from: 1,
                to: 2,
            },
        },
    },
]);

var vectorStore = await LanceDB.fromDocuments(documents, embeddings, { table });

Has anyone an idea on how to fix this? Or can tell me what I was doing wrong.

Thanks and kind regards

[dosubot[bot]]

🤖

Hey @AndiLeni, great to see you diving into another challenge! Let's tackle this one together. 🚀

The Request size limit exceeded error you're encountering typically occurs when the size of the data you're trying to process exceeds the maximum limit set by the server. In this case, it seems like the text source file you're trying to create embeddings for is too large.

To resolve this issue, you can break down your large text source file into smaller chunks and process each chunk separately. Here's how you can do it:

import { MistralAIEmbeddings } from "@langchain/mistralai";

/* Embed queries */
const embeddings = new MistralAIEmbeddings({
  apiKey: process.env.MISTRAL_API_KEY,
});

// Assuming `largeTextSource` is your large text source file
const largeTextSource = ["Hello world", "Bye bye", ...]; // replace with your actual data

const chunkSize = 100; // adjust this value based on the server's limit
for (let i = 0; i < largeTextSource.length; i += chunkSize) {
  const chunk = largeTextSource.slice(i, i + chunkSize);
  const documentRes = await embeddings.embedDocuments(chunk);
  console.log({ documentRes });
}

In this code, we're splitting the largeTextSource array into smaller chunks of size chunkSize and processing each chunk separately. You might need to adjust the chunkSize value based on the server's limit. You can refer to the examples/src/models/embeddings/mistral.ts for more details.

The maximum request size limit for the MistralAIEmbeddings API is determined by the batchSize parameter. According to the provided code, the default value of batchSize is 512. This means that the maximum number of documents to embed in a single request is 512. You can find this information in the libs/langchain-mistralai/src/embeddings.ts file.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• examples/src/models/embeddings/mistral.ts
• libs/langchain-mistralai/src/embeddings.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[AndiLeni]

@dosu-bot The issue is that requests are made too fast because only 5 requests per second are allowd. How can I slow down langchains request?

[dosubot[bot]]

🤖

To slow down the requests to the MistralAI API, you can introduce a delay between each request. This can be achieved by using the setTimeout function in JavaScript. However, since setTimeout is not promise-based, we need to wrap it in a promise to use it with async/await syntax.

Here's how you can modify your code:

// Define a function that returns a promise that resolves after a delay
function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

// Use the delay function in the loop
for (const document of documents) {
  const embedding = await embeddings.embed(document);
  await vectorStore.add(embedding);
  await delay(200); // Delay of 200ms to comply with the rate limit of 5 requests per second
}

In this code, the delay function is used to pause the execution of the loop for 200 milliseconds after each request. This ensures that no more than 5 requests are made per second, as 1000ms/5 = 200ms.

Please note that this is a simple rate limiting solution and might not be suitable for all use cases. For more complex scenarios, you might want to consider using a more sophisticated rate limiting library that can handle things like backoff and retry strategies.

For more information on how to use promises with setTimeout, you can refer to this MDN Web Docs guide.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• examples/src/models/chat/chat_stream_mistralai.ts
• libs/langchain-mistralai/src/embeddings.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[AlbertoMQ]

This happens on the basic RAG example "https://docs.mistral.ai/guides/rag/". This should work with such a basic and small example no?

[ehenon]

Hi @AndiLeni 👋🏻

You can try setting the batchSize parameter in the MistralAIEmbeddings constructor to something lower than the default value. This defines the maximum number of documents to embed in a single request. You can also embed your documents in batches, leaving a sufficient pause between each batch process via a function of this sort if needed:

export const sleep = async (ms: number) => new Promise((resolve) => { setTimeout(resolve, ms); });