How to use Output parser with ConversationalRetrievalQAChain?

[priyank-joktatkal]

Hello,

How can we use output parser with ConversationalRetrievalQAChain? I have attached my code bellow.

const chain = ConversationalRetrievalQAChain.fromLLM( model, vectorstore.asRetriever(15), { qaTemplate: QA_PROMPT, questionGeneratorTemplate: CONDENSE_PROMPT, verbose: true, } ); const chainCall = await chain.call({ question: sanitizedQuestion, chat_history: history || [], });

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're looking to use an output parser with the ConversationalRetrievalQAChain in LangChain. The output parser is a function that transforms the output of the chain into your desired format.

Here's how you can integrate an output parser into your code:

First, import the OutputParser from its location. Then, create an instance of the OutputParser.

import { OutputParser } from "../../parsers/output_parser.js"; // Import the OutputParser

const outputParser = new OutputParser(); // Create an instance of the OutputParser

Next, when creating the chain, pass the parse method of the OutputParser as an option.

const chain = ConversationalRetrievalQAChain.fromLLM(
    streamingModel,
    vectorStore.asRetriever(),
    {
      returnSourceDocuments: true,
      memory: new BufferMemory({
        memoryKey: "chat_history",
        inputKey: "question", // The key for the input to the chain
        outputKey: "text", // The key for the final conversational output of the chain
        returnMessages: true, // If using with a chat model
      }),
      questionGeneratorChainOptions: {
        llm: nonStreamingModel,
      },
      outputParser: outputParser.parse, // Pass the parse method of the OutputParser
    }
  );

Finally, call the chain and use the outputParser to parse the response.

const question = "What did the president say about Justice Breyer?";
const res = await chain.call({ question });
console.log({ streamedResponse: outputParser.parse(res) });

Please note that the actual implementation of the OutputParser class and its parse method is not provided in this context. You'll need to implement it according to your specific needs. The parse method should take the output of the chain and transform it into the desired format.

Also, it's worth mentioning that you can pass an alternative prompt for the question generation chain that also returns parts of the chat history relevant to the answer. This allows the QA chain to answer meta questions with the additional context. However, be aware that adding more context to the prompt in this way may distract the language model from other relevant retrieved information.

This is one potential way to use an output parser with the ConversationalRetrievalQAChain. If you find this solution helpful and believe it could benefit others, I encourage you to make a pull request to update the LangChain documentation. This way, other users facing the same issue can easily find this solution.

Sources

• langchain/src/chains/tests/conversational_retrieval_chain.int.test.ts
• examples/src/chains/conversation_qa_custom_prompt.ts
• examples/src/chains/conversational_qa_built_in_memory.ts
• docs/docs/modules/chains/index_related_chains/conversational_retrieval.mdx
• langchain/src/chains/conversational_retrieval_chain.ts
• examples/src/chains/conversational_qa_streaming.ts

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[haoyitedaniu]

This does not seem to work for zod used https://blog.langchain.dev/going-beyond-chatbots-how-to-make-gpt-4-output-structured-data-using-langchain/ Can anyone provide an example using the zod scheme and YET using ConversationalRetrievalQAChain

[sapana-k]

hello, did you find any solution to this issue? I'm stuck with the same issue..

[dosubot[bot]]

To use the Zod schema with ConversationalRetrievalQAChain in LangChain, you can follow this example:

1. Import necessary modules:

   import { ChatAnthropic } from "@langchain/anthropic";
   import {
     ChatPromptTemplate,
     MessagesPlaceholder,
   } from "@langchain/core/prompts";
   import { BaseMessage } from "@langchain/core/messages";
   import { createStuffDocumentsChain } from "langchain/chains/combine_documents";
   import { createHistoryAwareRetriever } from "langchain/chains/history_aware_retriever";
   import { createRetrievalChain } from "langchain/chains/retrieval";
   import { z } from "zod";

2. Initialize the retriever and language model:

   const retriever = ...your retriever;
   const llm = new ChatAnthropic();

3. Create the contextualize question prompt:

   const contextualizeQSystemPrompt = `
   Given a chat history and the latest user question
   which might reference context in the chat history,
   formulate a standalone question which can be understood
   without the chat history. Do NOT answer the question, just
   reformulate it if needed and otherwise return it as is.`;
   const contextualizeQPrompt = ChatPromptTemplate.fromMessages([
     ["system", contextualizeQSystemPrompt],
     new MessagesPlaceholder("chat_history"),
     ["human", "{input}"],
   ]);

4. Create the history-aware retriever:

   const historyAwareRetriever = await createHistoryAwareRetriever({
     llm,
     retriever,
     rephrasePrompt: contextualizeQPrompt,
   });

5. Create the question-answering prompt:

   const qaSystemPrompt = `
   You are an assistant for question-answering tasks. Use
   the following pieces of retrieved context to answer the
   question. If you don't know the answer, just say that you
   don't know. Use three sentences maximum and keep the answer
   concise.
   \n\n
   {context}`;
   const qaPrompt = ChatPromptTemplate.fromMessages([
     ["system", qaSystemPrompt],
     new MessagesPlaceholder("chat_history"),
     ["human", "{input}"],
   ]);

6. Create the question-answer chain:

   const questionAnswerChain = await createStuffDocumentsChain({
     llm,
     prompt: qaPrompt,
   });

7. Create the retrieval chain:

   const ragChain = await createRetrievalChain({
     retriever: historyAwareRetriever,
     combineDocsChain: questionAnswerChain,
   });

8. Define the Zod schema for structured output parsing:

   const structuredOutputParserSchema = z.object({
     destination: z
       .string()
       .optional()
       .describe('name of the question answering system to use or "DEFAULT"'),
     next_inputs: z
       .object({
         query: z
           .string()
           .describe("a potentially modified version of the original input"),
       })
       .describe("input to be fed to the next model"),
   });

9. Invoke the chain with chat history and input:

   const chat_history: BaseMessage[] = [];
   const response = await ragChain.invoke({
     chat_history,
     input: "...",
   });

This example demonstrates how to set up and use the ConversationalRetrievalQAChain with the necessary components in LangChain, including the integration of a Zod schema for structured output parsing ^[1][2].

To continue talking to Dosu, mention @dosu.