ChatOpenAi: Streaming blocked w/ response_format "json_schema"

[ClaysGit]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI, ChatOpenAICallOptions } from "@langchain/openai";
import { z } from "zod";

const chatPromptTemplate = ChatPromptTemplate.fromMessages([
  new SystemMessage("You are a helpful art history expert."),
  new HumanMessage("What was Michaelangelo's most famous work?"),
]);

const baseModel = new ChatOpenAI({ apiKey: "your api key goes here", model: "gpt-4o-2024-08-06" });
const modelOptions: ChatOpenAICallOptions = {
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "response-body",
      schema: z.object({ name: z.string(), description: z.string() }),
    },
  },
};
const model = baseModel.bind(modelOptions);

const sequence = chatPromptTemplate.pipe(model);
const stream = await sequence.stream({});

for await (const message of stream) {
  console.log(message);
}

Error Message and Stack Trace (if applicable)

OpenAI does not yet support streaming with "response_format" set to "json_schema". Falling back to non-streaming mode.

From: https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-openai/src/chat_models.ts#L1213-L1237

Description

Ref: #6821.

My issue is an unexpected, and seemingly unnecessary, reduction in capability with a recent release.

I am using ChatOpenAI with the new option for response_format json_schema. I am attempting to enable this with a streamed output. The .stream() method was previously working with response_format json_schema (meaning the response was streamed in multiple chunks and adhered to the provided schema).

With the latest @langchain/openai I am receiving a warning: OpenAI does not yet support streaming with "response_format" set to "json_schema". Falling back to non-streaming mode.. Along with this warning, the streaming sequence is changed to a non-streaming sequence.

I can go into the @langchain/openai/dist/chat_models.js dependency and remove the check for json_schema mode, after which streaming is once again working. Therefore the block on this functionality is unnecessary.

System Info

"npm info @langchain/openai"

@langchain/openai@0.3.0 | MIT | deps: 4 | versions: 56
OpenAI integrations for LangChain.js
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain-openai/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/@langchain/openai/-/openai-0.3.0.tgz
.shasum: 89329ab9350187269a471dac2c2f4fca5f1fc5a3
.integrity: sha512-yXrz5Qn3t9nq3NQAH2l4zZOI4ev2CFdLC5kvmi5SdW4bggRuM40SXTUAY3VRld4I5eocYfk82VbrlA+6dvN5EA==
.unpackedSize: 360.5 kB

dependencies:
js-tiktoken: ^1.0.12        openai: ^4.57.3             zod-to-json-schema: ^3.22.3 zod: ^3.22.4

maintainers:
- eric_langchain <eric@langchain.dev>
- maddyadams <madelyn.adams1@gmail.com>
- sam_noyes <sam@langchain.dev>
- vbarda <vadym@langchain.dev>
- davidduong <david@duong.cz>
- andrewnguonly <andrewnguonly@gmail.com>
- basproul <braceasproul@gmail.com>
- jacoblee93 <jacoblee93@gmail.com>
- nfcampos <nuno@boringbits.io>
- sullivan-sean <ssullivan61198@gmail.com>
- hwchase17 <hw.chase.17@gmail.com>

dist-tags:
latest: 0.3.0     next: 0.3.0-rc.0

published 3 days ago by jacoblee93 <jacoblee93@gmail.com>

[dosubot]

I found a similar discussion regarding the issue you're facing: ChatOpenAi: Streaming blocked w/ response_format "json_schema" ^[1]. This discussion highlights that the latest version of LangChain.js does not support streaming with the response_format set to json_schema because OpenAI does not yet support streaming with this response format.

However, recent tests indicate that streaming with json_schema is now possible. Here is an example of a test that demonstrates streaming with a json_schema response format:

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
import { ChatOpenAI } from "../chat_models.js";
import { concat } from "@langchain/core/utils/stream";
import { AIMessageChunk } from "@langchain/core/messages";

const weatherSchema = z.object({
  city: z.string().describe("The city to get the weather for"),
  state: z.string().describe("The state to get the weather for"),
  zipCode: z.string().describe("The zip code to get the weather for"),
  unit: z.enum(["fahrenheit", "celsius"]).describe("The unit to get the weather in"),
});

const model = new ChatOpenAI({
  model: "gpt-4o-2024-08-06",
}).bind({
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "get_current_weather",
      description: "Get the current weather in a location",
      schema: zodToJsonSchema(weatherSchema),
      strict: true,
    },
  },
});

const stream = await model.stream("What is the weather in San Francisco, 91626 CA?");
let full: AIMessageChunk | undefined;
for await (const chunk of stream) {
  full = !full ? chunk : concat(full, chunk);
}
console.log(full);

To restore this functionality, ensure you are using the latest version of LangChain.js and verify that your implementation aligns with the updated tests and examples ^[2].

To continue talking to Dosu, mention @dosu.

[ClaysGit]

Thanks @dosu. A couple notes:

• The linked issue is a question I submitted while trying to understand this change. After some testing locally, I just think this decision is a mistake.
• OpenAI does already support streaming with this response format. It supported streaming before the offiical LangChain support was added too.
• The latest version of LangChain.js does not restore this functionality. The issue is present in the main branch.

[dosubot]

The latest version of LangChain.js blocks streaming with the response_format set to json_schema because, according to the code, OpenAI does not yet support streaming with this response format. The relevant code snippet includes a warning message indicating this limitation:

if (
  params.response_format &&
  params.response_format.type === "json_schema"
) {
  console.warn(
    `OpenAI does not yet support streaming with "response_format" set to "json_schema". Falling back to non-streaming mode.`
  );
  const res = await this._generate(messages, options, runManager);
  const chunk = new ChatGenerationChunk({
    message: new AIMessageChunk({
      ...res.generations[0].message,
    }),
    text: res.generations[0].text,
    generationInfo: res.generations[0].generationInfo,
  });
  yield chunk;
  return runManager?.handleLLMNewToken(
    res.generations[0].text ?? "",
    undefined,
    undefined,
    undefined,
    undefined,
    { chunk }
  );
}

This code explicitly checks if the response_format is set to json_schema and, if so, falls back to non-streaming mode because OpenAI does not support streaming with this format ^[1].

If you believe this decision is a mistake and OpenAI does support streaming with json_schema, you might consider modifying the local code to remove this check, as you have done, or raising this issue with the maintainers to discuss the potential for updating the library to reflect the current capabilities of OpenAI.

To continue talking to Dosu, mention @dosu.

[jacoblee93]

Hey, yes it looks like this is now supported. Will patch, thanks for flagging!