ChatGoogleGenerativeAI: not call callbacks (_runManager?.handleLLMNewToken)

[Nanyi-Jiang]

Context

When implementing Gemini with ChatGoogleGenerativeAI, I tried to use the call and stream functions with a callback function handleLLMToken which will write the response to my client side

• Expected: Can see the response from Gemini sent to the client
• Actual: Nothing was sent, and the handleLLMToken I passed in was not executed

Investigation

I think it didn't call the callbacks (_runManager?.handleLLMNewToken) in the functions: _generate and _streamResponseChunks which caused the actual behavior to diverge from expected.

Implementation of ChatGoogleGenerativeAI _streamResponseChunks:

langchainjs/libs/langchain-google-genai/src/chat_models.ts

Lines 298 to 306 in 1794285

	for await (const response of stream) {
	const chunk = convertResponseContentToChatGenerationChunk(response);
	if (!chunk) {
	continue;
	}
	yield chunk;
	}
	}

The implementation of other models:

langchainjs/langchain/src/chat_models/anthropic.ts

Lines 302 to 310 in 1794285

	const delta = data.completion ?? "";
	yield new ChatGenerationChunk({
	message: new AIMessageChunk({
	content: delta,
	additional_kwargs,
	}),
	text: delta,
	});
	await runManager?.handleLLMNewToken(delta);

[dosubot]

🤖

Hey there, @Nanyi-Jiang! Fancy seeing you here again. Hope your code is behaving better than a caffeinated squirrel today! 🐿️

Based on the information provided, it appears that the _runManager parameter, which is responsible for managing callbacks, is not utilized in the _generate and _streamResponseChunks methods of the Langchain's ChatGoogleGenerativeAI model. This is why the callbacks that are passed in are not being called. Here is the relevant code:

async _generate(
  messages: BaseMessage[],
  options: this["ParsedCallOptions"],
  _runManager?: CallbackManagerForLLMRun
): Promise<ChatResult> {
  // ... code that generates the response but does not use _runManager ...
}

async *_streamResponseChunks(
  messages: BaseMessage[],
  options: this["ParsedCallOptions"],
  _runManager?: CallbackManagerForLLMRun
): AsyncGenerator<ChatGenerationChunk> {
  // ... code that generates the response but does not use _runManager ...
}

You can find this in the libs/langchain-google-genai/src/chat_models.ts file.

This behavior is unlike other models where the _runManager parameter is used to manage and call the callbacks.

There have been similar issues in the past, such as ChatOpenAI LLMChain handleLLMNewToken constructor chain doesnt trigger and Streaming stopped working since the latest version (0.0.30). The solutions to these issues involved ensuring that the callbacks are attached to the correct object and not using the overall verbose flag of the model for the handleLLMNewToken method.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• ChatOpenAI LLMChain handleLLMNewToken constructor chain doesnt trigger
• Streaming stopped working since the latest version (0.0.30)
• libs/langchain-google-genai/src/chat_models.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jacoblee93]

CC @bracesproul

[jacoblee93]

Thank you for flagging!

[bracesproul]

hey @Nanyi-Jiang this will be patched in the next package release!