Integrate JSON Mode for Gemini 1.5 (flash)

[mainpyp]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Please integrate Googles JSON Mode into langchain_google_genai.ChatGoogleGenerativeAI similar to ChatOpenAI(..., model_kwargs={"response_format": {"type": "json_object"}})

Motivation

I want to use gemini for a generation task and the output must be in json format.
OpenAI cannot be used for my task because the token limit is too small.
Right now I need to switch to Googles API but I want to continue using langchain!

Proposal (If applicable)

I don't have a proposal on how to implement the feature but it would be great to use the keyword like this:
model = ChatGoogleGenerativeAI(..., model_kwargs={"generation_config": {"response_mime_type": "application/json"}})

[HMDRAMS-DEV]

Any updates on this?

[vishah02]

Hi! We are a group of 4 students from the university of Toronto, and we are looking into implementing this shortly!

[vishah02]

Here is our proposal! We will be implementing this soon. Any feedback is appreciated :)

Outline of Changes
Configuration Adjustments: Adding an option in LangChain’s ChatGoogleGenerativeAI class to accept generation_config={"response_mime_type": "application/json"}, making it compatible with JSON Mode for Gemini 1.5.
Testing & Validation: Verifying that JSON outputs are consistent and well-structured for various prompts, particularly focusing on how different formats affect response generation.

Existing Architecture
Currently, Langchain-google contains the class ChatGoogleGenerativeAI(_BaseGoogleGenerativeAI, BaseChatModel) in libs/genai/langchain_google_genai/chat_models.py which uses Google’s GenAI chat model.
ChatGoogleGenerativeAI is an existing class in the langchain-google library which allows for interfacing with Gemini.
The existing implementation allows using Google’s API but does not support specifying the response_mime_type, which would let the model return JSON-formatted data.

Files to Modify
Chat_models.py (/libs/genai/langchain_google_genai/chat_models.py):
To update generation_config so that it is compatible with JSON mode. We will update and create new functions in the class ChatGoogleGenerativeAI.
Test_chat_models.py (libs/genai/tests/unit_tests/test_chat_models.py):
To add relevant test cases that will support the changes we have made.
langchain-google/libs/genai/langchain_google_genai/llms.py:
To add param validation to the BaseGoogleGenerativeAI class
New Files

This issue will likely not require addition of any new files, unless we decide to create a separate test file.

Pseudocode
Our idea for the pseudocode was to follow a similar structure to the code in libs\vertexai\langchain_google_vertexai\chat_models.py which already supports the JSON schema.

Outline of changes made to chat_models.py:

class ChatGoogleGenerativeAI(_BaseGoogleGenerativeAI, BaseChatModel):
response_mime_type: Optional[str] = None
if self.response_mime_type is not None:
   updated_params["response_mime_type"] = self.response_mime_type


if self.response_schema is not None:
           allowed_mime_types = ("application/json")
       if self.response_mime_type not in allowed_mime_types:
           error_message = (
               "`response_schema` is only supported when "
               f"`response_mime_type` is set to one of {allowed_mime_types})
               raise ValueError(error_message)




def _update(self, stop: Optional[List[str]] = None
    ) -> GenerationConfig:
    return GenerationConfig(
        **self._prepare_params(
            stop=stop,
            stream=stream,
        )
    )


def with_structured_output(
       if method == "json_mode":
           if isinstance(schema, type):
               schema_json = schema.model_json_schema()
               schema_json = replace_defs_in_schema(schema_json)
               self.response_schema = schema_json
               parser = PydanticOutputParser(pydantic_object=schema)
		  //also might need to add nested support
           else:
               parser = JsonOutputParser()
               self.response_schema = schema
           self.response_mime_type = "application/json"
           llm: Runnable = self
)

[vishah02]

PR for this is up!
langchain-ai/langchain-google#625