LangChain HuggingFace Integration tool_choice parameter is now out of date with TGI v3.0

[2016bgeyer]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

The following code

class add(BaseModel):
    """Add two integers."""

    a: int = Field(..., description="First integer")
    b: int = Field(..., description="Second integer")


class multiply(BaseModel):
    """Multiply two integers."""

    a: int = Field(..., description="First integer")
    b: int = Field(..., description="Second integer")

tools = [add, multiply]


callbacks = None

model_endpoint = 'private_url'
model_name = 'private_name'
huggingfacehub_api_token = 'private'

hardcoded_args_dict = {
    "model_endpoint": model_endpoint,
    "max_new_tokens": "20000",
    "top_k": 10,
    "top_p": 0.95,
    "typical_p": 0.95,
    "temperature": 0.1,
    "repetition_penalty": 1.03,
    "timeout": 240,
    "callbacks": None,
    "huggingfacehub_api_token": huggingfacehub_api_token,
    "model_kwargs": {"model_id": model_name},
}

chat_model = ChatHuggingFace(llm=llm, max_tokens=500)

chat_with_tools = chat.bind_tools(tools, tool_choice="required") # You can use this if you want actual functions instead of Pydantic Tools

chat_with_tools_parsed = chat_with_tools | PydanticToolsParser(
        tools=tools, first_tool_only=True
    )  # shouldn't need this if it is correct


query = "Add 2 and 4"

try:
    response = chat_with_tools_parsed.invoke(query)
    print(response)
    assert response == add(a=2, b=4)
except:
    response = chat_with_tools.invoke(query)
    print(response)
    assert response.tool_calls is not None and response.tool_calls[0]["name"] == "add"

Error Message and Stack Trace (if applicable)

huggingface_hub.errors.HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: private_url

Tool error: Tool with name required not found

Description

ChatHuggingFace should have support for the new tool_choice = "required" parameter from HuggingFace TGI v3.0.

Docs:
https://huggingface.co/docs/text-generation-inference/basic_tutorials/using_guidance#tool-choice-configuration

This is especially important because HuggingFace TGI changed their API to match OpenAI's schema of having "required" as the default and "auto" be a secondary option.

This discrepancy in HuggingFace's API vs OpenAI's has caused a number of people confusion & issues and they've finally fixed it. It would be great to have this updated properly in LangChain as well.

I think the code could look something like this:

def bind_tools(
        self,
        tools: Sequence[Union[Dict[str, Any], Type, Callable, BaseTool]],
        *,
        tool_choice: Optional[
            Union[dict, str, Literal["auto", "required", "none"]]
        ] = None,
        **kwargs: Any,
    ) -> Runnable[LanguageModelInput, BaseMessage]:
    """
    ...
        tool_choice: Which tool to require the model to call.
                Must be the name of the single provided function,
                "auto" (default if tools are provided) to automatically determine 
                which function to call (if any), "required" to force the LLM to 
                determine which function to call (similar to strict), or a dict
                of the form:
                {"type": "function", "function": {"name": <<tool_name>>}}.

    ...
    """

    formatted_tools = [convert_to_openai_tool(tool) for tool in tools]
    if tools:
        if isinstance(tool_choice, str):
            if tool_choice not in ("auto", "required", "none"):
                if len(formatted_tools) != 1:
                    raise ValueError(
                        "When specifying `tool_choice`, you must provide exactly one "
                        f"tool. Received {len(formatted_tools)} tools."
                    )
                else:
                    tool_choice = {
                        "type": "function",
                        "function": {"name": tool_choice},
                    }
        elif isinstance(tool_choice, dict) and tool_choice != {}:
            tool_names = [ft["function"]["name"] for ft in formatted_tools]
            if tool_choice["function"]["name"] not in tool_names:
                raise ValueError(
                    f"Tool choice {tool_choice} was specified, but the only "
                    f"provided tools were {tool_names}."
                )
        else:
            raise ValueError(
                f"Unrecognized tool_choice type. Expected str or dict. "
                f"Received: {tool_choice}"
            )
        kwargs["tool_choice"] = tool_choice
    return super().bind(tools=formatted_tools, **kwargs)

In addition, there are a few other tweaks in the class that would be helpful:

• Adding tool call id and type ("tool_call_type") to tool_calls in _convert_message_to_chat_message
• Alternative roles for ToolMessages. Many model tokenizer configs can only support user/assistant/user/assistant workflows even if the models can call tools

Also, for HuggingFaceEndpoints, please allow us to match HuggingFace's API spec for API keys ('-'):

• huggingfacehub_api_token default shouldn't throw an error

System Info

System Information

OS: Linux
OS Version: #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024
Python Version: 3.11.4 (main, Jun 7 2023, 18:32:58) [GCC 10.2.1 20210110]

Package Information

langchain_core: 0.3.21
langchain: 0.3.8
langchain_community: 0.3.8
langsmith: 0.1.143
langchain_cli: 0.0.31
langchain_cohere: 0.3.2
langchain_experimental: 0.3.3
langchain_huggingface: 0.1.2
langchain_openai: 0.2.9
langchain_text_splitters: 0.3.2
langchainhub: 0.1.21
langgraph_sdk: 0.1.36
langserve: 0.3.0