Agents with Chat Models Example Code Abnormal When Using google-serper Tool

[dsdashun]

System Info

• Langchain: 0.0.181
• OS: Ubuntu Linux 20.04
  • Kernel: Linux iZt4n78zs78m7gw0tztt8lZ 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Ubuntu version:

LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

• Python: Python 3.8.2

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

1. Use the example code provided in Quick Start: Agents with Chat Models, but replace the 'serpapi' tool with 'google-serper' tool .
   Here's the modified code:

from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI

chat = ChatOpenAI(temperature=0.3)

llm = OpenAI(temperature=0)
tools = load_tools(["google-serper", "llm-math"], llm=llm)

agent = initialize_agent(tools, chat, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True)

result = agent.run("Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?")
print(result)

When I execute the code above. Error occurred. Here's the error text:

(openai-test) dsdashun@iZt4n78zs78m7gw0tztt8lZ:~/workspaces/openai-test/langchain$ python3 get_started_chat_agent.py


> Entering new AgentExecutor chain...
Traceback (most recent call last):
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/chat/output_parser.py", line 22, in parse
    response = json.loads(action.strip())
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 4 column 2 (char 75)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "get_started_chat_agent.py", line 14, in <module>
    result = agent.run("Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?")
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 213, in run
    return self(args[0])[self.output_keys[0]]
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 116, in __call__
    raise e
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 113, in __call__
    outputs = self._call(inputs)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 792, in _call
    next_step_output = self._take_next_step(
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 672, in _take_next_step
    output = self.agent.plan(intermediate_steps, **inputs)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 385, in plan
    return self.output_parser.parse(full_output)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/chat/output_parser.py", line 26, in parse
    raise OutputParserException(f"Could not parse LLM output: {text}")
langchain.schema.OutputParserException: Could not parse LLM output: Question: Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?
Thought: I should use Serper Search to find out who Olivia Wilde's boyfriend is and then use Calculator to calculate his age raised to the 0.23 power.
Action:
```
{
  "action": "Serper Search",
  "action_input": "Olivia Wilde boyfriend"
},
{
  "action": "Calculator",
  "action_input": "Age of Olivia Wilde's boyfriend raised to the 0.23 power"
}
```

However, if I use the pdb debugger to debug the program step by step, and pause a little bit after running initialize_agent, everything is fine.

I didn't use the 'serpapi' tool, because I don't have an API key on it. So I cannot verify whether the original example code can be executed successfully on my machine using the 'serpapi' tool

Expected behavior

I expect the code can run successfully without any problems, even if I replace the search tool with a similar one.

[shajinzheng]

Likely will be fixed by #5371

[dsdashun]

Likely will be fixed by #5371

After upgrading to v0.0.189 , the problem still exists. But the error log has changed. Here's the new error message:

(openai-test) dsdashun@iZt4n78zs78m7gw0tztt8lZ:~/workspaces/openai-test/langchain$ python3 get_started_chat_agent.py


> Entering new AgentExecutor chain...
Traceback (most recent call last):
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/chat/output_parser.py", line 21, in parse
    response = parse_json_markdown(text)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/output_parsers/json.py", line 17, in parse_json_markdown
    parsed = json.loads(json_string)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "get_started_chat_agent.py", line 14, in <module>
    result = agent.run("Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?")
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 236, in run
    return self(args[0], callbacks=callbacks)[self.output_keys[0]]
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 140, in __call__
    raise e
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/chains/base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 951, in _call
    next_step_output = self._take_next_step(
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 773, in _take_next_step
    raise e
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 762, in _take_next_step
    output = self.agent.plan(
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/agent.py", line 444, in plan
    return self.output_parser.parse(full_output)
  File "/home/dsdashun/.local/share/virtualenvs/openai-test-pldc8tvg/lib/python3.8/site-packages/langchain/agents/chat/output_parser.py", line 25, in parse
    raise OutputParserException(f"Could not parse LLM output: {text}")
langchain.schema.OutputParserException: Could not parse LLM output: Question: Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?
Thought: I should use Google Serper to find the answer to the first question and then use Calculator to calculate the second part of the question.
Action:
```
{
  "action": "Google Serper",
  "action_input": "Olivia Wilde boyfriend"
}
```

[dosubot]

Hi, @dsdashun! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, you reported an issue related to an error that occurs when executing the example code for Agents with Chat Models using the 'google-serper' tool instead of 'serpapi'. It seems that shajinzheng suggested that the issue might be fixed by a specific pull request. However, you mentioned that even after upgrading to v0.0.189, the problem still exists, but the error log has changed.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!