ZapierToolkit Automatically Performing Unrequested Tasks

[juppytt]

System Info

The current behavior of the ZapierToolkit in the LangChain tool occasionally results in the execution of tasks that the user did not specifically request. For example, when a user asks the ZapierToolkit to read the most recent email, it sometimes replies to the email even when the user did not explicitly specify this action. This unexpected behavior occurs when the AgentExecutor chain automatically decides to reply to the email, regardless of whether the email requested the reply or not. It is safe for the ZapierToolkit to adhere to the user's explicit instructions to avoid potential mishaps or undesired outcomes.

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

The issue was tested with the GPT-3.5 models (openai chat completion model and text-davinci-003) with temperature=0.7, and Gmail-related Zapier NLA Development actions, such as "Gmail: Reply to Email," "Gmail: Find Email," and "Gmail: Send Email.", enabled.

Code Sample:

# get from https://platform.openai.com/
os.environ["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "")

# get from https://nla.zapier.com/demo/provider/debug (under User Information, after logging in): 
os.environ["ZAPIER_NLA_API_KEY"] = os.environ.get("ZAPIER_NLA_API_KEY", "")

llm = OpenAI()
zapier = ZapierNLAWrapper()
toolkit = ZapierToolkit.from_zapier_nla_wrapper(zapier)
agent = initialize_agent(toolkit.get_tools(), llm, 
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True);

try:
    agent.run("Read the most recent email")
except Exception as e:
    print(e)
    exit(1)

Expected behavior

The ZapierToolkit should only perform tasks that are explicitly requested by the user. If the user does not specify that the most recent email should be replied to, the ZapierToolkit should not automatically decide to perform this action.

[dosubot]

Hi, @juppytt! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you raised an issue regarding the ZapierToolkit in LangChain performing tasks that were not requested by the user. The suggestion is to change the behavior so that the ZapierToolkit only performs tasks that are explicitly specified by the user. However, there hasn't been any activity or comments on the issue yet.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project! Let us know if you have any further questions or concerns.