Function Calling with GPTAssistantAgent

[jtrugman]

Why are these changes needed?

There currently are not notebooks for using tools with the GPTAssistantAgent, this PR adds a light hearted example of a GPTAssistantAgent based Multiagent system that utilizes tools such as calling an API and writing to a file.

This notebook is designed to fetch Dad jokes related to specific themes and transform them into Sad jokes. The system employs two specialized agents, namely "The Dad" and "The Sad Joker".

Key Features:

• Dad Agent: Utilizes the get_dad_jokes function to fetch jokes from the Dad Joke API based on user-provided themes. This agent processes and sends these jokes to the Sad Joker for transformation.
• Sad Joker Agent: Receives Dad jokes and creatively transforms them into Sad jokes. It then writes these transformed jokes into a text file using the write_to_txt function.
• Function Integration: Both agents are enhanced with specific functions (get_dad_jokes and write_to_txt) which they can call directly to perform their tasks effectively.
• User Interaction: Through the group chat setup, the user can initiate conversations and set themes for the jokes

Technical Implementation:

• Function Setup: Defined necessary Python functions for fetching jokes and writing to text files.
• Agent Configuration: Configured each agent with specific roles and linked them with their respective functions to enable functional calling within the multi-agent system.
• Group Chat Mechanism: Integrated a group chat that includes both agents and a user proxy to manage the conversation flow and ensure coherent interaction among participants.

Hopefully this notebook helps users autogen more easily adopt the GPTAssistantAgent within their applications.

This change includes

Related issue number

N/A

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[ekzhu]

Also add a link in the example page:

https://microsoft.github.io/autogen/docs/Examples#multi-agent-chat-with-openai-assistants-in-the-loop

In website/docs/Examples.md

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 37.02%. Comparing base (4a44093) to head (8124f9f).
Report is 18 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2375      +/-   ##
==========================================
- Coverage   38.14%   37.02%   -1.12%     
==========================================
  Files          78       81       +3     
  Lines        7865     8084     +219     
  Branches     1683     1724      +41     
==========================================
- Hits         3000     2993       -7     
- Misses       4615     4840     +225     
- Partials      250      251       +1     

Flag	Coverage Δ
unittests	37.02% <ø> (-1.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonichi]

@ekzhu the PR is blocked by your change request.

[IANTHEREAL]

@jtrugman please take your time to address @ekzhu 's comment

[sonichi]

@ekzhu is your change request a must for this PR?
@jtrugman do you need help with addressing the change request?

[jtrugman]

Just got around to updating the PR, appreciate your patience on this.

I am running into an issue though with the new method @ekzhu suggested

The functions are registering with the agents' in the llm_config the_dad.llm_config["tools"]:

  'function': {'description': 'Fetches a list of dad jokes based on a search term. Allows pagination with page and limit parameters.',
   'name': 'get_dad_jokes',
   'parameters': {'type': 'object',
    'properties': {'search_term': {'type': 'string',
      'description': 'search_term'},
     'page': {'type': 'integer', 'default': 1, 'description': 'page'},
     'limit': {'type': 'integer', 'default': 10, 'description': 'limit'}},
    'required': ['search_term']}}}]

However the Agent's don't seem to be able to recognize that they have access to the function:

the_dad (to chat_manager):

It seems there was an error with the system trying to use a function that doesn't exist. No worries, I can still provide you with some cat-related dad jokes. Let's get started:

1. Why don't cats play poker in the jungle? Too many cheetahs.

2. What do you call a pile of cats? A meowtain.

3. Did you hear about the cat who swallowed a ball of wool? She had mittens.

4. What’s a cat’s favorite magazine? Good Mousekeeping.

5. Why was the cat sitting on the computer? To keep an eye on the mouse.

6. How do cats end a fight? They hiss and make up.

7. What do you call a cat who lives in an igloo? An eskimew.

8. Why did the cat run from the tree? Because it was afraid of the bark!
...

When I checked the OpenAI Platform, it's showing that the functions are not registered with the Assistant API, even though when I call the_dad.llm_config["tools"] I get back the proper tool (screenshot of OpenAI Platform below). So it looks like there is a disconnect between the autogen llm_config and what OpenAI has for the Assistant config which may be why the Assistant is not able to call the tool.

Have you guys seen this issue before with the new way to configure tools and GPTAssistantAgents?

[ekzhu]

I am running into an issue though with the new method @ekzhu suggested

It looks like the open ai assistant currently requires the tools to be registered upon agent creation. See: https://platform.openai.com/docs/assistants/tools/function-calling

So, instead of using register_function, you can use get_function_schema:

autogen/autogen/function_utils.py

Line 211 in 5b6ae32

def get_function_schema(f: Callable[..., Any], *, name: Optional[str] = None, description: str) -> Dict[str, Any]:

to generate the JSON schema for the function, and then add it through the tools fields of the assistant_config when creating the assistant.

[jtrugman]

Thanks, the get_function_schema is a cleaner method to add the JSON to the tools fields in the llm_config than I had originally. Just implemented.

One thing to note though is that I still had to register the function with the GPTAssistantAgent for the Agents to use the functions correctly. In this example, the_dad agent had the function registered (able to successfully call the function) and the_sad_joker did not have the function registered (not able to successfully call the function).

Another thing to note as well:

It looks like the open ai assistant currently requires the tools to be registered upon agent creation. See: https://platform.openai.com/docs/assistants/tools/function-calling

You can update the tools after the agent is created. I implemented this logic into autogen with the overwrite_tools config in GPTAssistantAgent. It can be done via

update_gpt_assistant(
      self._openai_client,
      assistant_id=openai_assistant_id,
      assistant_config={
          "tools": specified_tools,
          "tool_resources": openai_assistant_cfg.get("tool_resources", None),
      }

I think the notebook is ready to go, but let me know if you want me to make anymore changes. Using the get_function_schema is definitely helpful though, more efficient than how I was doing tool calling before. Thanks

[ekzhu]

Thanks @jtrugman this is looking great now!

On this note:

You can update the tools after the agent is created. I implemented this logic into autogen with the overwrite_tools config in GPTAssistantAgent. It can be done via

This is clearly a missing feature we need to support to make GPTAssistantAgent and ConversableAgent aligned. Can you create an issue for this? I think the implementation is prettly clear which is to use the overwrite_tools code path in register_for_llm implementation of GPTAssistantAgent.

[jtrugman]

Totally agree! Here is the GitHub Issue for this (#2575)