Added support for streaming tool calls

[davorrunje]

This is closing an issue coming from a discussion in PR #1118:

choice.delta.function_call is deprecated and we should use choice.delta.tool_calls instead (https://platform.openai.com/docs/api-reference/chat/streaming). Since tests are passing, we are probably not triggering tool_calls but a deprecated function_call

Other than that, some type hints were fixed so that two additional files pass mypy type checks. Mypy configuration is not part of this PR and no type-checking is added here.

Why are these changes needed?

Related issue number

Closes #1178

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

Attention: 68 lines in your changes are missing coverage. Please review.

Comparison is base (b548e55) 30.38% compared to head (1b99532) 40.47%.

Files	Patch %	Lines
autogen/oai/client.py	35.23%	67 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1184       +/-   ##
===========================================
+ Coverage   30.38%   40.47%   +10.09%     
===========================================
  Files          32       32               
  Lines        4302     4353       +51     
  Branches      994     1065       +71     
===========================================
+ Hits         1307     1762      +455     
+ Misses       2901     2464      -437     
- Partials       94      127       +33     

Flag	Coverage Δ
unittests	40.40% <36.44%> (+10.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tyler-suard-parker]

@davorrunje It is streaming now, thank you!

[tyler-suard-parker]

Ok this is streaming output to the terminal, which is great. However, I have AutoGen behind an API, and I want to stream the output back to the requestor. How would I do that using your repo?

[davorrunje]

Ok this is streaming output to the terminal, which is great. However, I have AutoGen behind an API, and I want to stream the output back to the requestor. How would I do that using your repo?

We could easily add a callback to be called, would that work for you?

[bitnom]

Ok this is streaming output to the terminal, which is great. However, I have AutoGen behind an API, and I want to stream the output back to the requestor. How would I do that using your repo?

We could easily add a callback to be called, would that work for you?

I've been doing a lot of work in this area but need to merge it with the latest main and this PR. Some important things to consider about callbacks:

• When a callback is called, it should be called with all possible data relating to the request/response, preferably as all keyword arguments.
• The callback function should be called for every chunk and every completed message regardless of whether stream is enabled.
• Callbacks can be both sync and async.
• User can specify a request_id: UUID when sending messages via user proxy, which will then be passed to the callback function for subsequent responses of agents.
• Each agent should have a unique agent_id: UUID which is passed to the callback function.

At least that's how I spec it. My implementation is pretty rough at this point. I've got it working except that my request_id doesn't change when I supply a new one yet for some reason. I'm working on it.

[davorrunje]

@davorrunje @bitnom As it is right now, I instantiate a User Proxy Agent and an Assistant Agent, then do self.user_proxy.initiate_chat. The conversation goes well, and the final answer streams to the terminal, but I need to send the stream back to a requestor. You mentioned a callback, that could be an option, or maybe make a new method, initiate_streaming_chat, which returns a generator immediately, and that generator can be used to get chunks as the conversation continues.

@tyler-suard-parker Can you please open an issue for that and we'll take it from there. I like the proposal, but it is outside of the scope of this PR. This is a cleanup on an already merged PR.

[bitnom]

@davorrunje @bitnom As it is right now, I instantiate a User Proxy Agent and an Assistant Agent, then do self.user_proxy.initiate_chat. The conversation goes well, and the final answer streams to the terminal, but I need to send the stream back to a requestor. You mentioned a callback, that could be an option, or maybe make a new method, initiate_streaming_chat, which returns a generator immediately, and that generator can be used to get chunks as the conversation continues.

You'll be able to use a generator if you like. An arbitrary dict can be specified alongside the callback, which is always passed back to it along with the openai responses. The issue I was having with it is resolved. Now to payback a bit of technical debt.