Portkey Integration with Autogen

[siddharthsambharia-portkey]

@qingyun-wu @marklysze

Why are these changes needed?

Created documentation for Integrating Portkey with Autogen. It provides a brief overview of Portkey's features and explains how it can be used to bring AutoGen agents into production.

Related issue number

Closes #3394

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[siddharthsambharia-portkey]

@microsoft-github-policy-service agree company="Portkey"

[thinkall]

Thanks for the PR, @siddharthsambharia-portkey . Could you also provide a notebook example so it

[siddharthsambharia-portkey]

Thanks for the PR, @siddharthsambharia-portkey . Could you also provide a notebook example so it
The PR already has a notebook showcasing a hands-on example of integrating Portkey with Autogen, check out our notebook

.

[gagb]

@jackgerrits -- can we please double check the language for ecosystem.

[marklysze]

@siddharthsambharia-portkey, thanks so much for the contribution!

I have a few questions if I can regarding non-OpenAI providers (such as Anthropic, Cohere, Groq, Mistral, etc.):

• How do you do provider-specification configuration parameters (e.g. top_k) - is it in the targets list/dictionary of the portkey config?
• Have you tested multiple providers with AutoGen? The reason I ask is, that in my experiences from creating specific client classes, some of these clients are particular on things like the role within messages, not wanting name to be on messages.
• Does tool/function calling work with different providers? (Update - I can see the features supported in the Portkeys docs, noted) Will it load balance/fallback to a provider that doesn't support tool calling when the request contains a tool?

I tried the sample code but I couldn't get it to balance/fallback. I wanted it to use Anthropic and Groq, both in the same group chat of the sample code. Is there anyway to test that? At the moment I'm trying to understand which AutoGen client class it is going to use when it load balances / fallbacks between different providers.

[siddharthsambharia-portkey]

@siddharthsambharia-portkey, thanks so much for the contribution!

I have a few questions if I can regarding non-OpenAI providers (such as Anthropic, Cohere, Groq, Mistral, etc.):

• How do you do provider-specification configuration parameters (e.g. top_k) - is it in the targets list/dictionary of the portkey config?
• Have you tested multiple providers with AutoGen? The reason I ask is, that in my experiences from creating specific client classes, some of these clients are particular on things like the role within messages, not wanting name to be on messages.
• Does tool/function calling work with different providers? (Update - I can see the features supported in the Portkeys docs, noted) Will it load balance/fallback to a provider that doesn't support tool calling when the request contains a tool?

I tried the sample code but I couldn't get it to balance/fallback. I wanted it to use Anthropic and Groq, both in the same group chat of the sample code. Is there anyway to test that? At the moment I'm trying to understand which AutoGen client class it is going to use when it load balances / fallbacks between different providers.

For provider-specific configuration parameters like top_k, you can add these fields as override_params in the config object. These are specified in the targets list/dictionary of the Portkey config. link to config Docs- https://docs.portkey.ai/docs/api-reference/config-object
Here's an example of how it would look:

JSON

{
  "strategy": {
    "mode": "fallback"
  },
  "targets": [
    {
      "provider": "groq",
      "api_key": "groq-api-key",
      "override_params": {
        "top_k": "0.4",
        "max_tokens": "100"
      }
    },
    {
      "provider": "anthropic",
      "api_key": "anthropic-api-key",
      "override_params": {
        "top_k": "0.6",
        "model": "claude-3-5-sonnet-20240620"
      }
    }
  ]
}

2. Regarding multiple providers with AutoGen, yes, it works well with non-OpenAI providers that support function calling.

3. Tool/function calling will work with providers that support it.
   Regarding load balancing/fallback to providers that don't support tool calling when the request contains a function call, the user needs to manage this carefully. Ideally you should group similar LLMs (in terms of function calling support) for fallback/load balancing to ensure compatibility.

[siddharthsambharia-portkey]

@gagb @marklysze I have made the resolved all the comments, is there anything else required to merge the PR?

Thanks!

[gagb]

Consider extending the write up with your answers to the questions that @marklysze asked-- others are likely to have similar questions. Otherwise looks good to go.

[siddharthsambharia-portkey]

Consider extending the write up with your answers to the questions that @marklysze asked-- others are likely to have similar questions. Otherwise looks good to go.

Yes, I will update the write-up and update Portkey docs for the same as well!
Thanks!

[marklysze]

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).

@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

[gagb]

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).

@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes that sounds like a good idea (at least for the current version of AutoGen).

[siddharthsambharia-portkey]

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).

@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case
.

[siddharthsambharia-portkey]

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).
@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case .

@marklysze @gagb Is there anything else you are waiting for to merge this PR?

[marklysze]

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).
@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case .

@marklysze @gagb Is there anything else you are waiting for to merge this PR?

My only concern without making any mention of a possible incompatibility due to the messaging structure is users raising issues when they get an incompatible role order. I think the wording can be simple, perhaps something like.
"Note: AutoGen messages will be passed to Portkey in line with OpenAI's API and this may be not be compatible with some LLMs that require messages to be in a certain role order."

[siddharthsambharia-portkey]

ible incompatibility due to the messaging structure is users raising issues when they get an incompatible rol

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).
@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case .

@marklysze @gagb Is there anything else you are waiting for to merge this PR?

My only concern without making any mention of a possible incompatibility due to the messaging structure is users raising issues when they get an incompatible role order. I think the wording can be simple, perhaps something like. "Note: AutoGen messages will be passed to Portkey in line with OpenAI's API and this may be not be compatible with some LLMs that require messages to be in a certain role order."

You're right; this could impact the user experience. However, Portkey serves as an AI Gateway, allowing users to easily switch between different models without altering the request process. Alternatively Autogen can recommend the best models for agents, and Portkey can facilitate using those models. This setup gives users flexibility to change models as new ones become available or for A/B testing, like switching from OpenAI to Llama 3.1

[gagb]

ible incompatibility due to the messaging structure is users raising issues when they get an incompatible rol

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).
@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case .

@marklysze @gagb Is there anything else you are waiting for to merge this PR?

My only concern without making any mention of a possible incompatibility due to the messaging structure is users raising issues when they get an incompatible role order. I think the wording can be simple, perhaps something like. "Note: AutoGen messages will be passed to Portkey in line with OpenAI's API and this may be not be compatible with some LLMs that require messages to be in a certain role order."

You're right; this could impact the user experience. However, Portkey serves as an AI Gateway, allowing users to easily switch between different models without altering the request process. Alternatively Autogen can recommend the best models for agents, and Portkey can facilitate using those models. This setup gives users flexibility to change models as new ones become available or for A/B testing, like switching from OpenAI to Llama 3.1

@siddharthsambharia-portkey -- can you please fix the code formatting errors using pre-commit run all-files and also added @marklysze 's suggestion and lets get this merged :)

[siddharthsambharia-portkey]

ible incompatibility due to the messaging structure is users raising issues when they get an incompatible rol

Do you know if there's a way to handle this?

Hey @marklysze this is Anthropic specific problem, they throw an error if there is an assistant message after the system message. Portkey is an AI Gateway, It can't change anything in the request. There is no way to handle this using Portkey

Yes, appreciate the challenge here (as we've had to create specific client/provider classes to cater for these cases).
@siddharthsambharia-portkey and @gagb, do we need to make a note of the preference for OpenAI API compatible providers to ensure better compatibility?

Yes, we could do that ideally but the models keep changing every few months, you can never be certain. I think the user can decide what model works best for his use case .

@marklysze @gagb Is there anything else you are waiting for to merge this PR?

My only concern without making any mention of a possible incompatibility due to the messaging structure is users raising issues when they get an incompatible role order. I think the wording can be simple, perhaps something like. "Note: AutoGen messages will be passed to Portkey in line with OpenAI's API and this may be not be compatible with some LLMs that require messages to be in a certain role order."

You're right; this could impact the user experience. However, Portkey serves as an AI Gateway, allowing users to easily switch between different models without altering the request process. Alternatively Autogen can recommend the best models for agents, and Portkey can facilitate using those models. This setup gives users flexibility to change models as new ones become available or for A/B testing, like switching from OpenAI to Llama 3.1

@siddharthsambharia-portkey -- can you please fix the code formatting errors using pre-commit run all-files and also added @marklysze 's suggestion and lets get this merged :)

sure that's awesome. I will do that right away

[siddharthsambharia-portkey]

cc @gagb @marklysze

[gagb]

cc @gagb @marklysze

@siddharthsambharia-portkey, marks suggest text is still not there? Can we please add that :)

[siddharthsambharia-portkey]

cc @gagb @marklysze

@siddharthsambharia-portkey, marks suggest text is still not there? Can we please add that :)

Yes my bad, added it. @gagb

[marklysze]

Looks good @siddharthsambharia-portkey, thanks!

[marklysze]

@siddharthsambharia-portkey, can you please:

• remove the trailing white-space on line 15 after 2.
• add a blank line at the end of the file

[siddharthsambharia-portkey]

done @marklysze, there was alredy a blank line at the end of the file although

[siddharthsambharia-portkey]

Hey @gagb @marklysze LGTM?