[BUG] Missing support for the parallel_tool_calls parameter in Azure openAI API

[gregwdata]

API Spec link

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-05-01-preview/generated.json

API Spec version

2024-05-01 preview

Describe the bug

support for the parallel_tool_calls parameter on the OpenAI API Spec: https://platform.openai.com/docs/api-reference/chat/create#chat-create-parallel_tool_calls is not implemented in the Azure API spec

Appears to be the cause of this issue filed on the OpenAI repo: openai/openai-python#1492

Expected behavior

Use parameter as described in: https://platform.openai.com/docs/api-reference/chat/create#chat-create-parallel_tool_calls

Actual behavior

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Unrecognized request argument supplied: parallel_tool_calls. Please contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 for further questions.', 'type': 'invalid_request_error', 'param': None, 'code': None}}

Reproduction Steps

from openai import AzureOpenAI # version 1.35.3
from azure.identity import DefaultAzureCredential, AzureAuthorityHosts, get_bearer_token_provider


token_provider = get_bearer_token_provider(...) # insert your token provider details here

client = AzureOpenAI(
    api_version="2024-05-01-preview", 
    azure_endpoint="...", # insert your endpoint here
    azure_ad_token_provider=token_provider,
)

response = client.chat.completions.create(
                model="...",   # insert your model here
                messages=[
                    {"role": 'user', "content": 'Please use a tool'}
                ],
                stream=False,
                tools=[{
                    'type': 'function', 
                    'function': {
                        'name': 'tool', 
                        'description': "A sample tool.", 
                        'parameters': {
                            'properties': {
                                'input_1': {
                                    'description': 'a list of integers for the tool', 
                                    'items': {'type': 'integer'}, 
                                    'title': 'Ranked text chunk indices', 'type': 'array'
                                    }
                                }, 
                            'required': ['input_1'], 
                            'title': 'Input for `tool`', 
                            'type': 'object'
                            }
                        }
                }],
                tool_choice='auto',
                parallel_tool_calls=False,
            )

print(response)

Environment

No response

[wooters]

Thank you for posting this @gregwdata! I've been banging my head against the wall for the past week trying to figure out what was wrong.

[corytomlinson]

Unfortunately another instance where Azure OpenAI is not able to keep up parity with OpenAI. This kind of discrepancy makes it really tough to develop on Azure. It’s bad enough having to wait weeks for the new models.

[axiangcoding]

For my case, using openai = "^1.26.0" and langchain-openai = "0.1.8" would not be a problem. If anyone are using langchain you can rollback the version first to temp fix the problem

[corytomlinson]

For my case, using openai = "^1.26.0" and langchain-openai = "0.1.8" would not be a problem. If anyone are using langchain you can rollback the version first to temp fix the problem

Issue has nothing to do with OpenAI or Langchain. It is the Azure OpenAI API that is behind. There is no temp fix for this problem. It requires Microsoft to support the parameter in their API.

[adrianzartis]

I had the same issue. Downgrading from openai==1.35.7 to openai==1.34.0 solved the problem.

[v-jiaodi]

@iscai-msft Please help take a look, thanks.

[kristapratico]

parallel_tool_calls is not supported by the Azure OpenAI service yet. When it is supported, it will be added to the spec under the API version which supports its usage. You can follow the what's new page on MS Learn for updates.

[wooters]

@kristapratico I believe there is still a bug here even though parallel_tool_calls is not supported yet.

Because:

• the OpenAI API sets parallel_tool_calls to True by default
• the Azure OpenAI service does not explicitly set parallel_tool_calls to False

So, parallel_tool_calls is ON when using the Azure OpenAI service, and there is no way to turn it off.

[kristapratico]

@wooters maybe I misunderstand, but the support coming would allow for the ability to turn parallel_tool_calls off. Both services have this on by default, and once supported by Azure, you'll be able to disable it, too.

[wooters]

@kristapratico Thanks for your response. I guess I misunderstood your comment:

parallel_tool_calls is not supported by the Azure OpenAI service yet.

By this, I guess what you mean is: "the ability to use the parallel_tool_calls arg to turn OFF parallel tool calls."

Thanks for clarifying. I look forward to this being added to Azure.

[HoangNguyen689]

@kristapratico
Can I ask when you plan to support this feature? Just following the changelog seems quite hopeless, and there isn’t any useful information at all.
At the very least, I would like a patch to make it compatible with the OpenAI API function signature, even if it’s not yet supported.

[janaka]

@kristapratico it would be really great if there was a feature release roadmap. Even if the time lines are by quarter.

We would like answers to the questions "when is feature A that OpenAI has already release going to be available on Azure OpenAI Inference API?" right now specifically parallel_tool_calls and tool_choice=required. Released on OpenAI in June and April respectively. When will they be available on Azure? How do we get the answer to that question?

[williamoverton]

This is blocking our ability to use Azure services. We need to be able to disable parallel_tool_calls.

[HoangNguyen689]

Waiting to support this feature.
I can't understand why the team from Azure is so late like this.

[pseudotensor]

Another reason not to use azure.

[qyj19920704]

openai.BadRequestError: Error code: 400 - {'error': {'message': "Unknown parameter: 'parallel_tool_calls'. (request id: 20240812154615201168063zjM398T3)", 'type': 'invalid_request_error', 'param': 'parallel_tool_calls', 'code': 'unknown_parameter'}}

[qyj19920704]

Excuse me, two months have passed without any reply or progress?

[axiangcoding]

No one working on this?

[trrwilson]

Hello! Providing a quick update on disabling parallel_tool_calls -- the work to support this parity feature across models/deployments is nearly complete and currently slated for the next service API release; barring unforeseen issues, it'll be part of a 2024-09-01-preview api-version label in a few weeks. We try to not share precise dates like this most of the time since so much can still be subject to change, but with how old this is it's owed a more concrete ETA.

As another engineer building on top of Azure OpenAI APIs (my team is responsible for client library integration), I empathize with the frustration and agree that it's taken way longer than anyone would like for this seemingly simple parity feature to become available. As for why it's taken so long, it boils down to the architectural differences between OpenAI's global endpoint and Azure OpenAI's deployment-based system: Azure OpenAI has component boundaries and abstractions that OpenAI doesn't, and while that confers some nice advantages in a lot of circumstances, it also turns out that consistently having properly configured traffic routed across many permutations of versioned model deployments, regions, and provisioning setups has revealed a lot of rough edges that spanned many teams. That's not an attempt to make an excuse -- we're using this and similar instances to continually improve these annoying parity latency situations -- but rather just to confirm that it's not been neglected or abandoned. It's just been a lot more complicated than anyone would've guessed.

Thank you for the continued patience!

[wooters]

Thanks so much for the update @trrwilson. Much appreciated!

[wooters]

@trrwilson related: do you know if the next service API release will include the ability to set tool_choice: "required"? (as described here and here)

[trrwilson]

@trrwilson related: do you know if the next service API release will include the ability to set tool_choice: "required"? (as described here and here)

That one's there now! required was recently made available on the 2024-07-01-preview service API label; here's the line where it's newly defined in the spec and here's the prior preview version where it was conspicuously absent.

We haven't yet published client library updates for all languages and protocol/REST-level calls may needed for those cases (.NET/Java/Go), but from an API perspective it's now supported.

[janaka]

@trrwilson really appreciate the update and especially some of the details you shared. Being as transparent as you can with this stuff goes a long way. I appreciate the difficulty of giving timelines but even just seeing things in a now/next/later roadmap would be great.

[janaka]

@trrwilson re feature like tool_choice=required what is the source of truth to know what's actually deployed? Given this particular feature was in the API spec of the latest stable version but not actually working, backend throwing a option not suport error.

[janaka]

@trrwilson related: do you know if the next service API release will include the ability to set tool_choice: "required"? (as described here and here)

That one's there now! required was recently made available on the 2024-07-01-preview service API label; here's the line where it's newly defined in the spec and here's the prior preview version where it was conspicuously absent.

We haven't yet published client library updates for all languages and protocol/REST-level calls may needed for those cases (.NET/Java/Go), but from an API perspective it's now supported.

FYI - just tried tool_choice=required and not working for us. Guessing best to handling via a support request.

[westernspion]

Is there any update on this? This breaks langgraph pretty horridly as it doesn't natively support parallelism

[KazumaMurao]

Is seems parallel_tool_calls parameter is supported with Azure OpenAI API version 2024-08-01-preview released yesterday.

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure#function-calling-with-structured-outputs

[janaka]

just been upgrading and test (via the C# lib) against a new 06-08-2024 deployment in eastus - though the Azure SDK doesn't have the REST API version add, so using reflection to set that).

• tool_choice=required - being accepted
• parallel_tool_calls=false - being accepted
• structured output - checking out at the moment

[corytomlinson]

@trrwilson can you please provide another update?

[nyanp]

Although it is not officially documented, the REST API for 2024-08-01-preview actually accepted the parallel_tool_calls parameter. It seems to be a bug in the API documentation.

[janaka]

Although it is not officially documented, the REST API for 2024-08-01-preview actually accepted the parallel_tool_calls parameter. It seems to be a bug in the API documentation.

Yep, works for us also. 'eastus'

[westernspion]

Can confirm this is working on api version, 2024-08-01-preview

[corytomlinson]

@nyanp @janaka @westernspion it doesn't work in all endpoints in all regions and 2024-08-01-preview is an undocumented API version. I still get many bad request errors even thought it appears to be "working". That makes it unusable so I was hoping for an official update on the timeline of completion to all endpoint in supported regions.

I also reached out to support to get an update since there has been no update here and received this back yesterday...

"Regarding this matter, the new parameter parallel_tool_calls is now supported by GPT4o-mini and 80% of GPT4o models. We are in the process of extending this support to additional GPT4o endpoints, with full implementation anticipated by the end of September."