azure-ai-inference sdk support for Managed Online Endpoint LLM deployments. AML, AI Foundry

[jakeatmsft]

• Package Name: azure-ai-inference
• Package Version:
• Operating System:
• Python Version:

Describe the bug
Customer has deployed multiple Managed Online Endpoints to AzureML as well as AI Foundry. Endpoints exposed a /score inference endpoint but are not compatible with azure-ai-inference sdk. When specifying scoring endpoint and using key credential, the sdk returns error of "Failed Dependencies"

To Reproduce
Steps to reproduce the behavior:

1. Deploy LLM to Managed Endpoint
2. Configure Key, Endpoint url: ('https://endpointname.westus2.inference.ml.azure.com/score)
3. Run simple completion example in docs: (https://learn.microsoft.com/en-us/azure/ai-studio/reference/reference-model-inference-api?tabs=python#inference-sdk-support)

Result:
Error:
HttpResponseError: Operation returned an invalid status 'Failed Dependency'
Content: {"detail":"Not Found"}

Expected behavior
Completion response returned

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dargilco.

[dargilco]

Adding @santiagxf

[dargilco]

@santiagxf to correct me if I'm wrong, but my understanding is that the /score route is for model-provider specific REST API, not the common "Azure AI Model Inference API" that the Python azure-ai-inference SDK supports. The common Inference API for chat completions has the route /chat/completions. @jakeatmsft What AI model are you using? Please try removing the /score from the endpoint you are using. The SDK automatically adds the /chat/completions to the given endpoint when making a ChatCompletionsClient.complete() call.

[santiagxf]

Thanks for reaching out @jakeatmsft. Can you please confirm the model you have deployed and that you are trying to use?

[jakeatmsft]

@santiagxf to correct me if I'm wrong, but my understanding is that the /score route is for model-provider specific REST API, not the common "Azure AI Model Inference API" that the Python azure-ai-inference SDK supports. The common Inference API for chat completions has the route /chat/completions. @jakeatmsft What AI model are you using? Please try removing the /score from the endpoint you are using. The SDK automatically adds the /chat/completions to the given endpoint when making a ChatCompletionsClient.complete() call.

This worked for my customer deploying llama-3.1-70b-instruct. I'll submit PR to update instructions.

[dargilco]

Jake's PR: #39038

[santiagxf]

I'm closing this issue as we have clarify that it was due to incorrect URL used in the inference client.