Image analyzer and Video Analyzer not work latest version

[heblasco]

When uploading an image or a video to analyze them.
Application is not working retuning a Bad Request.

[rliberoff]

So the issue is produced by the new azure-openai-token-limit and azure-openai-emit-token-metric policies which are (currently) incompatible with requests for vision.

In order to resolve this issue, we will introduce conditions to ascertain whether the policy should be applied or not.

References: Supported Azure OpenAI Service models

[rliberoff]

As additional context for a future fix, the issue occurs when using OpenAI models with Vision, like gpt-4-vision (i.e., gpt-4 with vision-preview version).

The error is:

HTTP/1.1 400 Bad Request
content-length: 262
content-type: application/json
date: Thu, 06 Jun 2024 13:33:01 GMT
ocp-apim-apiid: azure-open-ai-api
ocp-apim-operationid: ChatCompletions_Create
ocp-apim-subscriptionid: master
ocp-apim-trace-location: …
request-context: appId=cid-v1:c9cd3423-c7e6-4137-bf8d-c70fd6fb3166
vary: Origin
    
{
    "statusCode": 400,
    "message": "Unable to parse and estimate tokens from incoming request. Please ensure incoming request is of one of the following types: 'Chat Completion', 'Completion', 'Embeddings' and works with current prompt estimation mode of 'Auto'."
}

Tracing the request (using an Azure API Management) the following errors are recorded:

emit-metric (5.120 ms)
"No diagnostic settings have metric enabled. Metric emission skipped."

azure-openai-token-limit (11.945 ms)
{
    "message": "Expression was successfully evaluated.",
    "expression": "context.Request.IpAddress",
    "value": "20.121.82.216"
}

This is perhaps the most significant trace message. It shows an exception thrown by – what looks like – the policy for token limits:

language-model-request-handler (229.061 ms)
{
    "message": "Encountered exception when deserializing stream into type Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.Throttling.LMTokenLimit.OpenAIChatRequestContract. Exception details: System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.messages[0].content | LineNumber: 0 | BytePositionInLine: 110. ---> System.InvalidOperationException: Cannot get the value of a token type 'StartArray' as a string.\r\n   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ExpectedString(JsonTokenType tokenType)\r\n   at System.Text.Json.Utf8JsonReader.GetString()\r\n   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)\r\n   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\r\n   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)\r\n   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)\r\n   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)\r\n   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)\r\n   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)\r\n   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)\r\n   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\r\n   --- End of inner exception stack trace ---\r\n   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)\r\n   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\r\n   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.ContinueDeserialize(ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack)\r\n   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Stream utf8Json)\r\n   at Gateway.RateLimit.LMTokenLimitHelpers.TryParseJson[T](String source, IPipelineContext context, ILog logger, MemoryStream potentialJsonStream, T& result, Boolean expectSuccess) in C:\\__w\\1\\s\\Proxy\\Gateway.Policies.RateLimit\\LMTokenLimitHelpers.cs:line 53"
}

language-model-request-handler (15.635 ms)
{
    "message": "Unable to parse and determine prompt type from incoming request. Expected type was Auto. Contract request type "
}

language-model-request-handler (1.421 ms)
{
    "message": "Unable to parse and estimate tokens from incoming request. Please ensure incoming request is of one of the following types: 'Chat Completion', 'Completion', 'Embeddings' and works with current prompt estimation mode of 'Auto'."
}

---

The request message looks like this:

{
  "enhancements": {
    "ocr": {
      "enabled": true
    },
    "grounding": {
      "enabled": true
    }
  },
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are an AI assistant that helps people find information."
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "https://stgenai93df1.blob.core.windows.net/image-moderator/ring_and_book.jpg?sv=2023-08-03&se=2024-06-06T09%3A48%3A06Z&sr=c&sp=r&sig=9IPgl0shTAb3XBewYLsG%2Bf%2BPyjg43iAuTkS%2FwbU55X4%3D"
          }
        },
        {
          "type": "text",
          "text": "Describe the image in detail"
        }
      ]
    }
  ],
  "temperature": 0.7,
  "top_p": 0.95,
  "max_tokens": 800,
  "stream": false
}