[Bug]: GPT multimodal models don't support file uploads

[KBazGenexus]

What happened?

@krrishdholakia

When trying to pass a file within the message content to a multimodal model like GPT-4o, I found that it does not allow this. In contrast, with other models, such as Gemini, this process works seamlessly. It would be very helpful to have this functionality in OpenAI models to facilitate integration.

Example:

from litellm import completion
import base64

response = completion(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "You are a very professional document summarization specialist. Please summarize the given document."},
                {
                    "type": "image_url",
                    "image_url": {"url": "data:application/pdf;base64,UEsDBBQABgAIAAAAIQBURqfAkQEAAI8GAAATAAgCW0NvbnRlbnRfVHlwZXNd"}
                }
            ]
        }
    ]
}

Error:

BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'Invalid MIME type. Only image types are supported.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_image_format'}}

Is there a plan to support passing files in base64 format within the message content for multimodal models, similar to what is allowed with other providers like Gemini?

Thanks!

Relevant log output

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/litellm/llms/OpenAI/openai.py in completion(self, model_response, timeout, optional_params, logging_obj, model, messages, print_verbose, api_key, api_base, acompletion, litellm_params, logger_fn, headers, custom_prompt_dict, client, organization, custom_llm_provider, drop_params)
    809                     else:
--> 810                         raise e
    811         except OpenAIError as e:

17 frames
BadRequestError: Error code: 400 - {'error': {'message': 'Invalid MIME type. Only image types are supported.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_image_format'}}

During handling of the above exception, another exception occurred:

OpenAIError                               Traceback (most recent call last)
OpenAIError: Error code: 400 - {'error': {'message': 'Invalid MIME type. Only image types are supported.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_image_format'}}

During handling of the above exception, another exception occurred:

BadRequestError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/litellm/litellm_core_utils/exception_mapping_utils.py in exception_type(model, original_exception, custom_llm_provider, completion_kwargs, extra_kwargs)
    295                 ):
    296                     exception_mapping_worked = True
--> 297                     raise BadRequestError(
    298                         message=f"{exception_provider} - {message}",
    299                         llm_provider=custom_llm_provider,

BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'Invalid MIME type. Only image types are supported.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_image_format'}}

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.60.5

Twitter / LinkedIn details

https://www.linkedin.com/in/karen-baz/