Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Mistral Codestral doesn't work #8279

Open
ohmyboroda opened this issue Feb 5, 2025 · 13 comments
Open

[Bug]: Mistral Codestral doesn't work #8279

ohmyboroda opened this issue Feb 5, 2025 · 13 comments
Labels
bug Something isn't working

Comments

@ohmyboroda
Copy link

What happened?

When sending a POST request to /v1/completions, an error occurs

{"error":{"message":"litellm.BadRequestError: You passed in model=text-completion-codestral/codestral-latest. There is no 'model_name' with this string \nReceived Model Group=text-completion-codestral/codestral-latest\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}}

How to repeat:

  • Add Mistral AI model > mistral/codestral-latest
  • Make a POST request
curl -X POST http://localhost:4000/v1/completions 
-H "Content-Type: application/json" 
-H "Authorization: Bearer sk-1234" 
-d '{
    "model": "mistral/codestral-latest",
    "prompt": "Once upon a time"
}'

Relevant log output

2025-02-05T14:13:37.230709727Z            ^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230711616Z     ...<42 lines>...
2025-02-05T14:13:37.230713727Z     )
2025-02-05T14:13:37.230715505Z     ^
2025-02-05T14:13:37.230717375Z   File "/usr/lib/python3.13/site-packages/openai/_base_client.py", line 1849, in post
2025-02-05T14:13:37.230719579Z     return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
2025-02-05T14:13:37.230721782Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230723838Z   File "/usr/lib/python3.13/site-packages/openai/_base_client.py", line 1543, in request
2025-02-05T14:13:37.230726171Z     return await self._request(
2025-02-05T14:13:37.230730060Z            ^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230732320Z     ...<5 lines>...
2025-02-05T14:13:37.230734301Z     )
2025-02-05T14:13:37.230736079Z     ^
2025-02-05T14:13:37.230738005Z   File "/usr/lib/python3.13/site-packages/openai/_base_client.py", line 1644, in _request
2025-02-05T14:13:37.230740449Z     raise self._make_status_error_from_response(err.response) from None
2025-02-05T14:13:37.230742579Z openai.AuthenticationError: Error code: 401 - {'message': 'Unauthorized', 'request_id': '44dd2613eb987155420fa77dfecc4541'}
2025-02-05T14:13:37.230744875Z 
2025-02-05T14:13:37.230746671Z During handling of the above exception, another exception occurred:
2025-02-05T14:13:37.230748783Z 
2025-02-05T14:13:37.230750801Z Traceback (most recent call last):
2025-02-05T14:13:37.230752746Z   File "/usr/lib/python3.13/site-packages/litellm/main.py", line 3915, in atext_completion
2025-02-05T14:13:37.230754986Z     response = await response
2025-02-05T14:13:37.230757005Z                ^^^^^^^^^^^^^^
2025-02-05T14:13:37.230758912Z   File "/usr/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 820, in acompletion
2025-02-05T14:13:37.230761209Z     raise OpenAIError(
2025-02-05T14:13:37.230763079Z         status_code=status_code, message=message, headers=error_headers
2025-02-05T14:13:37.230765264Z     )
2025-02-05T14:13:37.230767153Z litellm.llms.openai.common_utils.OpenAIError: Error code: 401 - {'message': 'Unauthorized', 'request_id': '44dd2613eb987155420fa77dfecc4541'}
2025-02-05T14:13:37.230769597Z 
2025-02-05T14:13:37.230771560Z During handling of the above exception, another exception occurred:
2025-02-05T14:13:37.230773968Z 
2025-02-05T14:13:37.230775783Z Traceback (most recent call last):
2025-02-05T14:13:37.230777727Z   File "/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 3834, in completion
2025-02-05T14:13:37.230781209Z     response = await llm_call
2025-02-05T14:13:37.230785727Z                ^^^^^^^^^^^^^^
2025-02-05T14:13:37.230787542Z INFO:     172.23.0.4:40108 - "POST /v1/completions HTTP/1.1" 401 Unauthorized
2025-02-05T14:13:37.230787746Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2074, in atext_completion
2025-02-05T14:13:37.230790005Z     raise e
2025-02-05T14:13:37.230791912Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2062, in atext_completion
2025-02-05T14:13:37.230803635Z     response = await self.async_function_with_fallbacks(**kwargs)
2025-02-05T14:13:37.230806523Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230808894Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3017, in async_function_with_fallbacks
2025-02-05T14:13:37.230811561Z     raise original_exception
2025-02-05T14:13:37.230813672Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2835, in async_function_with_fallbacks
2025-02-05T14:13:37.230816116Z     response = await self.async_function_with_retries(*args, **kwargs)
2025-02-05T14:13:37.230818375Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230820635Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3126, in async_function_with_retries
2025-02-05T14:13:37.230823709Z     self.should_retry_this_error(
2025-02-05T14:13:37.230828894Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
2025-02-05T14:13:37.230830857Z         error=e,
2025-02-05T14:13:37.230832838Z         ^^^^^^^^
2025-02-05T14:13:37.230834690Z     ...<4 lines>...
2025-02-05T14:13:37.230836931Z         content_policy_fallbacks=content_policy_fallbacks,
2025-02-05T14:13:37.230839005Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230841153Z     )
2025-02-05T14:13:37.230842968Z     ^
2025-02-05T14:13:37.230844820Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3316, in should_retry_this_error
2025-02-05T14:13:37.230847246Z     raise error  # then raise error
2025-02-05T14:13:37.230849487Z     ^^^^^^^^^^^
2025-02-05T14:13:37.230851542Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3101, in async_function_with_retries
2025-02-05T14:13:37.230853838Z     response = await self.make_call(original_function, *args, **kwargs)
2025-02-05T14:13:37.230856024Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230858098Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3217, in make_call
2025-02-05T14:13:37.230860376Z     response = await response
2025-02-05T14:13:37.230862320Z                ^^^^^^^^^^^^^^
2025-02-05T14:13:37.230864357Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2143, in _atext_completion
2025-02-05T14:13:37.230869487Z     raise e
2025-02-05T14:13:37.230871635Z   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2130, in _atext_completion
2025-02-05T14:13:37.230873876Z     response = await response
2025-02-05T14:13:37.230875894Z                ^^^^^^^^^^^^^^
2025-02-05T14:13:37.230877801Z   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1358, in wrapper_async
2025-02-05T14:13:37.230880116Z     raise e
2025-02-05T14:13:37.230881913Z   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1217, in wrapper_async
2025-02-05T14:13:37.230884190Z     result = await original_function(*args, **kwargs)
2025-02-05T14:13:37.230886227Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230888302Z   File "/usr/lib/python3.13/site-packages/litellm/main.py", line 3950, in atext_completion
2025-02-05T14:13:37.230890598Z     raise exception_type(
2025-02-05T14:13:37.230892561Z           ~~~~~~~~~~~~~~^
2025-02-05T14:13:37.230894394Z         model=model,
2025-02-05T14:13:37.230896320Z         ^^^^^^^^^^^^
2025-02-05T14:13:37.230898172Z     ...<3 lines>...
2025-02-05T14:13:37.230900209Z         extra_kwargs=kwargs,
2025-02-05T14:13:37.230902116Z         ^^^^^^^^^^^^^^^^^^^^
2025-02-05T14:13:37.230904042Z     )
2025-02-05T14:13:37.230905913Z     ^
2025-02-05T14:13:37.230907709Z   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2201, in exception_type
2025-02-05T14:13:37.230910265Z     raise e
2025-02-05T14:13:37.230912098Z   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 390, in exception_type
2025-02-05T14:13:37.230914653Z     raise AuthenticationError(
2025-02-05T14:13:37.230916542Z     ...<5 lines>...
2025-02-05T14:13:37.230918579Z     )
2025-02-05T14:13:37.230921413Z litellm.exceptions.AuthenticationError: litellm.AuthenticationError: AuthenticationError: MistralException - Error code: 401 - {'message': 'Unauthorized', 'request_id': '44dd2613eb987155420fa77dfecc4541'}
2025-02-05T14:13:37.230925246Z Received Model Group=mistral/codestral-latest
2025-02-05T14:13:37.230927302Z Available Model Group Fallbacks=None
2025-02-05T14:13:38.580911130Z INFO:     127.0.0.1:34100 - "GET /health/liveliness HTTP/1.1" 200 OK
2025-02-05T14:13:43.849372629Z INFO:     127.0.0.1:42314 - "GET /health/liveliness HTTP/1.1" 200 OK
2025-02-05T14:13:49.110798057Z INFO:     127.0.0.1:42330 - "GET /health/liveliness HTTP/1.1" 200 OK
2025-02-05T14:13:54.377095247Z INFO:     127.0.0.1:48350 - "GET /health/liveliness HTTP/1.1" 200 OK
2025-02-05T14:13:59.686560317Z INFO:     127.0.0.1:48356 - "GET /health/liveliness HTTP/1.1" 200 OK
2025-02-05T14:14:04.976347480Z INFO:     127.0.0.1:53192 - "GET /health/liveliness HTTP/1.1" 200 OK

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.60.2

Twitter / LinkedIn details

No response
@ohmyboroda ohmyboroda added the bug Something isn't working label Feb 5, 2025
@ohmyboroda
Copy link
Author

Looks like a similar issue #8058

@ohmyboroda ohmyboroda changed the title [Bug]: [Bug]: Mistral Codestral doesn't work Feb 5, 2025
@miraclebakelaser
Copy link
Contributor

Assuming it's the same issue, I wrote the following in the other issue:

Can you check that you're using the right API key? The Codestral API keys are different from the regular API keys that Mistral issues to paying users of their chat models.

They're also issued on a different page, which makes it all the more confusing.

https://console.mistral.ai/codestral

Is that what you're running into?

@ohmyboroda
Copy link
Author

ohmyboroda commented Feb 5, 2025 •
edited
Loading

Assuming it's the same issue, I wrote the following in the other issue:

Can you check that you're using the right API key? The Codestral API keys are different from the regular API keys that Mistral issues to paying users of their chat models.
They're also issued on a different page, which makes it all the more confusing.
https://console.mistral.ai/codestral

Is that what you're running into?

I'm 100% sure the key is correct and copied from https://console.mistral.ai/codestral. It works directly through Postman and directly in Continue in VSCode/PyCharm.

Image Image Image

@ohmyboroda
Copy link
Author

Similar problem for gpt-o1, gpt-o3-mini.

Error occurred while generating model response. Please try again. Error: Error: 404 litellm.NotFoundError: OpenAIException - Error code: 404 - {'error': {'message': 'The model `o1` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}} Received Model Group=gpt-o1 Available Model Group Fallbacks=None

Error occurred while generating model response. Please try again. Error: Error: 404 litellm.NotFoundError: OpenAIException - Error code: 404 - {'error': {'message': 'The model `o3-mini` does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}} Received Model Group=gpt-o3-mini Available Model Group Fallbacks=None

4o, 4o-mini, 1o-mini, 1o-preview works fine.

@krrishdholakia
Copy link
Contributor

The model o1 does not exist or you do not have access to it.

@ohmyboroda OpenAI is saying you don't have model access

The model o3-mini does not exist or you do not have access to it.

same here

@krrishdholakia
Copy link
Contributor

Can you please share the raw request made by litellm to the provider - use --detailed_debug

https://docs.litellm.ai/docs/proxy/debugging#debug-logs

@ohmyboroda
Copy link
Author

ohmyboroda commented Feb 5, 2025 •
edited
Loading

The model o1 does not exist or you do not have access to it.

@ohmyboroda OpenAI is saying you don't have model access

The model o3-mini does not exist or you do not have access to it.

same here

Ok, make sense. I will check it later. Thanks.

Can you please share the raw request made by litellm to the provider - use --detailed_debug

https://docs.litellm.ai/docs/proxy/debugging#debug-logs

Ok, i will do that soon.

@ohmyboroda
Copy link
Author

@krrishdholakia apologies if this is a basic question, but where is the config.yaml file located for the command:
litellm --config /path/to/config.yaml --detailed_debug

Image

Does LiteLLM generate this file automatically, or do I need to create it manually? If so, what is the recommended location?

Thanks in advance for your help!

@krrishdholakia
Copy link
Contributor

It's user defined.

I'm confused, how did you run the proxy initially?

@ohmyboroda
Copy link
Author

It's user defined.

I'm confused, how did you run the proxy initially?

Self-hosted Docker Compose.

@krrishdholakia
Copy link
Contributor

but how did you add the codestral model?

@ohmyboroda
Copy link
Author

I wrote this at the very beginning of the thread

How to repeat:
Add Mistral AI model > mistral/codestral-latest
Make a POST request

Image

@ohmyboroda ohmyboroda reopened this Feb 5, 2025
@ohmyboroda
Copy link
Author

Accidentally completed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@krrishdholakia @miraclebakelaser @ohmyboroda