NotFound error when using mlx server

[cbschen1]

Dear mlx community,

I am working on MAC with apple silicon, and I ran into an issue running local LLMs using the mlx server. (the same code below works on llama-cpp server)

Below is the code I used:

openai_client = openai.OpenAI(api_key = "placeholder-api", base_url="http://localhost:8080")

response = openai_client.chat.completions.create(
model = 'mlx-community/Meta-Llama-3-70B-Instruct-4bit',
messages= [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": f"Question: {query}."},
]
)

I confirm that mlx server opens successfully, and running below on command line works

curl localhost:8080/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'

However, when I send query via the openai API, it gives me the NotFoundError: Not Found
"POST /chat/completions HTTP/1.1" 404 -

Any help would be sincerely appreciated!

[awni]

Fix is in #877