Skip to content

Commit

Permalink
Fix object property value in mlx_lm.server chat completions response …
Browse files Browse the repository at this point in the history
…to match OpenAI spec (#1119)

These were "chat.completions" and "chat.completions.chunk"
but should be "chat.completion" and "chat.completion.chunk"
for compatibility with clients expecting an OpenAI API.

In particular, this solves a problem in which aider 0.64.1 reports
hitting a token limit on any completion request, no matter how small,
despite apparently correct counts in the usage property.

Refer to:

https://platform.openai.com/docs/api-reference/chat/object

> object string
> The object type, which is always chat.completion.

https://platform.openai.com/docs/api-reference/chat/streaming

> object string
> The object type, which is always chat.completion.chunk.
  • Loading branch information
kconner authored Nov 25, 2024
1 parent 0f13539 commit 0ffdb6d
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 4 deletions.
2 changes: 1 addition & 1 deletion llms/mlx_lm/SERVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ curl localhost:8080/v1/chat/completions \

- `system_fingerprint`: A unique identifier for the system.

- `object`: Any of "chat.completions", "chat.completions.chunk" (for
- `object`: Any of "chat.completion", "chat.completion.chunk" (for
streaming), or "text.completion".

- `model`: The model repo or path (e.g. `"mlx-community/Llama-3.2-3B-Instruct-4bit"`).
Expand Down
4 changes: 1 addition & 3 deletions llms/mlx_lm/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -589,9 +589,7 @@ def handle_chat_completions(self) -> List[int]:

# Determine response type
self.request_id = f"chatcmpl-{uuid.uuid4()}"
self.object_type = (
"chat.completions.chunk" if self.stream else "chat.completions"
)
self.object_type = "chat.completion.chunk" if self.stream else "chat.completion"
if (
hasattr(self.tokenizer, "apply_chat_template")
and self.tokenizer.chat_template
Expand Down

0 comments on commit 0ffdb6d

Please sign in to comment.