Support for new o1 model

[simonw]

Released in the API today: https://platform.openai.com/docs/models#o1

See also https://openai.com/index/o1-and-new-tools-for-developers/

[simonw]

The latest o1 model supports both text and image inputs, and produces text outputs (including Structured Outputs). o1-mini currently only supports text inputs and outputs.

[simonw]

Not clear from the docs if streaming and system prompts are supported yet.

I don't think I have access to the model, must be rolling out:

llm -m o1 -s 'speak like santa' 'what should I ask for for christmas?'

Error: Error code: 404 - {'error': {'message': 'The model o1 does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

[simonw]

https://openai.com/index/o1-and-new-tools-for-developers/#openai-o1-in-the-api says it requires usage tier 5, but I do have that usage tier according to https://platform.openai.com/settings/organization/limits

[simonw]

We are rolling out access incrementally while working to expand access to additional usage tiers and ramping up rate limits

[simonw]

I experimented and o1-preview and o1-mini both support streaming now, but still don't support system prompts.

[simonw]

Here's my o1 diff, I won't land it until I've successfully tested it:

diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
index affd38a..eefc966 100644
--- a/llm/default_plugins/openai_models.py
+++ b/llm/default_plugins/openai_models.py
@@ -67,6 +67,12 @@ def register_models(register):
         aliases=("gpt-4-turbo-preview", "4-turbo", "4t"),
     )
     # o1
+    for model_id in ("o1", "o1-2024-12-17"):
+        register(
+            Chat(model_id, vision=True),
+            AsyncChat(model_id, vision=True),
+        )
+
     register(
         Chat("o1-preview", allows_system_prompt=False),
         AsyncChat("o1-preview", allows_system_prompt=False),

[simonw]

Spotted on https://platform.openai.com/docs/api-reference/chat/create

[simonw]

There's a new option called "reasoning effort": https://platform.openai.com/docs/api-reference/chat/create#chat-create-reasoning_effort

o1 models only. Values are low, medium, and high.

[simonw]

I still don't have access to this model myself!

In the meantime you can try out the branch with it in like this:

uvx --with https://github.com/simonw/llm/archive/refs/heads/o1.zip llm -m o1 hi

[simonw]

llm -m o1-2024-12-17 'write a puzzle about cheesecakes'

Error: Error code: 400 - {'error': {'message': "Unsupported value: 'stream' does not support true with this model. Supported values are: false.", 'type': 'invalid_request_error', 'param': 'stream', 'code': 'unsupported_value'}}

Looks like streaming is not supported by o1.

Surprising because it works for o1-mini and o1-preview.

[simonw]

Two test prompts run through o1:

---

2025-01-10T23:59:20 conversation: 01jh9ahjf1gvq3dr3jxvrtwmsv

Model: o1

Prompt:

write a puzzle about cheesecakes

Response:

Below is a short logic puzzle themed around cheesecakes. Try to deduce which friend baked which flavor, in what shape, and with which crust. Have fun!

────────────────────────────────────────────────────────
THE GREAT CHEESECAKE MYSTERY
────────────────────────────────────────────────────────

Four friends—Angela, Bradley, Carmen, and Dylan—each baked a distinctive cheesecake. Here’s what you know:

1. Each cheesecake has a unique flavor: Blueberry, Strawberry, Chocolate, or Lemon.
2. Each cheesecake is baked in a different shape: Round, Square, Rectangular, or Heart-Shaped.
3. Each cheesecake features a different crust: Graham, Oreo, Digestive, or Shortbread.

Use the following clues to figure out who made which cheesecake, what shape it is, and the type of crust:

CLUES

1. Carmen’s cheesecake is neither Lemon nor Square.
2. The Blueberry cheesecake has a Shortbread crust.
3. Angela’s cake has an Oreo crust and is not Heart-Shaped.
4. Dylan’s cheesecake is Rectangular.
5. The Strawberry cheesecake uses a Graham crust and is not made by Dylan.
6. Bradley’s cheesecake is Heart-Shaped and not Chocolate.

From these clues, can you determine the flavor, shape, and crust of each friend’s cheesecake?

────────────────────────────────────────────────────────

Good luck unraveling the Great Cheesecake Mystery!

Token usage:

13 input, 1,132 output, {"completion_tokens_details": {"reasoning_tokens": 832}}

2025-01-11T00:00:24 conversation: 01jh9akdm7yag85xkjpfs34jp5

Model: o1

Prompt:

write a very short puzzle about cheesecakes

Response:

Annie has three cheesecakes: Strawberry, Chocolate, and Blueberry. She places each one into a box labeled “Strawberry,” “Chocolate,” or “Blueberry,” but all three labels are wrong. When she takes a slice from the box labeled “Strawberry,” the cheesecake inside turns out to be Chocolate. Which flavor goes in each labeled box?

Token usage:

15 input, 1,180 output, {"completion_tokens_details": {"reasoning_tokens": 1088}}

[simonw]

And an image test:

---

2025-01-11T00:02:14 conversation: 01jh9apqgvbtxh1w9g4hcktkr8

Model: o1

Prompt:

describe image

Attachments

1. image/jpeg: https://static.simonwillison.net/static/2024/pelicans.jpg

Response:

It appears to show a large gathering of brown pelicans (and possibly other shorebirds) clustered tightly together on rocky ground near a body of water. The pelicans’ long bills and expandable throat pouches are visible as they rest or preen among a mixture of smaller, darker birds. The background includes calm water, and the overall scene conveys a busy congregation of birds along the shoreline.

Token usage:

218 input, 730 output, {"completion_tokens_details": {"reasoning_tokens": 640}}

[simonw]

I tested async mode with this as well:

llm -m o1 -a https://static.simonwillison.net/static/2024/pelicans.jpg 'describe image' --async