Skip to content

Commit

Permalink
Add ollama phi4 context size defaults (#23036)
Browse files Browse the repository at this point in the history
Add `phi4` maximum context length (128K).
By default this clamps to `16384` but if you have enough video memory
you can set it higher or connect to a non-local machine via settings:

```json
"language_models": {
  "ollama": {
    "api_url": "http://localhost:11434",
    "available_models": [
      {
        "name": "phi4",
        "display_name": "Phi4 64K",
         "max_tokens": 65536
      }
    ]
  }
}
```

Release Notes:

- Improve support for Phi4 with ollama.
  • Loading branch information
notpeter committed Jan 15, 2025
1 parent aa2d637 commit 9677a7d
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions crates/ollama/src/ollama.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ fn get_max_tokens(name: &str) -> usize {
"codellama" | "starcoder2" => 16384,
"mistral" | "codestral" | "mixstral" | "llava" | "qwen2" | "qwen2.5-coder"
| "dolphin-mixtral" => 32768,
"llama3.1" | "phi3" | "phi3.5" | "command-r" | "deepseek-coder-v2" | "yi-coder"
| "llama3.2" => 128000,
"llama3.1" | "phi3" | "phi3.5" | "phi4" | "command-r" | "deepseek-coder-v2"
| "yi-coder" | "llama3.2" => 128000,
_ => DEFAULT_TOKENS,
}
.clamp(1, MAXIMUM_TOKENS)
Expand Down

0 comments on commit 9677a7d

Please sign in to comment.