llm.get_models() returns multiple instances of the same model

[simonw]

Just noticed this:

import llm
from pprint import pprint
pprint(llm.get_models())

[<Chat 'gpt-4o'>,
 <Chat 'gpt-4o'>,
 <Chat 'gpt-4o-mini'>,
 <Chat 'gpt-4o-mini'>,
 <Chat 'gpt-4o-audio-preview'>,
 <Chat 'gpt-3.5-turbo'>,
 <Chat 'gpt-3.5-turbo'>,
 <Chat 'gpt-3.5-turbo'>,
 <Chat 'gpt-3.5-turbo-16k'>,
 <Chat 'gpt-3.5-turbo-16k'>,
 <Chat 'gpt-3.5-turbo-16k'>,
 <Chat 'gpt-4'>,
 <Chat 'gpt-4'>,
 <Chat 'gpt-4'>,
 <Chat 'gpt-4-32k'>,
 <Chat 'gpt-4-32k'>,
 <Chat 'gpt-4-1106-preview'>,
 <Chat 'gpt-4-0125-preview'>,
 <Chat 'gpt-4-turbo-2024-04-09'>,
 <Chat 'gpt-4-turbo'>,
 <Chat 'gpt-4-turbo'>,
 <Chat 'gpt-4-turbo'>,
 <Chat 'gpt-4-turbo'>,
 <Chat 'o1-preview'>,
 <Chat 'o1-mini'>,
 <Completion 'gpt-3.5-turbo-instruct'>,
 <Completion 'gpt-3.5-turbo-instruct'>,
 <Completion 'gpt-3.5-turbo-instruct'>]

[simonw]

It's suspicious that gpt-4-turbo shows up 4 times, but o1-preview and o1-mini only once.

Code in question:

llm/llm/default_plugins/openai_models.py

Lines 30 to 77 in e78fea1

	def register_models(register):
	# GPT-4o
	register(
	Chat("gpt-4o", vision=True), AsyncChat("gpt-4o", vision=True), aliases=("4o",)
	)
	register(
	Chat("gpt-4o-mini", vision=True),
	AsyncChat("gpt-4o-mini", vision=True),
	aliases=("4o-mini",),
	)
	register(
	Chat("gpt-4o-audio-preview", audio=True),
	AsyncChat("gpt-4o-audio-preview", audio=True),
	)
	# 3.5 and 4
	register(
	Chat("gpt-3.5-turbo"), AsyncChat("gpt-3.5-turbo"), aliases=("3.5", "chatgpt")
	)
	register(
	Chat("gpt-3.5-turbo-16k"),
	AsyncChat("gpt-3.5-turbo-16k"),
	aliases=("chatgpt-16k", "3.5-16k"),
	)
	register(Chat("gpt-4"), AsyncChat("gpt-4"), aliases=("4", "gpt4"))
	register(Chat("gpt-4-32k"), AsyncChat("gpt-4-32k"), aliases=("4-32k",))
	# GPT-4 Turbo models
	register(Chat("gpt-4-1106-preview"), AsyncChat("gpt-4-1106-preview"))
	register(Chat("gpt-4-0125-preview"), AsyncChat("gpt-4-0125-preview"))
	register(Chat("gpt-4-turbo-2024-04-09"), AsyncChat("gpt-4-turbo-2024-04-09"))
	register(
	Chat("gpt-4-turbo"),
	AsyncChat("gpt-4-turbo"),
	aliases=("gpt-4-turbo-preview", "4-turbo", "4t"),
	)
	# o1
	register(
	Chat("o1-preview", can_stream=False, allows_system_prompt=False),
	AsyncChat("o1-preview", can_stream=False, allows_system_prompt=False),
	)
	register(
	Chat("o1-mini", can_stream=False, allows_system_prompt=False),
	AsyncChat("o1-mini", can_stream=False, allows_system_prompt=False),
	)
	# The -instruct completion model
	register(
	Completion("gpt-3.5-turbo-instruct", default_max_tokens=256),
	aliases=("3.5-instruct", "chatgpt-instruct"),
	)

I'm suspicious of the aliases.

[simonw]

Here's the problem: we aren't de-duping for aliases here:

llm/llm/__init__.py

Lines 170 to 177 in e78fea1

	def get_models() -> List[Model]:
	"Get all registered models"
	return [model for model in get_model_aliases().values()]
	def get_async_models() -> List[AsyncModel]:
	"Get all registered async models"
	return [model for model in get_async_model_aliases().values()]

Bug was introduced in:

• llm.get_models() and .get_async_models() documented functions #640