api_key_name isn't used in extra-openai-models.yaml if OPENAI_API_KEY env var is set

[itsderek23]

If I add an openai-compatible model to extra-openai-models.yaml and specify an api_key_name, the CLI does not appear to use the key value. Instead, it appears to use the openai key.

Example key file:

{
  "// Note": "This file stores secret API credentials. Do not share!",
  "openai": "sk-redacted",
  "opstower": "redacted"
}

Example CLI command (inspecting server logs shows that the openai key value was passed):

llm -m opstower 'how many ec2 instances'

This works:

llm -m opstower 'how many ec2 instances' --key opstower

Orignally mentioned by @klauern in #139 (comment)_

This is on version 0.7.

[itsderek23]

I believe this happens if the env var OPENAI_API_KEY is set. If I unset this, it works.

Looking at get_key, the env_var is checked prior to default_key.

[simonw]

Relevant code:

llm/llm/__init__.py

Lines 97 to 105 in a4f55e9

	def get_key(key_arg, default_key, env_var=None):
	keys = load_keys()
	if key_arg in keys:
	return keys[key_arg]
	if key_arg:
	return key_arg
	if env_var and os.environ.get(env_var):
	return os.environ[env_var]
	return keys.get(default_key)

[simonw]

I don't like those variable names, and that function needs proper documentation. It's confusing to look at.

[simonw]

Oh that's weird... the main place that keys come from doesn't even call that utility method!

llm/llm/models.py

Lines 209 to 234 in a4f55e9

	class Model(ABC):
	model_id: str
	key: Optional[str] = None
	needs_key: Optional[str] = None
	key_env_var: Optional[str] = None
	can_stream: bool = False
	class Options(_Options):
	pass
	def get_key(self):
	if self.needs_key is None:
	return None
	if self.key is not None:
	return self.key
	if self.key_env_var is not None:
	key = os.environ.get(self.key_env_var)
	if key:
	return key
	message = "No key found - add one using 'llm keys set {}'".format(
	self.needs_key
	)
	if self.key_env_var:
	message += " or set the {} environment variable".format(self.key_env_var)
	raise NeedsKeyException(message)

[simonw]

The main place the get_key() function is called is here:

llm/llm/cli.py

Lines 211 to 213 in a4f55e9

	# Provide the API key, if one is needed and has been provided
	if model.needs_key:
	model.key = get_key(key, model.needs_key, model.key_env_var)

Where key is the --key X option from the CLI call.

[itsderek23]

I'll try to do a PR for this next week. Thanks for the pointers.

[simonw]

I wrote this:

diff --git a/llm/models.py b/llm/models.py
index 10006bf..4cfd1e9 100644
--- a/llm/models.py
+++ b/llm/models.py
@@ -217,15 +217,24 @@ class Model(ABC):
         pass
 
     def get_key(self):
+        from llm import get_key
+
         if self.needs_key is None:
+            # This model doesn't use an API key
             return None
+
         if self.key is not None:
+            # Someone already set model.key='...'
             return self.key
-        if self.key_env_var is not None:
-            key = os.environ.get(self.key_env_var)
-            if key:
-                return key
 
+        # Attempt to load a key using llm.get_key()
+        key = get_key(
+            explicit_key=None, key_alias=self.needs_key, env_var=self.key_env_var
+        )
+        if key:
+            return key
+
+        # Show a useful error message
         message = "No key found - add one using 'llm keys set {}'".format(
             self.needs_key
         )

But I need to come up with a robust test plan - both manual and automated - for ensuring it does what it needs to do.

[simonw]

I can test this more easily using the OpenRouter support I added in:

• Get openrouter.ai working with new headers: feature for extra-openai-models.yaml #165

[simonw]

Managed to replicate this bug:

llm -m claude 'say hi and your name'                 

 Hello, my name is Claude. It's nice to meet you!

But with the environment variable set:

OPENAI_API_KEY=x llm -m claude 'say hi and your name'

Error: Invalid credentials

[simonw]

This was a deliberate design decision:

llm/llm/__init__.py

Lines 114 to 118 in 341dbce

	# Environment variables over-ride the default key
	if env_var and os.environ.get(env_var):
	return os.environ[env_var]
	# Return the key stored for the default alias
	return stored_keys.get(key_alias)

I'm trying to remember why.

[simonw]

I'm going to change that decision.

If you want to use the key from your environment variable instead, you can do this:

llm "my prompt" --key $OPENAI_API_KEY

[simonw]

Updated documentation: https://llm.datasette.io/en/latest/setup.html#keys-in-environment-variables

[itsderek23]

Thank you @simonw !

[cmungall]

This is more of a note for myself if I end up searching the docs again.

When using extra-openai-models.yaml, it seems natural to assume you can do this:

- model_name: lbl/llama-3
  model_id: lbl/llama-3
  api_base: "https://api.cborg.lbl.gov"
  api_key_name: CBORG_API_KEY

and it will pick up CBORG_API_KEY from the env. In fact this doesn't happen - it needs to be done using llm keys, or on the command line as specified above.

I prefer the former, so I usually give a more conventional key name:

- model_name: lbl/llama-3
  model_id: lbl/llama-3
  api_base: "https://api.cborg.lbl.gov"
  api_key_name: cborg

and manage this using

llm keys set cborg