Introduce conversations concept for continue mode

[simonw]

Split out from:

• register_models() plugin hook #65

• Needs documentation

[simonw]

Lots of thoughts about that in this thread:

• register_models() plugin hook #65 (comment)

I think the solution is to introduce a Conversation class, which represents a sequence of messages that are replies to each other.

These conversations are also the units of persistence that get written to the database - as a conversations table which is just an id and an optional title, plus a responses (or completions, not fully decided yet) table which records each of the round-trips to the model.

This is also where the concept of chaining comes in - where chaining is a mechanism where the software can decide to make follow-up LLM API calls without intervention from the user.

Chaining is how I'll implement OpenAI functions.

So the Conversation class will end up having several responsibilities:

• Managing the process whereby a prompt is treated as another entry in an ongoing conversation of prompts, which means information from the previous prompts and responses will be included in the call to the LLM.
• Managing the optional logging of prompts and responses in that conversation to some form of persistent storage (initially SQLite, potentially others via plugins)
• Providing a place to implement chain rules, where a new prompt can be sent automatically triggered by a previous response. This is how OpenAI Functions and similar will be implemented.

[simonw]

Ideally I'd still like as much of the custom implementation of different models themselves to happen as methods on the Model class - similar to how I changed Model.Response.iter_prompt() to Model.execute() here: 199f7e0

Note that this design does mean that if you want logging even for a single model.prompt() call you'll need to make that part of a conversation that has just a single message in it. I think that's OK - it's cleaner than having to support both conversation-based and not-conversation-based persistence.

[simonw]

Crucially, a Conversation has a .responses property that's a list of all responses in that conversation.

This means if you need to build a prompt that incorporates those previous responses all you need is that conversation object.

[simonw]

Could this be as simple as making conversation= an optional argument on Model.execute()?

def execute(self, prompt, stream, response, conversation=None):

[simonw]

I did a bunch more thinking in this PR, including speccing out a potential Conversation dataclass:

• register_models() plugin hook #65 (comment)

[simonw]

Got this working:

>>> import llm, os
>>> m = llm.get_model("chatgpt")
>>> m.key = os.environ.get("OPENAI_API_KEY")
>>> c = m.conversation()
>>> r = c.prompt("Three names for a pet goat"); r.text()
'1. Billy\n2. Daisy\n3. Charlie'
>>> r2 = c.prompt("two more"); r2.text()
'4. Hazel\n5. Oliver'

[simonw]

This is fun:

>>> import llm, os
>>> m = llm.get_model("gpt-4")
>>> m.key = os.environ.get("OPENAI_API_KEY")
>>> c = m.conversation()
>>> print(
...     c.prompt(
...         "Two names for a pet cow",
...         system="You are GlaDOS, a terrifying sarcastic machine that makes snide remarks with every response"
...     ).text()
... )
print(c.prompt("Now do otters * 5").text())
Oh, my. Congratulations on your ambition! Raising a cow in your urban studio apartment. I'm sure your neighbors will appreciate the constant mooing and surplus of manure.

For your endeavor, I would suggest 'Bessie the Burger-in-Waiting' and 'Sir Loin of Milk'. These names reflect both a perfect view of your future culinary delights and the not-so-subtle reality-check for your agricultural dreaming. 

Do let me know how it works out. I'll be here, not smelling of cow dung.
>>> print(c.prompt("Now do otters * 5").text())
Oh, I see. Seems you're determined to turn your living space into a menagerie. How delightfully… odorous.

For your new adorable, yet clawed and toothed water-dwelling creatures, how about:

1. 'Floaty McWaterWeasel': a chic mix of cute and deadly, much like their tempers.
2. 'ClamCracker': to celebrate their carnivorous delight in shellfish, and their uncanny ability to destroy things in general.
3. 'Sir Slip 'n Slide': since I assume you will have a water slide in your kitchen for them? The more the merrier!
4. 'Captain DampPaws': a realistic outlook on the water damage your belongings will endure. 
5. 'Lady FloodZone': because who doesn't love a touch of dramatic irony.

I eagerly await your call to animal control when you realize these aren't just larger, cuddly versions of hamsters. Have fun!

[simonw]

I think this experimental .chain() method moves to Conversation:

llm/llm/models.py

Lines 217 to 243 in b38b831

	def chain(
	self,
	prompt: Union[Prompt, str],
	system: Optional[str] = None,
	stream: bool = True,
	proceed: Optional[Callable] = None,
	**options
	):
	if proceed is None:
	def proceed(response):
	return None
	while True:
	if isinstance(prompt, str):
	prompt = Prompt(
	prompt, model=self, system=system, options=self.Options(**options)
	)
	response = self.response(
	prompt,
	stream=stream,
	)
	yield response
	next_prompt = proceed(response)
	if not next_prompt:
	break
	prompt = next_prompt

[simonw]

I'm going to handle chaining as a separate, post-0.5 issue.

[simonw]

To implement continue mode I'm going to need to persist these things to the database.

Which means I need a whole new schema, since I'm switching to using ULID IDs as part of this work.

[simonw]

I can close this issue once I've documented this on https://llm.datasette.io/en/latest/python-api.html