Fix #7765: Fix exceed context length error

[rchan26]

Description

I was getting errors of exceeding my context window (see #7765) after a few messages and after some discussion, it seemed like the ChatMemoryBuffer is meant to guard against this and it does actually start to forget some history after a while. But a problem will occur if the system prompt (in the prefix-messages) is quite long and in addition to the chat history, we can actually get an error regarding exceeding the context window.

I should also add that the messages_to_prompts always expect that we have alternating user and assistant interactions and hence I wrote in a line so that when we are forgetting chat history, we delete pairs of messages so we still have these alternating messages. It's also possible to get errors if we forget chat history but we have the chat history starting with assistant messages in the chat history.

In this PR, I made some changes to the ChatMemoryBuffer.get() method by allowing you to pass in an initial_token_count argument which represents a kind of amount of context window to keep available and this gets added to each time we assess the token count of the chat history.

I have made changes to the context and simple chat engines where we first compute the number of tokens in the prefix-messages and pass that into the ChatMemoryBuffer.get() method and this ensures that we do not have a resulting all_messages which exceeds the length.

Happy to add some tests if feel necessary.

Fixes #7765 (this is currently closed, but I think it's still an open issue actually - see #7765 (comment))

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• I stared at the code and made sure it makes sense

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[Disiok]

Thanks for digging into this and finding the true cause of the bug! 🔥