Developing virtual characters for use with large language models can be a process that more closely resembles alchemy than engineering. Endless tinkering around with character descriptions, followed by unscientific "tests" where a few chat messages are exchanged with the bot, and at last a semi-educated guess on whether the change has been an improvement or not.
Chatbot Clinic is an extension for text-generation-webui that automates and formalizes that process. It brings science to the art of creating chatbots.
Each one can have its own context and generation parameters. This allows you to comparatively evaluate both character descriptions and parameter presets, or any combination thereof.
When you send a message to the chat, you receive a reply from each of the configured bots. Clicking on your preferred reply continues the chat with that reply, and records this as a vote for the bot that wrote it.
The replies are presented in random order and without identifying the bots, to avoid any potential source of bias and ensure that your vote is based solely on the content of the replies.
Numbers talk, bullshit walks! The statistics view gives you a breakdown of how many votes you gave to each bot's replies, so you can make an informed decision on which bot is "best", without having to rely on any hunches.
Installing Chatbot Clinic follows the standard process for installing text-generation-webui extensions, namely:
- Install and run text-generation-webui.
- Navigate to the Session tab.
- Paste the URL
https://github.com/p-e-w/chatbot_clinicinto the textbox labeled Install or update an extension and press Enter. - Restart text-generation-webui.
- In the Session tab, check the entry
chatbot_clinicunder Available extensions, then click Save UI defaults to settings.yaml, then click Apply flags/extensions and restart. - Make sure you have installed and loaded a language model.
- Navigate to the Chatbot Clinic tab to use the extension.
Copyright © 2023 Philipp Emanuel Weidmann (pew@worldwidemann.com)
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.
By contributing to this project, you agree to release your contributions under the same license.