Skip to content

Commit

Permalink
fead: add OpenChat3.6 8B
Browse files Browse the repository at this point in the history
  • Loading branch information
@umbertogriffo
umbertogriffo committed Jun 28, 2024
1 parent 27fd6b8 commit 3c171c7
Show file tree
Hide file tree
Showing 2 changed files with 79 additions and 12 deletions.
23 changes: 12 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,17 +133,18 @@ format.

### Supported Models

| 🤖 Model | Supported | Model Size | Notes and link to the model |
|------------------------------------------------|-----------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `llama-3` Meta Llama 3 Instruct | ✅ | 8B | Less accurate than OpenChat - [link](https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF) |
| `openchat` **Recommended** - OpenChat 3.5 0106 | ✅ | 7B | [link](https://huggingface.co/TheBloke/openchat-3.5-0106-GGUF) |
| `starling` Starling Beta | ✅ | 7B | Is trained from `Openchat-3.5-0106`. It's recommended if you prefer more verbosity over OpenChat - [link](https://huggingface.co/bartowski/Starling-LM-7B-beta-GGUF) |
| `neural-beagle` NeuralBeagle14 | ✅ | 7B | [link](https://huggingface.co/TheBloke/NeuralBeagle14-7B-GGUF) |
| `dolphin` Dolphin 2.6 Mistral DPO Laser | ✅ | 7B | [link](https://huggingface.co/TheBloke/dolphin-2.6-mistral-7B-dpo-laser-GGUF) |
| `zephyr` Zephyr Beta | ✅ | 7B | [link](https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) |
| `mistral` Mistral OpenOrca | ✅ | 7B | [link](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF) |
| `phi-3` Phi-3 Mini 4K Instruct | ✅ | 3.8B | [link](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf) |
| `stablelm-zephyr` StableLM Zephyr OpenOrca | ✅ | 3B | [link](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF) |
| 🤖 Model | Supported | Model Size | Notes and link to the model |
|--------------------------------------------------------|-----------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `llama-3` Meta Llama 3 Instruct | ✅ | 8B | Less accurate than OpenChat - [link](https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF) |
| `openchat-3.6` **Recommended** - OpenChat 3.6 20240522 | ✅ | 8B | [link](https://huggingface.co/bartowski/openchat-3.6-8b-20240522-GGUF) |
| `openchat-3.5` - OpenChat 3.5 0106 | ✅ | 7B | [link](https://huggingface.co/TheBloke/openchat-3.5-0106-GGUF) |
| `starling` Starling Beta | ✅ | 7B | Is trained from `Openchat-3.5-0106`. It's recommended if you prefer more verbosity over OpenChat - [link](https://huggingface.co/bartowski/Starling-LM-7B-beta-GGUF) |
| `neural-beagle` NeuralBeagle14 | ✅ | 7B | [link](https://huggingface.co/TheBloke/NeuralBeagle14-7B-GGUF) |
| `dolphin` Dolphin 2.6 Mistral DPO Laser | ✅ | 7B | [link](https://huggingface.co/TheBloke/dolphin-2.6-mistral-7B-dpo-laser-GGUF) |
| `zephyr` Zephyr Beta | ✅ | 7B | [link](https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) |
| `mistral` Mistral OpenOrca | ✅ | 7B | [link](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF) |
| `phi-3` Phi-3 Mini 4K Instruct | ✅ | 3.8B | [link](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf) |
| `stablelm-zephyr` StableLM Zephyr OpenOrca | ✅ | 3B | [link](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF) |

## Example Data

Expand Down
68 changes: 67 additions & 1 deletion chatbot/bot/model/settings/openchat.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from bot.model.model import Model


class OpenChatSettings(Model):
class OpenChat35Settings(Model):
url = "https://huggingface.co/TheBloke/openchat-3.5-0106-GGUF/resolve/main/openchat-3.5-0106.Q4_K_M.gguf"
file_name = "openchat-3.5-0106.Q4_K_M.gguf"
clients = [LlmClientType.LAMA_CPP]
Expand Down Expand Up @@ -63,3 +63,69 @@ class OpenChatSettings(Model):
Please also don't reformulate the follow up question, and write just a concise answer.
<|end_of_turn|>GPT4 Correct Assistant:
"""


class OpenChat36Settings(Model):
url = "https://huggingface.co/bartowski/openchat-3.6-8b-20240522-GGUF/resolve/main/openchat-3.6-8b-20240522-Q4_K_M.gguf"
file_name = "openchat-3.6-8b-20240522-Q4_K_M.gguf"
clients = [LlmClientType.LAMA_CPP]
config = {
"n_ctx": 4096, # The max sequence length to use - note that longer sequence lengths require much more resources
"n_threads": 8, # The number of CPU threads to use, tailor to your system and the resulting performance
"n_gpu_layers": 50, # The number of layers to offload to GPU, if you have GPU acceleration available
}
config_answer = {"temperature": 0.7, "stop": []}
system_template = (
"<|start_header_id|>system<|end_header_id|>You are a helpful, respectful and "
"honest assistant. <|eot_id|><|start_header_id|>GPT4 Correct User<|end_header_id|>"
)
qa_prompt_template = """{system}\n
Answer the question below:
{question}<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n
"""
ctx_prompt_template = """{system}\n
Context information is below.
---------------------
{context}
---------------------
Given the context information and not prior knowledge, answer the question below:
{question}<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n
"""
refined_ctx_prompt_template = """{system}\n
The original query is as follows: {question}
We have provided an existing answer: {existing_answer}
We have the opportunity to refine the existing answer
(only if needed) with some more context below.
---------------------
{context}
---------------------
Given the new context, refine the original answer to better answer the query.
If the context isn't useful, return the original answer.
Refined Answer:<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n
"""
refined_question_conversation_awareness_prompt_template = """{system}\n
Chat History:
---------------------
{chat_history}
---------------------
Follow Up Question: {question}
Given the above conversation and a follow up question, rephrase the follow up question to be a standalone question.
Standalone question:<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n
"""

refined_answer_conversation_awareness_prompt_template = """
You are engaging in a conversation with a human participant who is unaware that they might be
interacting with a machine. \n
Your goal is to respond in a way that convincingly simulates human-like intelligence and behavior. \n
The conversation should be natural, coherent, and contextually relevant. \n
Chat History:
---------------------
{chat_history}
---------------------
Follow Up Question: {question}\n
Given the context provided in the Chat History and the follow up question, please answer the follow up question above.
If the follow up question isn't correlated to the context provided in the Chat History, please just answer the follow up
question, ignoring the context provided in the Chat History.
Please also don't reformulate the follow up question, and write just a concise answer.
<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n
"""

0 comments on commit 3c171c7

Please sign in to comment.