server: Add `"custom"` chat template that uses `input_prefix` and `input_suffix` #10425

MaggotHATE · 2024-11-20T10:36:13Z

I have read the contributing guidelines
Self-reported review complexity:
- Low
- Medium
- High

This PR adds a new simple chat template that makes use of input_prefix and input_suffix to format messages. This is enough to be able to work with custom templates on server and is a more inline solution as opposed to changing .js parts of server (but also a less flexible one).

Currently, input_prefix and input_suffix are not used in server at all, and main uses both in its own way. As such, we can reuse prefix and suffix for server only without extra steps. It would also make adding template customization into UI quite easy (will add a bit later in this PR).

What this currently does is:

Adds "custom" template definition into server, automatically selected if no template is specified, but input_prefix and input_suffix are defined.
Passes input_prefix and input_suffix in format_chat (utils.hpp used by server), where they are used to simply format messages from json payload.

It is worth mentioning that main doesn't always apply prefix/suffix correctly (for example, with "chatml" format), so maybe it is worth adding prefix/suffix manually into each existing prompt format. I can do that later in a separate PR. @ngxson @slaren

* llama, common, server * main handles prefix and suffix separately, only adjust for example display

* `enable_chat_template` is no used by `server`

…gotHATE/llama.cpp-greedy-rework into server-chat-templates-custom

ngxson

This is a quite simple feature to add, so I prefer having less code for it. Also it's better if we can make it more isolated. For example, we could only implement this logic in server.cpp instead of common.cpp

ngxson · 2024-11-20T11:07:53Z

src/llama.cpp

@@ -22107,6 +22119,8 @@ static int32_t llama_chat_apply_template_internal(
 int32_t llama_chat_apply_template(
                const struct llama_model * model,
                              const char * tmpl,
+                              const char * prefix,
+                              const char * suffix,


I don't think we need to change llama_chat_apply_template. This feature can be implemented outside of llama.h library

I suppose it's better to implement it in utils.hpp then? A separate formatting function that is then used in format_chat function? Sorry for questions, I'm not quite familiar with server's structure.

Yes you can add it in server/utils.hpp, ideally as a branch inside format_chat (or can split into its own function if needed)

Thank you, already did that. I will see if all checks are good and add it into UI.

Looks like it will have to be a list of chat templates, in the end - with a choice of "custom" as this new simple solution.

common/common.h

ngxson

Btw I think you may need to add kinda prompt_prefix too. A template like this may not be possible without using a "global" prefix:

You are a helpful assistant

### Question:...

### Response:...

ngxson · 2024-11-20T15:03:46Z

examples/server/utils.hpp

@@ -325,10 +326,16 @@ inline std::string format_chat(const struct llama_model * model, const std::stri
            throw std::runtime_error("Missing 'content' (ref: https://github.com/ggerganov/llama.cpp/issues/8367)");
        }

-        chat.push_back({role, content});
+        if (tmpl == "custom") {


Maybe we can do this too?

Suggested change

if (tmpl == "custom") {

bool is_custom = !prefix.empty() || !suffix.empty();

if (is_custom) {

ngxson · 2024-11-20T15:04:35Z

examples/server/utils.hpp

    }

-    const auto formatted_chat = common_chat_apply_template(model, tmpl, chat, true);
+    if (tmpl != "custom") formatted_chat = common_chat_apply_template(model, tmpl, chat, true);


Then:

Suggested change

if (tmpl != "custom") formatted_chat = common_chat_apply_template(model, tmpl, chat, true);

if (!is_custom) formatted_chat = common_chat_apply_template(model, tmpl, chat, true);

ngxson · 2024-11-20T15:05:34Z

examples/server/utils.hpp

@@ -300,8 +300,9 @@ static llama_tokens format_infill(
 }

 // Format given chat. If tmpl is empty, we take the template from model metadata


Would be nice if we can leave a comment on how and when prefix/suffix is used

ngxson · 2024-11-20T15:08:32Z

examples/server/server.cpp

@@ -3220,15 +3220,20 @@ int main(int argc, char ** argv) {

    // if a custom chat template is not supplied, we will use the one that comes with the model (if any)
    if (params.chat_template.empty()) {
-        if (!ctx_server.validate_model_chat_template()) {
+        if (!params.input_prefix.empty() || !params.input_suffix.empty()) {


I think here we can refactor a bit:

if both chat_template, suffix/prefix are set, we throw an error says "only template or suffix/prefix can be set, but not both"

if chat_template is set, we use it

if suffix/prefix are set, we use it and discard chat_template

if none is specified, we use model's default template

In either cases, it would be nice to output a test formatted chat here

MaggotHATE added 2 commits November 20, 2024 14:42

Initial "prefix+suffix" chat template

53e0215

* llama, common, server * main handles prefix and suffix separately, only adjust for example display

Merge branch 'ggerganov:master' into server-chat-templates-custom

84da80c

github-actions bot added examples server labels Nov 20, 2024

MaggotHATE added 4 commits November 20, 2024 15:40

Fix trailing whitespace, reverted enable_chat_template in arg

9b58edf

* `enable_chat_template` is no used by `server`

Merge branch 'server-chat-templates-custom' of https://github.com/Mag…

dbe531e

…gotHATE/llama.cpp-greedy-rework into server-chat-templates-custom

Fix for simple-chat

b3e343e

Fixed test-chat-template

fc05038

ngxson reviewed Nov 20, 2024

View reviewed changes

Reverted to a simple solution withing server only.

ec6212e

MaggotHATE changed the title ~~llama, common, server: Add "custom" chat template that uses input_prefix and input_suffix~~ server: Add "custom" chat template that uses input_prefix and input_suffix Nov 20, 2024

ngxson reviewed Nov 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: Add `"custom"` chat template that uses `input_prefix` and `input_suffix` #10425

server: Add `"custom"` chat template that uses `input_prefix` and `input_suffix` #10425

MaggotHATE commented Nov 20, 2024 •

edited

Loading

ngxson left a comment

ngxson Nov 20, 2024

MaggotHATE Nov 20, 2024

ngxson Nov 20, 2024

MaggotHATE Nov 20, 2024 •

edited

Loading

ngxson left a comment

ngxson Nov 20, 2024

ngxson Nov 20, 2024

ngxson Nov 20, 2024

ngxson Nov 20, 2024

	if (tmpl == "custom") {
	bool is_custom = !prefix.empty() \|\| !suffix.empty();
	if (is_custom) {

	if (tmpl != "custom") formatted_chat = common_chat_apply_template(model, tmpl, chat, true);
	if (!is_custom) formatted_chat = common_chat_apply_template(model, tmpl, chat, true);

		@@ -300,8 +300,9 @@ static llama_tokens format_infill(
		}

		// Format given chat. If tmpl is empty, we take the template from model metadata

server: Add "custom" chat template that uses input_prefix and input_suffix #10425

Are you sure you want to change the base?

server: Add "custom" chat template that uses input_prefix and input_suffix #10425

Conversation

MaggotHATE commented Nov 20, 2024 • edited Loading

ngxson left a comment

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

MaggotHATE Nov 20, 2024

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

MaggotHATE Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

ngxson left a comment

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

ngxson Nov 20, 2024

Choose a reason for hiding this comment

server: Add `"custom"` chat template that uses `input_prefix` and `input_suffix` #10425

server: Add `"custom"` chat template that uses `input_prefix` and `input_suffix` #10425

MaggotHATE commented Nov 20, 2024 •

edited

Loading

MaggotHATE Nov 20, 2024 •

edited

Loading