tokenizers: support apply_tool_use_template

[gary149]

Feature request

Support apply_tool_use_template https://github.com/huggingface/transformers/blob/76a33a10923ccc1074917f6b6a1e719e626b7dc9/src/transformers/models/cohere/tokenization_cohere_fast.py#L420

Motivation

This could be useful to support https://huggingface.co/CohereForAI/c4ai-command-r-v01

[xenova]

Sure I can add that! The functionality is already supported actually with the apply_chat_template function (just with different arguments), see #647. For example:

import { AutoTokenizer } from "@xenova/transformers";

const tokenizer = await AutoTokenizer.from_pretrained("Xenova/c4ai-command-r-v01-tokenizer")

// define conversation input:
const conversation = [
  { role: "user", content: "Whats the biggest penguin in the world?" }
]
// Define tools available for the model to use:
const tools = [
  {
    name: "internet_search",
    description: "Returns a list of relevant document snippets for a textual query retrieved from the internet",
    parameter_definitions: {
      query: {
        description: "Query to search the internet with",
        type: "str",
        required: true
      }
    }
  },
  {
    name: "directly_answer",
    description: "Calls a standard (un-augmented) AI chatbot to generate a response given the conversation history",
    parameter_definitions: {}
  }
]


// render the tool use prompt as a string:
const tool_use_prompt = tokenizer.apply_chat_template(
  conversation,
  {
    chat_template: "tool_use",
    tokenize: false,
    add_generation_prompt: true,
    tools,
  }
)
console.log(tool_use_prompt)