Update tokenizer apply_chat_template functionality

[xenova]

This PR adds functionality to support the new C4AI Command-R tokenizer.

Example usage:

Tools

import { AutoTokenizer } from "@xenova/transformers";

const tokenizer = await AutoTokenizer.from_pretrained("Xenova/c4ai-command-r-v01-tokenizer")

// define conversation input:
const conversation = [
  { role: "user", content: "Whats the biggest penguin in the world?" }
]
// Define tools available for the model to use:
const tools = [
  {
    name: "internet_search",
    description: "Returns a list of relevant document snippets for a textual query retrieved from the internet",
    parameter_definitions: {
      query: {
        description: "Query to search the internet with",
        type: "str",
        required: true
      }
    }
  },
  {
    name: "directly_answer",
    description: "Calls a standard (un-augmented) AI chatbot to generate a response given the conversation history",
    parameter_definitions: {}
  }
]


// render the tool use prompt as a string:
const tool_use_prompt = tokenizer.apply_chat_template(
  conversation,
  {
    chat_template: "tool_use",
    tokenize: false,
    add_generation_prompt: true,
    tools,
  }
)
console.log(tool_use_prompt)

RAG

import { AutoTokenizer } from "@xenova/transformers";

const tokenizer = await AutoTokenizer.from_pretrained("Xenova/c4ai-command-r-v01-tokenizer")

// define conversation input:
const conversation = [
  { role: "user", content: "Whats the biggest penguin in the world?" }
]
// define documents to ground on:
const documents = [
  { title: "Tall penguins", text: "Emperor penguins are the tallest growing up to 122 cm in height." },
  { title: "Penguin habitats", text: "Emperor penguins only live in Antarctica." }
]

// render the RAG prompt as a string:
const grounded_generation_prompt = tokenizer.apply_chat_template(
  conversation,
  {
    chat_template: "rag",
    tokenize: false,
    add_generation_prompt: true,

    documents,
    citation_mode: "accurate", // or "fast"
  }
)
console.log(grounded_generation_prompt);

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.