notebooks/art_of_prompt_design/prompt_boundaries_and_token_healing.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`The Art of Prompt Design`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt Boundaries and Token Healing\n",
    "\n",
    "This (written jointly with <a href=\"https://medium.com/@marcotcr\">Marco Tulio Ribeiro</a>) is part 2 of a series on <b>the art of prompt design</b> (part 1 <a href=\"https://medium.com/towards-data-science/the-art-of-prompt-design-use-clear-syntax-4fc846c1ebd5\">here</a>), where we talk about controlling large language models (LLMs) with <a href=\"https://github.com/microsoft/guidance\">`guidance`</a>.\n",
    "\n",
    "In this post, we'll discuss how the greedy tokenization methods used by language models can introduce unintended token splits into your prompts, leading to puzzling generations.\n",
    "\n",
    "Language models are not trained on raw text, but rather on tokens, which are chunks of text that often occur together, similar to words. This impacts how language models 'see' text, including prompts (since prompts are just sets of tokens). GPT-style models utilize tokenization methods like [Byte Pair Encoding](https://en.wikipedia.org/wiki/Byte_pair_encoding) (BPE), which map all input bytes to token ids in an optimized/greedy manner. This is fine for training, but it can lead to subtle issues during inference, as shown in the example below.\n",
    "\n",
    "<!-- TODO\n",
    "Standard greedy token mapping works well during training, but it can lead to subtle issues during prompting and inference. These issues arise because the greedy token boundaries often don't line up with the end of the prompt, especially when considering the generated tokens that will come next. While the end of a prompt will always align with a token boundary in practice, as the prompt is tokenized before being extended by the model, there may be instances where the first characters of the completion are part of a longer token that would span the prompt boundary. In such cases, the longer token cannot be used even though the model would expect it based on the training data.\n",
    "\n",
    "The inability to use tokens that span prompt boundaries can lead to subtle yet important biases in the model's output. -->\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example of a prompt boundary problem\n",
    "Consider the following example, where we are trying to generate an HTTP URL string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f8122cee6644f8580553af690800002",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'The link is <a href=\"http: //www.google.com/search?q'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import transformers\n",
    "\n",
    "# we use StableLM as an example, but these issues impact all models to varying degrees\n",
    "generator = transformers.pipeline('text-generation', model='stabilityai/stablelm-base-alpha-3b')\n",
    "\n",
    "def raw_gen(prompt, temp=0):\n",
    "    kwargs = {}\n",
    "    if temp > 0:\n",
    "        kwargs[\"temperature\"] = temp\n",
    "        kwargs[\"do_sample\"] = True\n",
    "    return generator(prompt, max_new_tokens=10, pad_token_id=0, **kwargs)[0][\"generated_text\"]\n",
    "raw_gen('The link is <a href=\"http:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f8122cee6644f8580553af690800002",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'The link is <a href=\"http: //www.google.com/search?q'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import transformers\n",
    "\n",
    "# we use StableLM as an example, but these issues impact all models to varying degrees\n",
    "generator = transformers.pipeline('text-generation', model='stabilityai/stablelm-base-alpha-3b')\n",
    "\n",
    "def raw_gen(prompt):\n",
    "    return generator(prompt, max_new_tokens=10, pad_token_id=0)[0][\"generated_text\"]\n",
    "raw_gen('The link is <a href=\"http:')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the output generated by the LLM does not complete the url with the obvious next characters (two forward slashes). It instead creates an invalid URL string with a space in the middle. This is surprising, because the `//` completion is extremely obvious after `http:`. To understand why this happens, let's change our prompt boundary so that our prompt does not include the colon character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The link is <a href=\"http://www.youtube.com/v/s'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_gen('The link is <a href=\"http')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the language model generates a valid url string like we expect. To understand why the  `:` matters, we need to look at the tokenized representation of the prompts. Below is the tokenization of the prompt that ends in a colon (the prompt without the colon has the same tokenization, except for the last token):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len = 9\n",
      "510\t`The`\n",
      "3048\t` link`\n",
      "310\t` is`\n",
      "654\t` <`\n",
      "66\t`a`\n",
      "3860\t` href`\n",
      "568\t`=\"`\n",
      "2413\t`http`\n",
      "27\t`:`\n"
     ]
    }
   ],
   "source": [
    "def print_tokens(tokens):\n",
    "    print(\"len = \" + str(len(tokens)))\n",
    "    for i in tokens:\n",
    "        print(str(i) + \"\\t`\" + generator.tokenizer.decode([i]) + \"`\")\n",
    "\n",
    "print_tokens(generator.tokenizer.encode('The link is <a href=\"http:'))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now note what the tokenization of a valid URL looks like, paying careful attention to token `1358`, right after `http`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len = 18\n",
      "510\t`The`\n",
      "3048\t` link`\n",
      "310\t` is`\n",
      "654\t` <`\n",
      "66\t`a`\n",
      "3860\t` href`\n",
      "568\t`=\"`\n",
      "2413\t`http`\n",
      "1358\t`://`\n",
      "2700\t`www`\n",
      "15\t`.`\n",
      "9906\t`google`\n",
      "15\t`.`\n",
      "681\t`com`\n",
      "16\t`/`\n",
      "8716\t`search`\n",
      "32\t`?`\n",
      "82\t`q`\n"
     ]
    }
   ],
   "source": [
    "print_tokens(generator.tokenizer.encode('The link is <a href=\"http://www.google.com/search?q'))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This particular LLM uses a greedy/optimized tokenization method, almost always preferring the longest possible token, i.e. `://` will be preferred over `:` in full text (e.g. in training).\n",
    "\n",
    "While URLs in training are encoded with token 1358 (`://`), our prompt makes the LLM see token `27` (`:`) instead, which throws off completion by artificially splitting `://`.\n",
    "In fact, the model can be pretty sure that seeing token `27` (`:`) means what comes next is very unlikely to be anything that could have been encoded together with the colon using a \"longer token\" like `://`, since in the model's training data those characters would have been encoded together with the colon (an exception to this that we will discuss later is <a href=\"https://arxiv.org/abs/1804.10959\">subword regularization</a> during training). The fact that seeing a token means both seeing the embedding of that token **and also** that whatever comes next wasn't compressed by the greedy tokenizer is easy to forget, but it is important in prompt boundaries.\n",
    "\n",
    "Let's search over the string representation of all the tokens in the model's vocabulary, to see which ones start with a colon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len = 34\n",
      "27\t`:`\n",
      "1358\t`://`\n",
      "1450\t`::`\n",
      "5136\t`:\"`\n",
      "6098\t`:**`\n",
      "8048\t`:\\`\n",
      "10477\t`:(`\n",
      "13522\t`:=`\n",
      "18031\t`:\"){`\n",
      "18459\t`:#`\n",
      "19282\t`:</`\n",
      "21382\t`:[`\n",
      "21610\t`:/`\n",
      "22314\t`:-`\n",
      "22426\t`:'`\n",
      "23338\t`:_`\n",
      "25731\t`:@\"`\n",
      "25942\t`:=\\`\n",
      "27506\t`:*`\n",
      "27976\t`:%`\n",
      "30337\t`:``\n",
      "34417\t`:]`\n",
      "35490\t`:$`\n",
      "37731\t`:)`\n",
      "41210\t`::::`\n",
      "41924\t`:{`\n",
      "42841\t`:--`\n",
      "43118\t`:.`\n",
      "44662\t`:&`\n",
      "46064\t`:\")`\n",
      "46186\t`:{\\`\n",
      "47279\t`:$$\\`\n",
      "48471\t`:**]{}`\n",
      "49777\t`:\",`\n"
     ]
    }
   ],
   "source": [
    "tokens = generator.tokenizer.convert_ids_to_tokens(range(generator.tokenizer.vocab_size))\n",
    "colon_tokens = [i for i,t in enumerate(tokens) if t.startswith(\":\")]\n",
    "print_tokens(colon_tokens)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there are **34** different tokens starting with a colon, and thus ending a prompt with a colon means the model will likely not generate completions with any of these 34 token strings. *This subtle and powerful bias can have all kinds of unintended consequences.* And this applies to **any** string that could be potentially extended to make a longer single token (not just `:`).  Even our \"fixed\" prompt ending with \"http\" has a built in bias as well, as it communicates to the model that what comes after \"http\" is likely not \"s\" (otherwise \"http\" would not have been encoded as a separate token):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len = 2\n",
      "2413\t`http`\n",
      "3614\t`https`\n"
     ]
    }
   ],
   "source": [
    "http_tokens = [i for i,t in enumerate(tokens) if t.startswith(\"http\")]\n",
    "print_tokens(http_tokens)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lest you think this is an arcane problem that only touches URLs, remember that most tokenizers treat tokens differently depending on whether they start with a space, punctuation, quotes, etc, and thus **ending a prompt with any of these can lead to wrong token boundaries**, and break things:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I read a book about ~~the~~ the history of the world and the'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Accidentally adding a space, will lead to weird generation\n",
    "raw_gen('I read a book about ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I read a book about the history of the New Orleans Mafia and the'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# No space, works as expected\n",
    "raw_gen('I read a book about')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example of this is the \"[\" character. Consider the following prompt and completion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'An example [\"like this\"] and another example [like this] are shown in FIG. 1.'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# guidance('''An example [\"like this\"] and another example [{{gen max_tokens=10 token_healing=False}}''', caching=False)()\n",
    "raw_gen('An example [\"like this\"] and another example [')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is the second string not quoted? Because by ending our prompt with the ' [' token, we are telling the model that it should not generate completions that match the following 27 longer tokens (one of which adds the quote character, `15640`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len = 27\n",
      "544\t` [`\n",
      "1008\t` [@`\n",
      "3921\t` [*`\n",
      "4299\t` [**`\n",
      "8168\t` []`\n",
      "8605\t` [[`\n",
      "14412\t` ['`\n",
      "15640\t` [\"`\n",
      "16731\t` [$`\n",
      "20629\t` [$\\`\n",
      "21810\t` [(`\n",
      "21938\t` […]`\n",
      "23734\t` [****,`\n",
      "24345\t` [],`\n",
      "24430\t` [\\`\n",
      "26991\t` [];`\n",
      "27075\t` [^`\n",
      "27501\t` []{`\n",
      "28591\t` [-`\n",
      "31789\t` [...]`\n",
      "33440\t` [{`\n",
      "42989\t` [_`\n",
      "43521\t` [<`\n",
      "44308\t` [``\n",
      "44965\t` [[*`\n",
      "49193\t` [#`\n",
      "49824\t` [(\\[`\n"
     ]
    }
   ],
   "source": [
    "space_bracket_tokens = [i for i,t in enumerate(tokens) if t .startswith(\"Ġ[\")] # note the Ġ is converted to a space by the tokenizer\n",
    "print_tokens(space_bracket_tokens)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Token boundary bias happens everywhere. *About 70% of the 10k most common tokens for the StableLM model used above are prefixes of longer possible tokens, and so cause token boundary bias when they are the last token in a prompt.* Keeping track of all these possible extension biases during prompt design is impractical so most people just ignore them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "69.49%\n"
     ]
    }
   ],
   "source": [
    "# count the number of tokens that have longer extensions\n",
    "count = 0\n",
    "for i in range(10000):\n",
    "    m = 0\n",
    "    for j in range(generator.tokenizer.vocab_size):\n",
    "        if tokens[j].startswith(tokens[i]):\n",
    "            m += 1\n",
    "        if m > 1:\n",
    "            break\n",
    "    # m = guidance.llm.prefix_matches(guidance.llm.decode([i]))\n",
    "    if m > 1:\n",
    "        count += 1\n",
    "print(str(100*count/10000)+\"%\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixing unintended bias with \"token healing\"\n",
    "\n",
    "What can we do to avoid these unintended biases? One option is to always end our prompts with tokens that cannot be extended into longer tokens (for example a role tag for chat-based models), but this is a severe limitation.  \n",
    "\n",
    "Instead, `guidance` has a feature called \"token healing\", which automatically backs up the generation process by one token before the end of the prompt, then constrains the first token generated to have a prefix that matches the last token in the prompt. In our URL example, this would mean removing the `:`, and forcing generation of the first token to have a `:` prefix.   \n",
    "Token healing allows users to express prompts however they wish, without worrying about token boundaries.\n",
    "\n",
    "For example, let's re-run some of the URL examples above with token healing turned on (it's on by default for Transformer models, so we remove `token_healing=False`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>The link is &lt;a href=&quot;http:<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>//</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>man</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>7</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>now</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>com</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>/</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ann</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ounce</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>/</span></pre>"
      ],
      "text/plain": [
       "<guidance.models.transformers._transformers.Transformers at 0x7faad5dbc310>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from guidance import models, gen\n",
    "\n",
    "# load StableLM from huggingface\n",
    "lm = models.Transformers(\"stabilityai/stablelm-base-alpha-3b\", device=0)\n",
    "\n",
    "# With token healing we generate valid URLs, even when the prompt ends with a colon:\n",
    "lm + 'The link is <a href=\"http:' + gen(max_tokens=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>The link is &lt;a href=&quot;http<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>://</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>download</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>mac</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>rom</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>edia</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>com</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>/</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>get</span></pre>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['The link is <a href=\"https://www.goal.com/en-',\n",
       " 'The link is <a href=\"http://man7now0uvers.com/',\n",
       " 'The link is <a href=\"http://889946.com/Vista-',\n",
       " 'The link is <a href=\"http://download28.yellowoya.com/mov',\n",
       " 'The link is <a href=\"https://github.com/oraoutdoor/',\n",
       " 'The link is <a href=\"https://chrome.google.com/webstore',\n",
       " 'The link is <a href=\"http://usat.org/album/19/',\n",
       " 'The link is <a href=\"http://manapama.org/store_video',\n",
       " 'The link is <a href=\"https://www.hrefng.com/template',\n",
       " 'The link is <a href=\"http://download.macromedia.com/get']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[str(lm + 'The link is <a href=\"http' + gen(max_tokens=10, temperature=1)) for i in range(10)]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we don't have to worry about extra spaces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>I read a book about <span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>a</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> little</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> girl</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> who</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> had</span></pre>"
      ],
      "text/plain": [
       "<guidance.models.transformers._transformers.Transformers at 0x7faad5e19fd0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# Accidentally adding a space will not impact generation\n",
    "lm + 'I read a book about ' + gen(max_tokens=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>I read a book about<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> a</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> little</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> girl</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> who</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> had</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> a</span></pre>"
      ],
      "text/plain": [
       "<guidance.models.transformers._transformers.Transformers at 0x7faadc78a150>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will generate the same text as above \n",
    "lm + 'I read a book about' + gen(max_tokens=6)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we now get quoted strings even when the prompt ends with a \" [\" token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>An example [&quot;like this&quot;] and another example [<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>&quot;</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>like</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> this</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>&quot;]</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>Hi</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>,</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> I</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>&#x27;m</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> trying</span></pre>"
      ],
      "text/plain": [
       "<guidance.models.transformers._transformers.Transformers at 0x7faad5e42f90>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm + 'An example [\"like this\"] and another example [' + gen(max_tokens=10)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What about subword regularization?\n",
    "\n",
    "If you are familiar with how language models are trained, you may be wondering how <a href=\"https://arxiv.org/abs/1804.10959\">subword regularization</a> fits into all this. Subword regularization is a technique where during training sub-optimial tokenizations are randomly introduced to increase the model's robustness to token boundary issues. This means that the model does not always see the best tokenization. Subword regularization is great at helping the model be more robust to token boundaries, but it does not remove the bias that the model has towards the standard optimized (near greedy) tokenization. This means that while depending on the amount of subword regularization during training models may exhibit more or less token boundaries bias, all models still have this bias. And as shown above it can still have a powerful and unexpected impact on the model output."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "When you write prompts, remember that greedy tokenization can have a significant impact on how language models interpret your prompts, particularly when the prompt ends with a token that could be extended into a longer token. This easy-to-miss source of bias can impact your results in surprising and unintended ways.\n",
    "\n",
    "To address to this, either end your prompt with a non-extendable token, or use something like `guidance`'s \"token healing\" feature so you can to express your prompts however you wish, without worrying about token boundary artifacts. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix: Did we just get unlucky with the link example?\n",
    "\n",
    "No, and random sampling can verify that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The link is <a href=\"http: //www.plawesomenet.com',\n",
       " 'The link is <a href=\"http:\\\\\\\\\\\\/\\\\/(a|iris|art.',\n",
       " 'The link is <a href=\"http:\\n```<a href=\"test.pdf\"',\n",
       " 'The link is <a href=\"http://www.ihg.com/hotels',\n",
       " 'The link is <a href=\"http:\\nTUTORIAL_REVIEW_PAGE']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with the colon we almost always get an invalid link\n",
    "[raw_gen('The link is <a href=\"http:', temp=1) for _ in range(5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The link is <a href=\"http://www.youtube.com/linksyep',\n",
       " 'The link is <a href=\"http://www.realceteam.com/',\n",
       " 'The link is <a href=\"http://a.k-k-2.html',\n",
       " 'The link is <a href=\"http://www.scotlanded.gov.',\n",
       " 'The link is <a href=\"http://info.infoabooks.com/']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# without the colon we always get a valid link\n",
    "[raw_gen('The link is <a href=\"http', temp=1) for _ in range(5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; opacity: 0.5; border: none; background: #cccccc;\">\n",
    "<div style=\"text-align: center; opacity: 0.5\">Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}