ws_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e30f831",
   "metadata": {},
   "source": [
    "# VSB,FEI - Generative AI Workshop\n",
    "\n",
    "The aim of the workshop is to get an overview of possibilities in the generative artificial intelligence (GenAI) domain.\n",
    "\n",
    "* We will use [Python](https://www.python.org/) and [Huggingface](https://huggingface.co/).\n",
    "\n",
    "![meme01](https://github.com/rasvob/PopAI-FEI-Workshop/blob/main/images/llm_meme_02.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b474f9",
   "metadata": {},
   "source": [
    "# There are many LLM-based online solutions available nowadays\n",
    "* We will use the one from the Hugging Face library and self-host the model on our server\n",
    "* 💡 There are many different models, one of the most popular is [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)\n",
    "    * The models are quite comparable, there are no huge differences usually\n",
    "    * It is easy to switch among different models as the HuggingFace wraps the models with (more or less) unified API\n",
    "        * The Mixtral-8x7B is now the most *hyped model* as it outperforms Llama 2 70B on [most benchmarks](https://mistral.ai/news/mixtral-of-experts/)\n",
    "\n",
    "### Pros:\n",
    "* **Greater Control:** Users would have more control over the deployment and utilization of the language model, allowing for customization based on specific needs.\n",
    "* **Privacy and Security:** Users might have increased confidence in the security and privacy of their data since the language model would be hosted on their own servers.\n",
    "* **Reduced Latency:** Local hosting could lead to lower latency, as requests and responses wouldn't need to travel over the internet.\n",
    "\n",
    "### Cons:\n",
    "* **Resource Intensiveness:** Large language models can be computationally expensive. Self-hosting might require substantial computational resources, including powerful servers and significant amounts of memory.\n",
    "* **Scalability Issues:** Managing scalability for widespread use could be challenging for individual users or smaller organizations. OpenAI's infrastructure is designed to handle large-scale demands.\n",
    "* **Maintenance and Updates:** Regular updates and maintenance are crucial for the performance and security of language models. Self-hosting would necessitate users to actively manage updates and patches."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75b8bc3d",
   "metadata": {},
   "source": [
    "## We need to install the following packages and load the model into GPU\n",
    "* 💡 Beware that this requires a lot of memory, so you might need to use a machine with a good GPUs\n",
    "    * We tested it on 4x RTX 3090 24GB GPUs\n",
    "    * It still required quantization to fit into 4x 24GB GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "971a0636",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install transformers accelerate bitsandbytes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e146ce3d-5820-4eab-a475-5a6aa3cd75ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78c8a623",
   "metadata": {},
   "source": [
    "## Each model has its own tokenizer and configuration\n",
    "\n",
    "* A tokenizer is needed to convert raw text into tokens, which are the basic units of input for a language model. \n",
    "* Tokenization is an important step in natural language processing tasks because it breaks down text into smaller, meaningful units that can be processed by the model. \n",
    "* The tokenizer also handles special tokens, such as padding tokens, start-of-sentence tokens, and end-of-sentence tokens, which are necessary for proper model input formatting. \n",
    "* Different models may have different tokenization methods and vocabulary, so each model typically has its own tokenizer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f519b795-79f4-48cd-968a-8b622995a967",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a16dc5",
   "metadata": {},
   "source": [
    "The `bnb_config` is a configuration object that is used to specify certain settings for the `BitsAndBytes` module.\n",
    "\n",
    "* `load_in_4bit=True`: This parameter indicates that the model weights will be loaded in 4-bit format, which means that each weight value will be represented using only 4 bits of memory instead of the usual 32 bits. \n",
    "    * This helps to reduce the memory footprint of the model.\n",
    "\n",
    "* `bnb_4bit_use_double_quant=True`: This parameter enables the use of double quantization for 4-bit weights. Double quantization is a technique that further compresses the 4-bit weights by quantizing them again using a different quantization method. \n",
    "    * This helps to reduce the memory usage even further.\n",
    "\n",
    "* `bnb_4bit_quant_type=\"nf4\"`: This parameter specifies the type of quantization to be used for the 4-bit weights. In this case, \"nf4\" stands for \"non-uniform 4-bit quantization\", which means that the quantization levels are not evenly spaced. \n",
    "    * This allows for more efficient representation of the weights.\n",
    "\n",
    "* `bnb_4bit_compute_dtype=torch.bfloat16`: This parameter specifies the data type to be used for computations involving the 4-bit weights. \n",
    "    * In this case, `torch.bfloat16` is used, which is a 16-bit floating-point format that provides a good balance between precision and memory usage.\n",
    "\n",
    "### 💡 By using this configuration, the model can achieve significant memory savings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "37f869e2-2402-4455-980f-be7c5d9c47e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "bnb_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_use_double_quant=True,\n",
    "    bnb_4bit_quant_type=\"nf4\",\n",
    "    bnb_4bit_compute_dtype=torch.bfloat16\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e795504",
   "metadata": {},
   "source": [
    "## Now we can load the model with the BitsAndBytesConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f62d42ad-b32a-4d24-b560-3b38fa1873b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30ff49274712416d95439b5a748029e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/19 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map=\"auto\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990e4991",
   "metadata": {},
   "source": [
    "## LLM usually need specific input prompt formatting\n",
    "This Python function, `format_prompt`, is used to format a conversation history into a specific structure. It takes two parameters:\n",
    "\n",
    "1. `message`: This is the latest user message that needs to be added to the conversation history.\n",
    "\n",
    "2. `history`: This is a list of tuples, where each tuple represents a previous interaction in the conversation. \n",
    "\n",
    "* 💡 Each tuple contains two elements: the user's message and the bot's response.\n",
    "\n",
    "The function starts by initializing a string `prompt` with the value **\"\\<s\\>\"**, which might be used to denote the start of the context.\n",
    "* The second **\"\\<\\\\s\\>\"** tag denotes an end to the conversation context (i.e. history)\n",
    "\n",
    "Then, for each user message and bot response in the history, it appends to `prompt` a formatted string that includes the user message enclosed within **[INST]** and **[/INST]** tags\n",
    "\n",
    "After going through all the history, it appends the latest user message (the `message` parameter) to `prompt`, again enclosed within **[INST]** and **[/INST]** tags.\n",
    "\n",
    "Finally, the function returns the fully formatted `prompt` string. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bb43c787-1e97-4c11-80bb-877801d2faeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_prompt(message, history):\n",
    "  prompt = \"<s>\"\n",
    "  for user_prompt, bot_response in history:\n",
    "    prompt += f\"[INST] {user_prompt} [/INST]\"\n",
    "    prompt += f\" {bot_response}</s> \"\n",
    "  prompt += f\"[INST] {message} [/INST]\"\n",
    "  return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89d2a0e8",
   "metadata": {},
   "source": [
    "## Init the history list, i.e. the conversation context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "57fa96ed-f43f-4864-b8e9-21a60fe06010",
   "metadata": {},
   "outputs": [],
   "source": [
    "history = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f352aeb7",
   "metadata": {},
   "source": [
    "Python function, `chat`, is used to generate a response from a chat model to a given user message. It takes two parameters:\n",
    "\n",
    "1. `message`: This is the user's message that the chat model needs to respond to.\n",
    "\n",
    "2. `max_new_tokens`: This is the maximum number of tokens that the model is allowed to generate for its response. The default value is 256.\n",
    "\n",
    "\n",
    "* In summary, this function takes a user's message, processes it, feeds it to a chat model, gets the model's response, and prints it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "07144a50-bed1-4a30-be19-ea8859bbe2fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def chat(message, max_new_tokens=256):\n",
    "    global history\n",
    "    formatted = format_prompt(message, history)\n",
    "    inputs = tokenizer(formatted, return_tensors=\"pt\").to('cuda')\n",
    "    outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=0.9, top_p=0.95, repetition_penalty=1.0, pad_token_id=tokenizer.eos_token_id, do_sample=True)\n",
    "    text_output = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
    "    history.append((message, text_output))\n",
    "    print(text_output) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886e2e9c",
   "metadata": {},
   "source": [
    "![meme02](https://github.com/rasvob/PopAI-FEI-Workshop/blob/main/images/llm_meme_01.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "dacea228-01a5-4e10-a804-d0a9794003dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"Can you birefly explain how the QuickSort algorithm works and provide a Python implementation?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61855cb",
   "metadata": {},
   "source": [
    "#### 💡 `%%time` is a Jupyter Notebook magic command to measure the time of execution of a cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9f3bcbc7-0503-4a54-a86d-efb751bd4320",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INST] Can you birefly explain how the QuickSort algorithm works and provide a Python implementation? [/INST] QuickSort is a divide-and-conquer algorithm used for sorting. It works by selecting a \"pivot\" element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The sub-arrays are then recursively sorted. This process continues until the base case is reached, which is when the array has only one or zero elements.\n",
      "\n",
      "Here is a Python implementation of QuickSort:\n",
      "\n",
      "```python\n",
      "def quicksort(arr):\n",
      "    if len(arr) <= 1:\n",
      "        return arr\n",
      "    pivot = arr[len(arr) // 2]\n",
      "    left = [x for x in arr if x < pivot]\n",
      "    middle = [x for x in arr if x == pivot]\n",
      "    right = [x for x in arr if x > pivot]\n",
      "    return quicksort(left) + middle + quicksort(right)\n",
      "\n",
      "# Example usage:\n",
      "arr = [3,6,8,10,1,2,1]\n",
      "print(quicksort(arr)) # Output: [1, 1, 2, 3, 6, 8, 10]\n",
      "```\n",
      "In the above example, the `quicksort` function takes an array as input and returns a new sorted array. It first checks if the array has one or zero elements and returns it as is if that's the case. If not, it selects a pivot element (in this case, the middle element) and creates three lists: one for elements less than the pivot, one for elements equal to the pivot, and one for elements greater than the pivot. These three lists are then recursively sorted and concatenated to produce the final sorted array.\n",
      "\n",
      "It's worth noting that there are many ways to select the pivot and partition the array. The approach used in this example is called the \"Lomuto partition scheme\" and is a simple and intuitive way to implement QuickSort. However, it has a worst-case time complexity of O(n^2) when the input array is already sorted or contains many duplicate elements. More advanced partition schemes, such as the \"Hoare partition scheme,\" can improve the worst-case time complexity to O(n log n).\n",
      "CPU times: user 47.8 s, sys: 96.6 ms, total: 47.9 s\n",
      "Wall time: 47.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "chat(prompt, max_new_tokens=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1962bcd6",
   "metadata": {},
   "source": [
    "## When the response is created, you can continue the conversation by running the cell below\n",
    "* You can just use `continue` prompt to continue the conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "00c727d2-8c3c-46d2-a18a-d918b427bf67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INST] Can you birefly explain how the QuickSort algorithm works and provide a Python implementation? [/INST] [INST] Can you birefly explain how the QuickSort algorithm works and provide a Python implementation? [/INST] QuickSort is a divide-and-conquer algorithm used for sorting. It works by selecting a \"pivot\" element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The sub-arrays are then recursively sorted. This process continues until the base case is reached, which is when the array has only one or zero elements.\n",
      "\n",
      "Here is a Python implementation of QuickSort:\n",
      "\n",
      "```python\n",
      "def quicksort(arr):\n",
      "    if len(arr) <= 1:\n",
      "        return arr\n",
      "    pivot = arr[len(arr) // 2]\n",
      "    left = [x for x in arr if x < pivot]\n",
      "    middle = [x for x in arr if x == pivot]\n",
      "    right = [x for x in arr if x > pivot]\n",
      "    return quicksort(left) + middle + quicksort(right)\n",
      "\n",
      "# Example usage:\n",
      "arr = [3,6,8,10,1,2,1]\n",
      "print(quicksort(arr)) # Output: [1, 1, 2, 3, 6, 8, 10]\n",
      "```\n",
      "In the above example, the `quicksort` function takes an array as input and returns a new sorted array. It first checks if the array has one or zero elements and returns it as is if that's the case. If not, it selects a pivot element (in this case, the middle element) and creates three lists: one for elements less than the pivot, one for elements equal to the pivot, and one for elements greater than the pivot. These three lists are then recursively sorted and concatenated to produce the final sorted array.\n",
      "\n",
      "It's worth noting that there are many ways to select the pivot and partition the array. The approach used in this example is called the \"Lomuto partition scheme\" and is a simple and intuitive way to implement QuickSort. However, it has a worst-case time complexity of O(n^2) when the input array is already sorted or contains many duplicate elements. More advanced partition schemes, such as the \"Hoare partition scheme,\" can improve the worst-case time complexity to O(n log n).  [INST] continue [/INST] Sure! As I mentioned earlier, there are many ways to select the pivot and partition the array in QuickSort. The Lomuto partition scheme, which is the simplest and most intuitive way, is easy to understand and implement but has a worst-case time complexity of O(n^2) in certain scenarios.\n",
      "\n",
      "The Hoare partition scheme, on the other hand, is a more complex but more efficient way to implement QuickSort. It was invented by Charles Antony Richard Hoare, the inventor of QuickSort itself. Here is a Python implementation of QuickSort using the Hoare partition scheme:\n",
      "\n",
      "```python\n",
      "def quicksort(arr):\n",
      "    if len(arr) <= 1:\n",
      "        return arr\n",
      "    pivot = arr[len(arr) // 2]\n",
      "    i, j = 0, len(arr) - 1\n",
      "    while i <= j:\n",
      "        while arr[i] < pivot:\n",
      "            i += 1\n",
      "        while arr[j] > pivot:\n",
      "            j -= 1\n",
      "        if i <= j:\n",
      "            arr[i], arr[j] = arr[j], arr[i]\n",
      "            i, j = i + 1, j - 1\n",
      "    return quicksort(arr[:i]) + [pivot] + quicksort(arr[j+1:])\n",
      "\n",
      "# Example usage:\n",
      "arr = [3,6,8,10,1,2,1]\n",
      "print(quicksort(arr)) # Output: [1, 1, 2, 3, 6, 8, 10]\n",
      "```\n",
      "In this implementation, the `quicksort` function takes an array as input and returns a new sorted array. It first checks if the array has one or zero elements and returns it as is if that's the case. If not, it selects a pivot element (in this case, the middle element) and initializes two pointers, `i` and `j`, to the beginning and end of the array, respectively.\n",
      "\n",
      "The function then enters a while loop, where it iterates until the `i` and `j` pointers meet or cross each other. In each iteration, the function increments the `i` pointer until it finds an element greater than or equal to the pivot, and decrements the `j` pointer\n"
     ]
    }
   ],
   "source": [
    "chat('continue', max_new_tokens=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1498b9e1",
   "metadata": {},
   "source": [
    "## 💡 Bonus: The output is in *Markdown* thus you can create a parser and render it in more convinient format\n",
    "\n",
    "#### Can you birefly explain how the QuickSort algorithm works and provide a Python implementation? \n",
    "\n",
    "QuickSort is a divide-and-conquer algorithm used for sorting. It works by selecting a \"pivot\" element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The sub-arrays are then recursively sorted. This process continues until the base case is reached, which is when the array has only one or zero elements.\n",
    "\n",
    "Here is a Python implementation of QuickSort:\n",
    "\n",
    "```python\n",
    "def quicksort(arr):\n",
    "    if len(arr) <= 1:\n",
    "        return arr\n",
    "    pivot = arr[len(arr) // 2]\n",
    "    left = [x for x in arr if x < pivot]\n",
    "    middle = [x for x in arr if x == pivot]\n",
    "    right = [x for x in arr if x > pivot]\n",
    "    return quicksort(left) + middle + quicksort(right)\n",
    "\n",
    "# Example usage:\n",
    "arr = [3,6,8,10,1,2,1]\n",
    "print(quicksort(arr)) # Output: [1, 1, 2, 3, 6, 8, 10]\n",
    "```\n",
    "In the above example, the `quicksort` function takes an array as input and returns a new sorted array. It first checks if the array has one or zero elements and returns it as is if that's the case. If not, it selects a pivot element (in this case, the middle element) and creates three lists: one for elements less than the pivot, one for elements equal to the pivot, and one for elements greater than the pivot. These three lists are then recursively sorted and concatenated to produce the final sorted array.\n",
    "\n",
    "It's worth noting that there are many ways to select the pivot and partition the array. The approach used in this example is called the \"Lomuto partition scheme\" and is a simple and intuitive way to implement QuickSort. However, it has a worst-case time complexity of O(n^2) when the input array is already sorted or contains many duplicate elements. More advanced partition schemes, such as the \"Hoare partition scheme,\" can improve the worst-case time complexity to O(n log n).  \n",
    "\n",
    "#### continue\n",
    "\n",
    "Sure! As I mentioned earlier, there are many ways to select the pivot and partition the array in QuickSort. The Lomuto partition scheme, which is the simplest and most intuitive way, is easy to understand and implement but has a worst-case time complexity of O(n^2) in certain scenarios.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d05a83",
   "metadata": {},
   "source": [
    "# Do you want to try other LLMs easily or even compare them? \n",
    "## Check out the https://chat.lmsys.org/ tool!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e1d204f",
   "metadata": {},
   "source": [
    "![meme03](https://github.com/rasvob/PopAI-FEI-Workshop/blob/main/images/thats_all.jpg?raw=true)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}