Gemma/Gemma2_on_Groq.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nH85BOCo7YYk"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "cellView": "form",
        "id": "9tQNAByc7U9g"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7r2q0wS7bxf"
      },
      "source": [
        "# Gemma 2 running on Groq\n",
        "\n",
        "This cookbook shows how to call the Gemma 2 9B IT model hosted on [Groq Cloud](https://console.groq.com/). Groq is an AI accelerater company that provides [free Gemma 2 9B access](https://console.groq.com/docs/models) with blazing fast inference speed.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/Gemma2_on_Groq.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHrL4tqs7mYK"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "For this tutorial, you do not need a hardware accelerator since the model is hosted on Groq Cloud. A CPU instance will suffice.\n",
        "\n",
        "### Genearate a Groq API key\n",
        "Visit the [Groq Cloud console](https://console.groq.com/keys) and generate a key. Store the key as a Colab secret named `GROQ_API_KEY`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bF1csLohRrH6"
      },
      "source": [
        "# Calling Gemma 2 on Groq\n",
        "\n",
        "Performing a chat completion with Gemma 2 on Groq is very easy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "x3xUWUrFRpjK"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting groq\n",
            "  Downloading groq-0.9.0-py3-none-any.whl (103 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m103.5/103.5 kB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from groq) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from groq) (1.7.0)\n",
            "Collecting httpx<1,>=0.23.0 (from groq)\n",
            "  Downloading httpx-0.27.0-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from groq) (2.8.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from groq) (1.3.1)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from groq) (4.12.2)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->groq) (3.7)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->groq) (1.2.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->groq) (2024.6.2)\n",
            "Collecting httpcore==1.* (from httpx<1,>=0.23.0->groq)\n",
            "  Downloading httpcore-1.0.5-py3-none-any.whl (77 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->groq)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->groq) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->groq) (2.20.0)\n",
            "Installing collected packages: h11, httpcore, httpx, groq\n",
            "Successfully installed groq-0.9.0 h11-0.14.0 httpcore-1.0.5 httpx-0.27.0\n",
            "A Large Language Model (LLM) is a type of artificial intelligence that excels at understanding and generating human-like text. \n",
            "\n",
            "Here's a breakdown:\n",
            "\n",
            "**What makes them \"Large\"?**\n",
            "\n",
            "* **Massive Datasets:** LLMs are trained on enormous amounts of text data, often encompassing books, articles, websites, and code. This vast exposure to language allows them to learn complex patterns and relationships within it.\n",
            "* **Numerous Parameters:** They have billions, even trillions, of parameters—think of these as adjustable knobs that control the model's behavior. The more parameters, the greater the model's capacity to learn and generate nuanced text.\n",
            "\n",
            "**What can LLMs do?**\n",
            "\n",
            "* **Text Generation:**\n",
            "\n",
            "Write stories, poems, articles, summaries, and even code.\n",
            "* **Language Translation:**\n",
            "\n",
            "Convert text from one language to another with remarkable accuracy.\n",
            "* **Question Answering:**\n",
            "\n",
            "Provide informative answers to a wide range of questions based on the knowledge gained during training.\n",
            "* **Dialogue Systems:**\n",
            "\n",
            "Engage in natural-sounding conversations and provide human-like responses.\n",
            "* **Text Summarization:**\n",
            "\n",
            "Condense large amounts of text into concise summaries.\n",
            "\n",
            "**Examples of LLMs:**\n",
            "\n",
            "* **GPT-3 (Generative Pre-trained Transformer 3)** by OpenAI\n",
            "* **LaMDA (Language Model for Dialogue Applications)** by Google\n",
            "* **BERT (Bidirectional Encoder Representations from Transformers)** by Google\n",
            "* **BLOOM (BigScience Large Open-science Open-access Multilingual Language Model)** by BigScience\n",
            "\n",
            "**Important Considerations:**\n",
            "\n",
            "* **Bias:** LLMs can inherit biases present in the training data, leading to potentially unfair or prejudiced outputs.\n",
            "* **Factual Errors:** While impressive, LLMs can sometimes generate incorrect information. It's crucial to fact-check their outputs.\n",
            "* **Ethical Implications:** The power of LLMs raises ethical concerns regarding misuse, such as generating fake news or impersonating individuals.\n",
            "\n",
            "\n",
            "Let me know if you have any other questions about LLMs!\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!pip install groq\n",
        "\n",
        "from groq import Groq\n",
        "from google.colab import userdata\n",
        "\n",
        "client = Groq(\n",
        "    api_key=userdata.get(\"GROQ_API_KEY\"),\n",
        ")\n",
        "\n",
        "chat_completion = client.chat.completions.create(\n",
        "    messages=[\n",
        "        {\n",
        "            \"role\": \"user\",\n",
        "            \"content\": \"What is Large Language Model?\",\n",
        "        }\n",
        "    ],\n",
        "    model=\"gemma2-9b-it\",\n",
        ")\n",
        "\n",
        "print(chat_completion.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rvpqC645TXLN"
      },
      "source": [
        "Groq also offers OpenAI API compatibility."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "puX0h_DmSZfO"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting OpenAI\n",
            "  Downloading openai-1.35.10-py3-none-any.whl (328 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m328.3/328.3 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from OpenAI) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from OpenAI) (1.7.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from OpenAI) (0.27.0)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from OpenAI) (2.8.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from OpenAI) (1.3.1)\n",
            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from OpenAI) (4.66.4)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from OpenAI) (4.12.2)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->OpenAI) (3.7)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->OpenAI) (1.2.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->OpenAI) (2024.6.2)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->OpenAI) (1.0.5)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->OpenAI) (0.14.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->OpenAI) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->OpenAI) (2.20.0)\n",
            "Installing collected packages: OpenAI\n",
            "Successfully installed OpenAI-1.35.10\n",
            "Large language models (LLMs) are incredibly versatile tools with a wide range of applications. Here are some examples of what you can do with an LLM like me:\n",
            "\n",
            "**Communication and Language:**\n",
            "\n",
            "* **Text generation:** Write stories, poems, articles, emails, letters, and more.\n",
            "\n",
            "* **Translation:** Translate text from one language to another.\n",
            "* **Summarization:** Condense large amounts of text into shorter summaries.\n",
            "* **Conversation:** Engage in natural-sounding conversations with humans.\n",
            "* **Dialogue creation:** Write scripts for movies, plays, or video games.\n",
            "\n",
            "**Creativity and Entertainment:**\n",
            "\n",
            "* **Storytelling:** Generate creative narratives and ideas for stories.\n",
            "* **Poetry writing:** Compose poems in various styles and forms.\n",
            "* **Songwriting:** Assist in writing lyrics or composing melodies.\n",
            "* **Scriptwriting:** Create dialogue and plot for videos, games, or plays.\n",
            "\n",
            "**Education and Research:**\n",
            "\n",
            "* **Question answering:** Provide answers to questions based on given information.\n",
            "* **Tutoring:** Offer explanations and assistance with learning concepts.\n",
            "* **Research assistance:** Summarize research papers, identify relevant information, and generate hypotheses.\n",
            "* **Code generation:** Write code in multiple programming languages.\n",
            "\n",
            "**Productivity and Business:**\n",
            "\n",
            "* **Email drafting:** Assist in composing professional emails.\n",
            "* **Meeting summarization:** Create concise summaries of meeting discussions.\n",
            "* **Customer service:** Provide automated responses to common customer inquiries.\n",
            "* **Content creation:** Generate marketing materials, social media posts, and other content.\n",
            "\n",
            "**Other Applications:**\n",
            "\n",
            "* **Personalization:** Tailor experiences and recommendations based on user preferences.\n",
            "* **Accessibility:** Provide text-to-speech and speech-to-text capabilities.\n",
            "* **Data analysis:** Identify patterns and insights in large datasets.\n",
            "\n",
            "It's important to remember that LLMs are still under development and have limitations. They may sometimes generate incorrect or biased information, so it's crucial to critically evaluate their outputs.\n",
            "\n",
            "Overall, LLMs offer a powerful and versatile platform for a wide range of applications, transforming the way we interact with information and technology.\n",
            "\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!pip install OpenAI\n",
        "from openai import OpenAI\n",
        "\n",
        "groq = OpenAI(\n",
        "    api_key=userdata.get(\"GROQ_API_KEY\"), base_url=\"https://api.groq.com/openai/v1\"\n",
        ")\n",
        "\n",
        "response = groq.chat.completions.create(\n",
        "    model=\"gemma2-9b-it\",\n",
        "    messages=[\n",
        "        {\n",
        "            \"role\": \"user\",\n",
        "            \"content\": \"What can you do with a Large Language Model?\",\n",
        "        }\n",
        "    ],\n",
        ")\n",
        "print(response.choices[0].message.content)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Gemma2_on_Groq.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}