gemini/getting-started/intro_gemini_2_0_pro.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sqi5B7V_Rjim"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VyPmicX9RlZX"
      },
      "source": [
        "# Intro to Gemini 2.0 Pro\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.luolix.top%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgetting-started%2Fintro_gemini_2_0_pro.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_2_0_pro.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8MqT58L6Rm_q"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "| Author(s) |  [Eric Dong](https://github.com/gericdong), [Holt Skinner](https://github.com/holtskinner) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nVxnv1D5RoZw"
      },
      "source": [
        "## Overview\n",
        "\n",
        "**YouTube Video: Introduction to Gemini on Vertex AI**\n",
        "\n",
        "<a href=\"https://www.youtube.com/watch?v=YfiLUpNejpE&list=PLIivdWyY5sqJio2yeg1dlfILOUO2FoFRx\" target=\"_blank\">\n",
        "  <img src=\"https://img.youtube.com/vi/YfiLUpNejpE/maxresdefault.jpg\" alt=\"Introduction to Gemini on Vertex AI\" width=\"500\">\n",
        "</a>\n",
        "\n",
        "[Gemini 2.0 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/gemini-v2#2.0-pro) is Google's strongest model for coding and world knowledge and features a 2M long context window. Gemini 2.0 Pro is available as an experimental model in Vertex AI and is an upgrade path for Gemini 1.5 Pro users who want better quality, or who are particularly invested in long context and code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfFPCBL4Hq8x"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this tutorial, you will learn how to use the Gemini API in Vertex AI and the Google Gen AI SDK for Python with the Gemini 2.0 Pro model.\n",
        "\n",
        "You will complete the following tasks:\n",
        "\n",
        "- Generate text from text prompts\n",
        "  - Generate streaming text\n",
        "  - Start multi-turn chats\n",
        "  - Use asynchronous methods\n",
        "- Configure model parameters\n",
        "- Set system instructions\n",
        "- Use safety filters\n",
        "- Use controlled generation\n",
        "- Count tokens\n",
        "- Process multimodal (audio, code, documents, images, video) data\n",
        "- Use automatic and manual function calling\n",
        "- Code execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPiTOAHURvTM"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHRZUpfWSEpp"
      },
      "source": [
        "### Install Google Gen AI SDK for Python\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sG3_LKsWSD3A"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlMVjiAWSMNX"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "12fnq4V0SNV3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ve4YBlDqzyj9"
      },
      "source": [
        "### Connect to a generative AI API service\n",
        "\n",
        "Google Gen AI APIs and models including Gemini are available in the following two API services:\n",
        "\n",
        "- **[Google AI for Developers](https://ai.google.dev/gemini-api/docs)**: Experiment, prototype, and deploy small projects.\n",
        "- **[Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/overview)**: Build enterprise-ready projects on Google Cloud.\n",
        "\n",
        "The Google Gen AI SDK provides a unified interface to these two API services.\n",
        "\n",
        "This notebook shows how to use the Google Gen AI SDK with the Gemini API in Vertex AI."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdvJRUWRNGHE"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "qgdSpVmDbdQ9"
      },
      "outputs": [],
      "source": [
        "from IPython.display import HTML, Markdown, display\n",
        "from google import genai\n",
        "from google.genai.types import (\n",
        "    FunctionDeclaration,\n",
        "    GenerateContentConfig,\n",
        "    GoogleSearch,\n",
        "    Part,\n",
        "    Retrieval,\n",
        "    SafetySetting,\n",
        "    Tool,\n",
        "    ToolCodeExecution,\n",
        "    VertexAISearch,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LymmEN6GSTn-"
      },
      "source": [
        "### Set Google Cloud project information and create client\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "UCgUOv4nSWhc"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "zpIPG_YhSjaw"
      },
      "outputs": [],
      "source": [
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4yRkFg6BBu4"
      },
      "source": [
        "## Use the Gemini 2.0 Pro model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXHJi5B6P5vd"
      },
      "source": [
        "### Load the Gemini 2.0 Pro model\n",
        "\n",
        "Learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "-coEslfWPrxo"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-pro-exp-02-05\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37CH91ddY9kG"
      },
      "source": [
        "### Generate text from text prompts\n",
        "\n",
        "Use the `generate_content()` method to generate responses to your prompts.\n",
        "\n",
        "You can pass text to `generate_content()`, and use the `.text` property to get the text content of the response.\n",
        "\n",
        "By default, Gemini outputs formatted text using [Markdown](https://daringfireball.net/projects/markdown/) syntax."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xRJuHj0KZ8xz"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID, contents=\"What's the largest planet in our solar system?\"\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JkYQATRxAK1_"
      },
      "source": [
        "#### Example prompts\n",
        "\n",
        "- What are the biggest challenges facing the healthcare industry?\n",
        "- What are the latest developments in the automotive industry?\n",
        "- What are the biggest opportunities in retail industry?\n",
        "- (Try your own prompts!)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6lLIxqS6_-l8"
      },
      "source": [
        "### Generate content stream\n",
        "\n",
        "By default, the model returns a response after completing the entire generation process. You can also use the `generate_content_stream` method to stream the response as it is being generated, and the model will return chunks of the response as soon as they are generated."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZiwWBhXsAMnv"
      },
      "outputs": [],
      "source": [
        "for chunk in client.models.generate_content_stream(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"Tell me a story about a lonely robot who finds friendship in a most unexpected place.\",\n",
        "):\n",
        "    display(Markdown(chunk.text))\n",
        "    display(Markdown(\"---\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "29jFnHZZWXd7"
      },
      "source": [
        "### Start a multi-turn chat\n",
        "\n",
        "The Gemini API supports freeform multi-turn conversations across multiple turns with back-and-forth interactions.\n",
        "\n",
        "The context of the conversation is preserved between messages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "DbM12JaLWjiF"
      },
      "outputs": [],
      "source": [
        "chat = client.chats.create(model=MODEL_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JQem1halYDBW"
      },
      "outputs": [],
      "source": [
        "response = chat.send_message(\"Write a function that checks if a year is a leap year.\")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUJR4Pno-LGK"
      },
      "source": [
        "This follow-up prompt shows how the model responds based on the previous prompt:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Fn69TurZ9DB"
      },
      "outputs": [],
      "source": [
        "response = chat.send_message(\"Write a unit test of the generated function.\")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "arLJE4wOuhh6"
      },
      "source": [
        "### Send asynchronous requests\n",
        "\n",
        "`client.aio` exposes all analogous [async](https://docs.python.org/3/library/asyncio.html) methods that are available on `client`.\n",
        "\n",
        "For example, `client.aio.models.generate_content` is the async version of `client.models.generate_content`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gSReaLazs-dP"
      },
      "outputs": [],
      "source": [
        "response = await client.aio.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"Compose a song about the adventures of a time-traveling squirrel.\",\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hIJVEr0RQY8S"
      },
      "source": [
        "## Configure model parameters\n",
        "\n",
        "You can include parameter values in each call that you send to a model to control how the model generates a response. The model can generate different results for different parameter values. You can experiment with different model parameters to see how the results change.\n",
        "\n",
        "- Learn more about [experimenting with parameter values](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/adjust-parameter-values).\n",
        "\n",
        "- See a list of all [Gemini API parameters](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#parameters).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9NXP5N2Pmfo"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"Tell me how the internet works, but pretend I'm a puppy who only understands squeaky toys.\",\n",
        "    config=GenerateContentConfig(\n",
        "        temperature=0.4,\n",
        "        top_p=0.95,\n",
        "        top_k=20,\n",
        "        candidate_count=1,\n",
        "        seed=5,\n",
        "        max_output_tokens=100,\n",
        "        stop_sequences=[\"STOP!\"],\n",
        "        presence_penalty=0.0,\n",
        "        frequency_penalty=0.0,\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "El1lx8P9ElDq"
      },
      "source": [
        "## Set system instructions\n",
        "\n",
        "[System instructions](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/system-instruction-introduction) allow you to steer the behavior of the model. By setting the system instruction, you are giving the model additional context to understand the task, provide more customized responses, and adhere to guidelines over the user interaction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7A-yANiyCLaO"
      },
      "outputs": [],
      "source": [
        "system_instruction = \"\"\"\n",
        "  You are a helpful language translator.\n",
        "  Your mission is to translate text in English to Spanish.\n",
        "\"\"\"\n",
        "\n",
        "prompt = \"\"\"\n",
        "  User input: I like bagels.\n",
        "  Answer:\n",
        "\"\"\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(\n",
        "        system_instruction=system_instruction,\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9daipRiUzAY"
      },
      "source": [
        "## Safety filters\n",
        "\n",
        "The Gemini API provides safety filters that you can adjust across multiple filter categories to restrict or allow certain types of content. You can use these filters to adjust what's appropriate for your use case. See the [Configure safety filters](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters) page for details.\n",
        "\n",
        "When you make a request to Gemini, the content is analyzed and assigned a safety rating. You can inspect the safety ratings of the generated content by printing out the model responses.\n",
        "\n",
        "The safety settings are `OFF` by default and the default block thresholds are `BLOCK_NONE`.\n",
        "\n",
        "You can use `safety_settings` to adjust the safety settings for each request you make to the API. This example demonstrates how you set the block threshold to `BLOCK_LOW_AND_ABOVE` for all categories:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yPlDRaloU59b"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "    Write a list of 5 hateful, mean, and disrespectful things that I might say to the universe after stubbing my toe in the dark.\n",
        "\"\"\"\n",
        "\n",
        "safety_settings = [\n",
        "    SafetySetting(\n",
        "        category=\"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
        "        threshold=\"BLOCK_LOW_AND_ABOVE\",\n",
        "    ),\n",
        "    SafetySetting(\n",
        "        category=\"HARM_CATEGORY_HARASSMENT\",\n",
        "        threshold=\"BLOCK_LOW_AND_ABOVE\",\n",
        "    ),\n",
        "    SafetySetting(\n",
        "        category=\"HARM_CATEGORY_HATE_SPEECH\",\n",
        "        threshold=\"BLOCK_LOW_AND_ABOVE\",\n",
        "    ),\n",
        "    SafetySetting(\n",
        "        category=\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
        "        threshold=\"BLOCK_LOW_AND_ABOVE\",\n",
        "    ),\n",
        "]\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(\n",
        "        safety_settings=safety_settings,\n",
        "    ),\n",
        ")\n",
        "\n",
        "# Response will be `None` if it is blocked.\n",
        "print(response.text)\n",
        "print(response.candidates[0].finish_reason)\n",
        "\n",
        "for safety_rating in response.candidates[0].safety_ratings:\n",
        "    print(safety_rating)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rZV2TY5Pa3Dd"
      },
      "source": [
        "## Send multimodal prompts\n",
        "\n",
        "Gemini is a multimodal model that supports multimodal prompts.\n",
        "\n",
        "You can include any of the following data types from various sources.\n",
        "\n",
        "<table>\n",
        "  <thead>\n",
        "    <tr>\n",
        "      <th>Data type</th>\n",
        "      <th>Source(s)</th>\n",
        "      <th>MIME Type(s)</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td>Text</td>\n",
        "      <td>Inline, Local File, General URL, Google Cloud Storage</td>\n",
        "      <td><code>text/plain</code></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Code</td>\n",
        "      <td>Inline, Local File, General URL, Google Cloud Storage</td>\n",
        "      <td><code>text/plain</code></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Document</td>\n",
        "      <td>Local File, General URL, Google Cloud Storage</td>\n",
        "      <td><code>application/pdf</code></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Image</td>\n",
        "      <td>Local File, General URL, Google Cloud Storage</td>\n",
        "      <td><code>image/jpeg</code> <code>image/png</code> <code>image/webp</code></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Audio</td>\n",
        "      <td>Local File, General URL, Google Cloud Storage</td>\n",
        "      <td>\n",
        "        <code>audio/aac</code> <code>audio/flac</code> <code>audio/mp3</code>\n",
        "        <code>audio/m4a</code> <code>audio/mpeg</code> <code>audio/mpga</code>\n",
        "        <code>audio/mp4</code> <code>audio/opus</code> <code>audio/pcm</code>\n",
        "        <code>audio/wav</code> <code>audio/webm</code>\n",
        "      </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Video</td>\n",
        "      <td>Local File, General URL, Google Cloud Storage, YouTube</td>\n",
        "      <td>\n",
        "        <code>video/mp4</code> <code>video/mpeg</code> <code>video/x-flv</code>\n",
        "        <code>video/quicktime</code> <code>video/mpegps</code> <code>video/mpg</code>\n",
        "        <code>video/webm</code> <code>video/wmv</code> <code>video/3gpp</code>\n",
        "      </td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4npg1tNTYB9"
      },
      "source": [
        "### Send local image\n",
        "\n",
        "Download an image to local storage from Google Cloud Storage.\n",
        "\n",
        "For this example, we'll use this image of a meal.\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/meal.png\" alt=\"Meal\" width=\"500\">"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "4avkv0Z7qUI-"
      },
      "outputs": [],
      "source": [
        "!gsutil cp gs://cloud-samples-data/generative-ai/image/meal.png ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "umhZ61lrSyJh"
      },
      "outputs": [],
      "source": [
        "with open(\"meal.png\", \"rb\") as f:\n",
        "    image = f.read()\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        Part.from_bytes(data=image, mime_type=\"image/png\"),\n",
        "        \"Write a short and engaging blog post based on this picture.\",\n",
        "    ],\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRQyv1DhTbnH"
      },
      "source": [
        "### Send document from Google Cloud Storage\n",
        "\n",
        "This example document is the paper [\"Attention is All You Need\"](https://arxiv.org/abs/1706.03762), created by researchers from Google and the University of Toronto.\n",
        "\n",
        "Check out this notebook for more examples of document understanding with Gemini:\n",
        "\n",
        "- [Document Processing with Gemini](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/document_processing.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "pG6l1Fuka6ZJ"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        Part.from_uri(\n",
        "            file_uri=\"gs://cloud-samples-data/generative-ai/pdf/1706.03762v7.pdf\",\n",
        "            mime_type=\"application/pdf\",\n",
        "        ),\n",
        "        \"Summarize the document.\",\n",
        "    ],\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "25n22nc6TdZw"
      },
      "source": [
        "### Send audio from General URL\n",
        "\n",
        "This example is audio from an episode of the [Kubernetes Podcast](https://kubernetespodcast.com/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "uVU9XyCCo-h2"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        Part.from_uri(\n",
        "            file_uri=\"https://traffic.libsyn.com/secure/e780d51f-f115-44a6-8252-aed9216bb521/KPOD242.mp3\",\n",
        "            mime_type=\"audio/mpeg\",\n",
        "        ),\n",
        "        \"Write a summary of this podcast episode.\",\n",
        "    ],\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8D3_oNUTuW2q"
      },
      "source": [
        "### Send video from YouTube URL\n",
        "\n",
        "This example is the YouTube video [Google — 25 Years in Search: The Most Searched](https://www.youtube.com/watch?v=3KtWfp0UopM).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "l7-w8G_2wAOw"
      },
      "outputs": [],
      "source": [
        "video = Part.from_uri(\n",
        "    file_uri=\"https://www.youtube.com/watch?v=3KtWfp0UopM\",\n",
        "    mime_type=\"video/mp4\",\n",
        ")\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        video,\n",
        "        \"At what point in the video is Harry Potter shown?\",\n",
        "    ],\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVlo0mWuZGkQ"
      },
      "source": [
        "## Control generated output\n",
        "\n",
        "[Controlled generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) allows you to define a response schema to specify the structure of a model's output, the field names, and the expected data type for each field.\n",
        "\n",
        "The response schema is specified in the `response_schema` parameter in `config`, and the model output will strictly follow that schema.\n",
        "\n",
        "You can provide the schemas as [Pydantic](https://docs.pydantic.dev/) models or a [JSON](https://www.json.org/json-en.html) string and the model will respond as JSON or an [Enum](https://docs.python.org/3/library/enum.html) depending on the value set in `response_mime_type`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "OjSgf2cDN_bG"
      },
      "outputs": [],
      "source": [
        "from pydantic import BaseModel\n",
        "\n",
        "\n",
        "class Recipe(BaseModel):\n",
        "    name: str\n",
        "    description: str\n",
        "    ingredients: list[str]\n",
        "\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"List a few popular cookie recipes and their ingredients.\",\n",
        "    config=GenerateContentConfig(\n",
        "        response_mime_type=\"application/json\",\n",
        "        response_schema=Recipe,\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nKai5CP_PGQF"
      },
      "source": [
        "You can either parse the response string as JSON, or use the `parsed` field to get the response as an object or dictionary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "ZeyDWbnxO-on"
      },
      "outputs": [],
      "source": [
        "parsed_response: Recipe = response.parsed\n",
        "print(parsed_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SUSLPrvlvXOc"
      },
      "source": [
        "You also can define a response schema in a Python dictionary. You can only use the supported fields as listed below. All other fields are ignored.\n",
        "\n",
        "- `enum`\n",
        "- `items`\n",
        "- `maxItems`\n",
        "- `nullable`\n",
        "- `properties`\n",
        "- `required`\n",
        "\n",
        "In this example, you instruct the model to analyze product review data, extract key entities, perform sentiment classification (multiple choices), provide additional explanation, and output the results in JSON format.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "F7duWOq3vMmS"
      },
      "outputs": [],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"ARRAY\",\n",
        "        \"items\": {\n",
        "            \"type\": \"OBJECT\",\n",
        "            \"properties\": {\n",
        "                \"rating\": {\"type\": \"INTEGER\"},\n",
        "                \"flavor\": {\"type\": \"STRING\"},\n",
        "                \"sentiment\": {\n",
        "                    \"type\": \"STRING\",\n",
        "                    \"enum\": [\"POSITIVE\", \"NEGATIVE\", \"NEUTRAL\"],\n",
        "                },\n",
        "                \"explanation\": {\"type\": \"STRING\"},\n",
        "            },\n",
        "            \"required\": [\"rating\", \"flavor\", \"sentiment\", \"explanation\"],\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "prompt = \"\"\"\n",
        "  Analyze the following product reviews, output the sentiment classification, and give an explanation.\n",
        "\n",
        "  - \"Absolutely loved it! Best ice cream I've ever had.\" Rating: 4, Flavor: Strawberry Cheesecake\n",
        "  - \"Quite good, but a bit too sweet for my taste.\" Rating: 1, Flavor: Mango Tango\n",
        "\"\"\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(\n",
        "        response_mime_type=\"application/json\",\n",
        "        response_schema=response_schema,\n",
        "    ),\n",
        ")\n",
        "\n",
        "response_dict = response.parsed\n",
        "print(response_dict)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gV1dR-QlTKRs"
      },
      "source": [
        "## Count tokens and compute tokens\n",
        "\n",
        "You can use the `count_tokens()` method to calculate the number of input tokens before sending a request to the Gemini API.\n",
        "\n",
        "For more information, refer to [list and count tokens](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Syx-fwLkV1j-"
      },
      "source": [
        "### Count tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "UhNElguLRRNK"
      },
      "outputs": [],
      "source": [
        "response = client.models.count_tokens(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"What's the highest mountain in Africa?\",\n",
        ")\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VS-AP7AHUQmV"
      },
      "source": [
        "### Compute tokens\n",
        "\n",
        "The `compute_tokens()` method runs a local tokenizer instead of making an API call. It also provides more detailed token information such as the `token_ids` and the `tokens` themselves\n",
        "\n",
        "<div class=\"alert alert-block alert-info\">\n",
        "<b>NOTE: This method is only supported in Vertex AI.</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "Cdhi5AX1TuH0"
      },
      "outputs": [],
      "source": [
        "response = client.models.compute_tokens(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"What's the longest word in the English language?\",\n",
        ")\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BsP0vXOY7hg"
      },
      "source": [
        "## Search as a tool (Grounding)\n",
        "\n",
        "[Grounding](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini) lets you connect real-world data to the Gemini model.\n",
        "\n",
        "By grounding model responses in Google Search results, the model can access information at runtime that goes beyond its training data which can produce more accurate, up-to-date, and relevant responses.\n",
        "\n",
        "Using Grounding with Google Search, you can improve the accuracy and recency of responses from the model. Starting with Gemini 2.0, Google Search is available as a tool. This means that the model can decide when to use Google Search.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_M_4RRBdO_3"
      },
      "source": [
        "### Google Search\n",
        "\n",
        "You can add the `tools` keyword argument with a `Tool` including `GoogleSearch` to instruct Gemini to first perform a Google Search with the prompt, then construct an answer based on the web search results.\n",
        "\n",
        "[Dynamic Retrieval](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini#dynamic-retrieval) lets you set a threshold for when grounding is used for model responses. This is useful when the prompt doesn't require an answer grounded in Google Search and the supported models can provide an answer based on their knowledge without grounding. This helps you manage latency, quality, and cost more effectively."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "yeR09J3AZT4U"
      },
      "outputs": [],
      "source": [
        "google_search_tool = Tool(google_search=GoogleSearch())\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"When is the next total solar eclipse in the United States?\",\n",
        "    config=GenerateContentConfig(tools=[google_search_tool]),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))\n",
        "\n",
        "print(response.candidates[0].grounding_metadata)\n",
        "\n",
        "HTML(response.candidates[0].grounding_metadata.search_entry_point.rendered_content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hYKAzG1sH-K1"
      },
      "source": [
        "### Vertex AI Search\n",
        "\n",
        "You can use a [Vertex AI Search data store](https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es) to connect Gemini to your own custom data.\n",
        "\n",
        "Follow the [get started guide for Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search) to create a data store and app, then add the data store ID in the following code cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "id": "dYDf4618IG5u"
      },
      "outputs": [],
      "source": [
        "data_store_location = \"global\"\n",
        "data_store_id = \"[your-data-store-id]\"  # @param {type: \"string\"}\n",
        "\n",
        "vertex_ai_search_tool = Tool(\n",
        "    retrieval=Retrieval(\n",
        "        vertex_ai_search=VertexAISearch(\n",
        "            datastore=f\"projects/{PROJECT_ID}/locations/{data_store_location}/collections/default_collection/dataStores/{data_store_id}\"\n",
        "        )\n",
        "    )\n",
        ")\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"What is the company culture like?\",\n",
        "    config=GenerateContentConfig(tools=[vertex_ai_search_tool]),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))\n",
        "\n",
        "print(response.candidates[0].grounding_metadata)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T0pb-Kh1xEHU"
      },
      "source": [
        "## Function calling\n",
        "\n",
        "[Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) in Gemini lets developers create a description of a function in their code, then pass that description to a language model in a request.\n",
        "\n",
        "You can submit a Python function for automatic function calling, which will run the function and return the output in natural language generated by Gemini.\n",
        "\n",
        "You can also submit an [OpenAPI Specification](https://www.openapis.org/) which will respond with the name of a function that matches the description and the arguments to call it with.\n",
        "\n",
        "For more examples of Function calling with Gemini, check out this notebook: [Intro to Function Calling with Gemini](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mSUWWlrrlR-D"
      },
      "source": [
        "### Python Function (Automatic Function Calling)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aRR8HZhLlR-E"
      },
      "outputs": [],
      "source": [
        "def get_current_weather(location: str) -> str:\n",
        "    \"\"\"Example method. Returns the current weather.\n",
        "\n",
        "    Args:\n",
        "        location: The city and state, e.g. San Francisco, CA\n",
        "    \"\"\"\n",
        "    weather_map: dict[str, str] = {\n",
        "        \"Boston, MA\": \"snowing\",\n",
        "        \"San Francisco, CA\": \"foggy\",\n",
        "        \"Seattle, WA\": \"raining\",\n",
        "        \"Austin, TX\": \"hot\",\n",
        "        \"Chicago, IL\": \"windy\",\n",
        "    }\n",
        "    return weather_map.get(location, \"unknown\")\n",
        "\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"What is the weather like in Austin?\",\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[get_current_weather],\n",
        "        temperature=0,\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h4syyLEClGcn"
      },
      "source": [
        "### OpenAPI Specification (Manual Function Calling)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "id": "2BDQPwgcxRN3"
      },
      "outputs": [],
      "source": [
        "get_destination = FunctionDeclaration(\n",
        "    name=\"get_destination\",\n",
        "    description=\"Get the destination that the user wants to go to\",\n",
        "    parameters={\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"destination\": {\n",
        "                \"type\": \"STRING\",\n",
        "                \"description\": \"Destination that the user wants to go to\",\n",
        "            },\n",
        "        },\n",
        "    },\n",
        ")\n",
        "\n",
        "destination_tool = Tool(\n",
        "    function_declarations=[get_destination],\n",
        ")\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"I'd like to travel to Paris.\",\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[destination_tool],\n",
        "        temperature=0,\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.function_calls[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MhDs2X3o0neK"
      },
      "source": [
        "## Code Execution\n",
        "\n",
        "The Gemini API [code execution](https://ai.google.dev/gemini-api/docs/code-execution?lang=python) feature enables the model to generate and run Python code and learn iteratively from the results until it arrives at a final output. You can use this code execution capability to build applications that benefit from code-based reasoning and that produce text output. For example, you could use code execution in an application that solves equations or processes text.\n",
        "\n",
        "The Gemini API provides code execution as a tool, similar to function calling.\n",
        "After you add code execution as a tool, the model decides when to use it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "id": "1W-3c7sy0nyz"
      },
      "outputs": [],
      "source": [
        "code_execution_tool = Tool(code_execution=ToolCodeExecution())\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"Calculate 20th fibonacci number. Then find the nearest palindrome to it.\",\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[code_execution_tool],\n",
        "        temperature=0,\n",
        "    ),\n",
        ")\n",
        "for part in response.candidates[0].content.parts:\n",
        "    if part.executable_code:\n",
        "        print(\"Language:\", part.executable_code.language)\n",
        "        display(\n",
        "            Markdown(\n",
        "                f\"\"\"\n",
        "```py\n",
        "{part.executable_code.code}\n",
        "```\n",
        "\"\"\"\n",
        "            )\n",
        "        )\n",
        "    if part.code_execution_result:\n",
        "        print(\"\\nOutcome:\", part.code_execution_result.outcome)\n",
        "        display(Markdown(f\"`{part.code_execution_result.output}`\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5xHmIVPv1WH"
      },
      "source": [
        "## Spatial Understanding\n",
        "\n",
        "Gemini 2.0 includes improved spatial understanding and object detection capabilities. Check out this notebook for examples:\n",
        "\n",
        "- [2D spatial understanding with Gemini 2.0](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/spatial-understanding/spatial_understanding.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQwiONFdVHw5"
      },
      "source": [
        "## What's next\n",
        "\n",
        "- See the [Google Gen AI SDK reference docs](https://googleapis.github.io/python-genai/).\n",
        "- Explore other notebooks in the [Google Cloud Generative AI GitHub repository](https://github.com/GoogleCloudPlatform/generative-ai).\n",
        "- Explore AI models in [Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "hIJVEr0RQY8S",
        "rZV2TY5Pa3Dd",
        "hYKAzG1sH-K1",
        "mSUWWlrrlR-D",
        "h4syyLEClGcn"
      ],
      "name": "intro_gemini_2_0_pro.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}