01A-visual-segments-frames-shots-scenes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6416fcdf-75c5-49fc-a9ec-adb8018df284",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Video segments: frames, shots and scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe197858",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "_Video time segmentation_ is an important data preparation step that can help unlock the full potential of video content for analysis and automation. By breaking down a video into meaningful segments, you can better understand the structure and context of the content, enabling a wide range of applications such as:\n",
    "\n",
    "* Identifying key events, scenes, or chapters within the video\n",
    "* Inserting metadata like ad markers or chapter markers\n",
    "* Reusing relevant clips or segments for new purposes\n",
    "* Applying advanced analytics and foundation models to specific parts of the video\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b61422",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "![Video file decomposed into frames, shots and scenes](./static/images/01-visual-segments.jpg) \n",
    "\n",
    "***Figure:** Video file decomposed into frames, shots (numbered), and scenes (colored)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2545f85-1a23-4923-8577-f471f0293ce2",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "In this notebook, you'll explore techniques to decompose a video into smaller segments using visual cues. Specifically, you'll:\n",
    "\n",
    "* Break down the video into frames, shots, and scenes using visual analysis.\n",
    "* Generate composite images for video segments that can be used in prompts to Foundation Models (FMs) on Amazon Bedrock to generate insights about the content of video clips.\n",
    "* Practice prompt engineering with a Foundation Model to understand video content.\n",
    "\n",
    "By the end of this notebook, you'll have a collection of segmentations for your video that can serve as a foundation for further analysis, automation, and reuse of the video assets. Let's get started!\n",
    "\n",
    "The outputs of this notebook will be used in the use case sections later on in this workshop.  \n",
    "\n",
    "<br/><br/>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 We'll be using some Python libraries to accomplish the segmentation tasks during this part of the workshop.  The libraries are in the <b>lib/</b> folder of this project if you would like to dive deep into the code.  We are not going in to all the implementation details for this section because we want to leave more time for focusing on prompt engineering and solving use cases.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Video segmentation can be done along the temporal dimension or along the spatial dimension.  In the context of this notebook, the term “segmentation” will always be <i>time segmentation</i>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0182508-2fb3-4ad5-a154-02d94b92b493",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Key terms and definitions\n",
    "\n",
    "You can refer back to this section if you want the definition of terms used in the notebook.\n",
    "\n",
    "- **Frame** - frame image extracted from the video content\n",
    "- **Frame sampling** - the selection of a subset of representative frames from a video\n",
    "- **Shot** - continuous sequences of frames between two edits or cuts that defines one action\n",
    "- **Scene** - continuous sequence of action taking place in a specific location and time, consisting of a series of shots.\n",
    "- **Frame accurate timestamp** - a timestamp that can be mapped to a specific frame.  Frame accurate timestamps are useful for synchronization of video elements"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "690876f8-42e1-43fa-9319-f0f6fda8e572",
   "metadata": {
    "editable": false,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Workflow\n",
    "\n",
    "The purpose of this lab is to give hands-on practice working with the visual elements of video at the frame, shot, and scene level and practice prompt engineering with frame sequences that represent video clips.  We'll be working with AWS services from the Sagemaker notebook throughout the activity.\n",
    "\n",
    "![scene detection and contextualization workflow](./static/images/01-scenes-shots-workflow-w-ouputs-drawio.png)\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Click on the list icon in the left navigation panel in this Jupyter notebook to see the outline of the notebook and where you currently are.\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6ed3f92-e7fd-4e2f-a8df-8646718e00ee",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1488c3b7-2901-497e-8a1e-10d768f31943",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Import python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a453ab94-9b8a-42fe-a0e5-2b2e742d71fb",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "from termcolor import colored\n",
    "from IPython.display import JSON\n",
    "from IPython.display import Video\n",
    "from IPython.display import Image as DisplayImage\n",
    "from lib.frames import VideoFrames\n",
    "from lib.shots import Shots\n",
    "from lib.scenes import Scenes\n",
    "from lib.transcript import Transcript\n",
    "from lib import util\n",
    "import requests\n",
    "from tqdm import tqdm\n",
    "from lib import frame_utils\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "import time\n",
    "from functools import cmp_to_key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3022f442-6ef9-4621-8c15-09eab7d82398",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Retrieve saved values from previous notebooks\n",
    "\n",
    "To run this notebook, you need to have run the previous notebook: [00_prerequisites.ipynb](./00-prequisites.ipynb), where you installed package dependencies and gathered some information from the SageMaker environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "501ad005-d406-404e-9d6a-ef4d673980d6",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get variables from the previous notebook\n",
    "%store -r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aa59027-3d7e-4f4a-bd9c-cfc4a6d6b078",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54db107e-04e8-4604-9286-dac677a2eed8",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Download the sample video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e7890d0-7859-4a59-853f-616da28b668c",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def download_video(url: str, output_path: str) -> str:\n",
    "    \"\"\"\n",
    "    Download test video if not already present\n",
    "    \"\"\"\n",
    "    \n",
    "    if os.path.exists(output_path):\n",
    "        print(f\"Video already exists at {output_path}\")\n",
    "        return output_path\n",
    "    \n",
    "    print(\"Downloading test video...\")\n",
    "    response = requests.get(url, stream=True)\n",
    "    total_size = int(response.headers.get('content-length', 0))\n",
    "    \n",
    "    with open(output_path, 'wb') as file, tqdm(\n",
    "        desc=output_path,\n",
    "        total=total_size,\n",
    "        unit='iB',\n",
    "        unit_scale=True,\n",
    "        unit_divisor=1024,\n",
    "    ) as pbar:\n",
    "        for data in response.iter_content(chunk_size=1024):\n",
    "            size = file.write(data)\n",
    "            pbar.update(size)\n",
    "            \n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6aa45f9-432b-4b91-ae6e-287e5c022788",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Choose a sample video\n",
    "\n",
    "* Meridian,Mystery from Netflix - This content is available under the [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/legalcode)\n",
    "* Sintel, 2010, Animation and Fantasy from The Blender Foundation - This content is available under the [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/legalcode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e26c3cfa-2e63-4611-83cb-98b932c16650",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video = {}\n",
    "\n",
    "# Video Alternatives\n",
    "# Drama\n",
    "MERIDIAN='Netflix_Open_Content_Meridian.mp4'\n",
    "SINTEL='Sintel2010-720p.mp4'\n",
    "\n",
    "video[\"path\"] = MERIDIAN\n",
    "video[\"url\"] = f\"https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/7db2455e-0fa6-4f6d-9973-84daccd6421f/{video['path']}\"\n",
    "\n",
    "video[\"output_dir\"] = Path(video[\"path\"]).stem\n",
    "\n",
    "download_video(video[\"url\"], video[\"path\"])\n",
    "\n",
    "Video(url=video[\"url\"], width=640, height=360, html_attributes=\"controls muted autoplay\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57281a0b-9167-4e38-b3cc-d349bc93ac33",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Sample frames \n",
    "\n",
    "In this section you will extract video frames at one frame per second with a 392×220 pixel resolution.  These settings are optimized for visual quality and computational efficiency through numerous experiments. In this process, we sample one frame per second.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 392×220 pixel resolution is chosen to optimize the number of frames we can present to our chosen Foundation Model, Anthropic Claude Sonnet 3.5, while still retaining the level of detail we need for our use cases.  Different use cases may use higher or lower resolutions for lower cost or higher quality. \n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 One frame per second sampling is a design choice that is suitable for the content used here, but can be adjusted for high-motion, high-frame-rate videos such as sports or more static video such as newsroom footage.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f2c7fd1-0b85-43d8-b07c-b98224eb6215",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Extract frames from the video\n",
    "\n",
    "You'll be using a Python package, [VideoFrames](./lib/frames.py), to work with the video at the frame level.  This package is available on GitHub.  When you call the main method of VideoFrames, it will do the following steps to help prepare the video for frame based analysis with machine learning:\n",
    "\n",
    "1. Extract frames from the video, sampling at the specified frame rate, and store the resulting images in the folder `./<video name>/frames/`.\n",
    "2. The resulting frame metadata contains the following attributes for each frame:\n",
    "\n",
    "* **timestamp_millis** — the timestamp, in milliseconds, where the frame appears in the video.  We'll use this timestamp to related video analysis results back to the video timeline.\n",
    "* **image_file** — the location of the image in the `frames` folder.\n",
    "* **id** - the unique frame id\n",
    "\n",
    "Let's give it a try. \n",
    "\n",
    "⏳ Generating frames will take a few minutes to process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d271cbf6-00bc-464e-99e9-aed7366c22fa",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Number of frames to sample per second of video\n",
    "FRAME_SAMPLING_RATE = 1\n",
    "\n",
    "video[\"frames\"] = VideoFrames(video[\"path\"], session['bucket'], max_res=(392, 220), sample_rate_fps=FRAME_SAMPLING_RATE, force=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e83fee98-3a79-4f2b-9d24-ddaa9d7d4751",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(JSON(video[\"frames\"].frames[0], root=\"first frame\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1cb110d-0f5b-430e-b87a-a4437caf7c0c",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Visualize the extracted frames\n",
    "\n",
    "Next, let's visualize the extracted frames of the video.  \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "🤔 Do you notice any visual patterns in the frames? Based on the frames, can you predict how many shots are in the video?\n",
    "</div>\n",
    "\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "400ef001-10c7-4b0a-8976-ba056dbf61a1",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['frames'].display_frames(start=0, end=len(video['frames'].frames))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a5f0fe-5a11-4a59-bddb-9946b0acfdc4",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Detect shots\n",
    "\n",
    "A shot is a continuous sequence of frames between two edits or cuts that define one action.  Usually, a shot represents a single camera position, but sometimes, shots may contain camera movements such as panning or zooming.  Frames that belong to the same shot should be similar.  Therefore, one way to implement shot detection would be to use image embeddings such as Amazon [Titan Multimodal Embeddings](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html) to group similar frames to shots.   However, since we are operating on sampled frames, the accuracy of timestamps for the shots would be limited by our frame sampling rate.\n",
    "\n",
    "For use cases like ad insertion, editing and search, the ideal is to use **frame accurate** timestamps that identify the exact frame where the shots begin and end.  [Amazon Rekognition's Segment API](https://docs.aws.amazon.com/rekognition/latest/dg/segments.html) is a video analysis service that automatically detects technical cues and shot boundaries in video content which provides frame-accurate timestamps for each shot boundary.\n",
    "\n",
    "Use the [Shots](lib/shots.py) Python library to generate shots using the Amazon Rekognition's Shot Segment API and map the shots to the frames we sampled in the previous step.  We'll use this result to build large grouping of shots into scenes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18621ebe-436c-494b-872d-a947466a7a53",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Shots library\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note</b>: The code for shot detection is located in the <b>lib</b> folder of this project if you want to dive deeper, but the objective of this exercise is to understand the concept of a shot.    \n",
    "</div>\n",
    "\n",
    "Click the link to open [lib/shots.py](lib/shots.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6bd5618-5d58-4662-a554-67f6d4e50125",
   "metadata": {},
   "source": [
    "## Run shot detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ff52fcf-8ffe-432c-9f71-a26d60dcdd5d",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video[\"shots\"] = Shots(video[\"frames\"], method=\"RekognitionShots\")\n",
    "\n",
    "print(f\"Number of shots: {len(video['shots'].shots)} from {len(video['frames'].frames)} frames\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c826c59c-2969-467b-ba9c-d342616ff242",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Take a moment to look at the metadata for a sample shot from the results.  Each shot contains:\n",
    "\n",
    "* **method** - the method used to group frames into shots.  Possible values are `SimilarFrames` or `RekognitionShots`\n",
    "* **start_ms** - the starting timestamp of the shot\n",
    "* **end_ms** - the ending timestamp of the shot\n",
    "* **duration_ms** - the duration of the shot\n",
    "* **video_asset_dir** - the location of the metadata collected for this video\n",
    "* **start_frame_id** - the frame the shot begins with\n",
    "* **end_frame_id** - the frame the shot ends with\n",
    "* **composite_images** - a series of equal sized frame grids containing the frames for the shots.  Composite images can be used as inputs to multi-modal foundation models to generate insights about the shot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a88c372d-25d7-40bc-a4b3-c2e006323964",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Show metadata for shot 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d653fdc-3c75-4c74-a927-944fb7f968e5",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(JSON(video[\"shots\"].shots[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fc3760d-8ce2-465c-919c-c1842f7915a2",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The `Shots` method creates a set of composite images consisting of the frames in each shot.  We will be using these composite images later on as inputs to Anthropic Claude Sonnet 3 on Amazon Bedrock to generate inferences to understand what is happening in the shots.  For now, you can examine the resulting composite images to visualize the shots. \n",
    "\n",
    "Show the composite images for shot 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6717b0eb-3ae9-42da-812b-174ab02a9e1d",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "shot = video[\"shots\"].shots[1]\n",
    "for idx, composite_image in enumerate(shot['composite_images']):\n",
    "    \n",
    "    print (f'\\nShot {shot[\"id\"] } Composite image file { idx+1 } of { len(shot[\"composite_images\"]) }: { composite_image[\"file\"] }\\n')\n",
    "    display(DisplayImage(filename=composite_image['file']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4686b410-1e13-4c26-8745-f994ef5523b1",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "Before we move on to scene detection, let's take a look at all of the generated shots.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Use the scroll bar in the output box to view the shots. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b74a1ede-670f-4d79-a271-1fe63737fe71",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# visualize the shots\n",
    "for counter, shot in enumerate(video[\"shots\"].shots):\n",
    "    print(f'\\nSHOT {counter}: frames {shot[\"start_frame_id\"] } to {shot[\"end_frame_id\"] } =======\\n')\n",
    "    video['frames'].display_frames(start=shot[\"start_frame_id\"], end=shot[\"end_frame_id\"]+1)\n",
    "    \n",
    "    # ALTERNATIVE: Display composite images for each shot\n",
    "    #for image_file in shot['composite_images']:\n",
    "    #    display(DisplayImage(filename=image_file['file'], height=75))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf721f8-6f3f-4137-9019-09757c11bac3",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Use the scroll bar in the output box to view the shots in the cell above.  You can also drag the lower right corner to increase the size of the output cell.  \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4edc4856-45b3-4335-afb9-b0fbdefd2e8e",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "🤔 As you look through the lists of shots, do you notice any segments that have unexpected results?  If you do, you may have discovered some tricky situations for automatic video segmentation.  These include:\n",
    "* Motion of the subject including rolling credits, cars, etc.\n",
    "* Motion of the camera in the form of panning shots and zooming shots\n",
    "* Fades and other transition effects\n",
    "\n",
    "\n",
    "Before we move on to scene detection, let's play a couple of adjacent shots and observe how they will look as a video clip.  Play `shot[12]`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69d46686-f4f4-4f11-bbf3-bac5228bd0e4",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "start = video['shots'].shots[12]['start_ms']/1000\n",
    "end = video['shots'].shots[12]['end_ms']/1000\n",
    "shot_url = f'{video[\"url\"]}#t={start},{end}'\n",
    "Video(url=shot_url, width=640, height=360)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bbf3c24-b9f7-4791-aba4-470cfb619382",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Play `shot[13]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac497fb1-6e91-4d10-bb68-e99563b9155b",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "start = video['shots'].shots[13]['start_ms']/1000\n",
    "end = video['shots'].shots[13]['end_ms']/1000\n",
    "shot_url = f'{video[\"url\"]}#t={start},{end}'\n",
    "Video(url=shot_url, width=640, height=360)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4dac4a-9954-4851-8e31-2f37b0c5e81c",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "🤔 If you are running the workshop with Meridian, these shots show two people in the same room having a conversation that spans the shots.  Because these two shots are in the same setting, they belong together into a higher level grouping.  In the next section, we'll group shots together based on the visual information to get a more holistic view of the scene."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a814895-5bbf-40dc-86e8-a1cec69a747f",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Detect scenes\n",
    "\n",
    "\n",
    "Even after identifying individual camera shots, there may still be semantically similar shots depicting the same setting. To further cluster these into distinct scenes, we expand frame comparison beyond adjacent frames. By looking at similar frames across an expanded time window, we can identify shots that are likely part of the same contiguous scene. We calculate pairwise similarity scores between all frames within a given time window. Frames with similarity scores above a certain threshold are considered part of the same scene group. This process performs recursively across all frames in a shot. \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 The time window size and similarity threshold are parameters that can significantly impact the accuracy of scene boundary detection. In our example, a 30 second time window and 0.80 similarity threshold gave the best scene clustering results across our video samples, but this can be adjusted.\n",
    "</div>\n",
    "\n",
    "We accomplish scene grouping by first indexing all video frames using TME again and storing the embeddings along with their shot information and timestamps into a vector database, as illustrated in the following figure.\n",
    "\n",
    "![shots-to-scenes.png](./static/images/01-vectorization.jpg)\n",
    "\n",
    "We then perform a recursive similarity search against this indexed frame corpus. For each frame, we find all other frames within a 3-minute time window in both directions with greater than 80% contextual similarity based on their vector representations. The shot information for these highly similar frames is recorded. This process iterates for all frames in a shot to compile contextually similar shots. This process repeats across all shots, and the compiled results look like this example:\n",
    "\n",
    "    shot 1 –> 2, 3, 4\n",
    "    shot 2 –> 1, 3\n",
    "    shot 3 –> 2, 4, 5\n",
    "    shot 7 –> 8, 9\n",
    "\n",
    "Finally, we run a reduction process to group shots that are mutually identified as highly similar into distinct scene groups as follows:\n",
    "\n",
    "    shot 1, 2, 3, 4, 5 → scene 1\n",
    "    shot 7, 8, 9 → scene 2\n",
    "\n",
    "This allows us to segment the initially detected shot boundaries into higher-level semantic scene boundaries based on visual and temporal coherence. The end-to-end process is illustrated in the following diagram."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88b9f11f-b9c7-44e5-b63b-ad87684e0646",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Create frame embeddings\n",
    "\n",
    "Image embeddings are numerical representations (vectors) of images that capture their essential features and characteristics.  These embeddings make it possible to perform mathematical operations on images and compare them in ways that align with human visual perception.  \n",
    "\n",
    "We'll create an image embedding for each frame from our video using [Amazon Titan Multimodal Embeddings](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html) in Amazon Bedrock.  We'll be using helper functions in the [lib/frames.py](./lib/frames.py) to accomplish this task.  \n",
    "\n",
    "The calculated frame embeddings will be added to each frame in the `Frames` object that is stored in the `video` variable.\n",
    "\n",
    "Calling the method `make_titan_multimodal_embeddings()` from the `Frames` class will create frame embeddings and store them with the metadata for each frame.  \n",
    "\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "If you get an <b>AccessDenied</b> error at this point, make sure you completed the step to enable model access for Amazon Titan Multimodal Embeddings and Anthropic Claude Sonnet 3 in the Amazon Bedrock console.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "⏳ Generating embeddings should take 2-5 minutes.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cc64e12-f20b-4391-96da-af9580d78bdd",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['frames'].make_titan_multimodal_embeddings()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b35a6727-2ea7-4a32-b05d-24821f88eae2",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Use the next cell to print the metadata for the first frame and examine the `titan_multimodal_embedding` attribute.   It's a large vector that encodes the content of the frame in the vector space for the `amazon.titan-embed-image-v1` version of the Titan Multimodal Embeddings model.  When we compare this vector with other vectors encoded using the same model version, we can determine if they are similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4e7303e-6dc8-4809-a2f4-98ba416c6db2",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(JSON(video[\"frames\"].frames[0], root=\"first frame\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a85819de-2430-4308-849d-044a25c0b104",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Next, lets do some comparisons of the first few frames using embeddings.  First, print a few of the sampled frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edae9046-0242-4dc2-bcf8-1752de7979f7",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['frames'].display_frames(start=0, end=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b3150cb-f2c3-45b5-a329-b92cd0471578",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "In order to compare frames, we need a way to compare embeddings.  We'll implement a [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) function using the Python numpy package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d45abdf-096e-436b-9ec6-127d02329a91",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy import dot\n",
    "from numpy.linalg import norm\n",
    "\n",
    "def cosine_similarity(a, b):\n",
    "    cos_sim = dot(a, b) / (norm(a) * norm(b))\n",
    "    return cos_sim"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5aeaf5-6262-484e-9e63-0bbf52d2bd2c",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Next, let's test comparing some frames.\n",
    "\n",
    "Compare the first black frame to the second frame which is a view of a city street.  As expected, the similarity score is low as these frames are not very similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22d262bd-964c-4e1e-92db-7e91f68decd5",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "frms = video['frames'].frames\n",
    "cosine_similarity(frms[0]['titan_multimodal_embedding'], frms[1]['titan_multimodal_embedding'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99c541c-09f2-4998-baf2-0e8c337511bb",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Now, compare the second frame to the third frame.  The similarity score should be higher, since the main difference in these frames is the lettering with the words \"Los Angeles 1947\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ac48082-b77e-483e-b33c-8bb83fc57c41",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "cosine_similarity(frms[1]['titan_multimodal_embedding'], frms[2]['titan_multimodal_embedding'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7d367b3-8253-4fa6-a907-4ada3a6acd54",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Populate a FAISS vector store\n",
    "\n",
    "We will be using a local FAISS vector store so we can use a single search command to find all the frames that are similar to a particular frame all at once.  Our search function will use the same cosine similarity method we explored in the previous section. There are a number of different databases on AWS that can be used as a vector store.  One popular choice is [Amazon Opensearch](https://aws.amazon.com/opensearch-service/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33b5b0e0-be79-429f-8535-78f7f5eb0ada",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['frames'].make_vector_store()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c83f33d1-cab9-4af3-bb4e-b8610fc14c86",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Test similarity search using the vector store\n",
    "\n",
    "Let's use the vector store to find similar frames for the second frame in the video.  This is the first frame of the sequence that displays the words \"Los Angeles 1947\".  Based on inspection of the frames, we should get 3 _adjacent_ similar frames, but there are also several similar frames that are not adjacent.   \n",
    "\n",
    "Our similarity search function uses a [cosine similarity function](https://en.wikipedia.org/wiki/Cosine_similarity) to determine the [K nearest neighbors](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) in the vector space. There are two parameters that you can adjust to tune the result of the similarity search:\n",
    "\n",
    "* MIN_SIMILARITY is the similarity threshold.  \n",
    "* TIME_RANGE is the maximum time range to compare frames to from the input frame\n",
    "\n",
    "You can try different values of these parameters to get a feel for how the results change. The frames 0-19 are displayed after this to help visualize  the results.  Here are some good values to try:\n",
    "\n",
    "* MIN_SIMILARITY = .85, TIME_RANGE = 30\n",
    "* MIN_SIMILARITY = .70, TIME_RANGE = 30\n",
    "* MIN_SIMILARITY = .80, TIME_RANGE = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb87383f-4a7b-4614-93b2-d1494a300101",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "MIN_SIMILARITY = .80\n",
    "TIME_RANGE = 30\n",
    "FRAME_ID = 1\n",
    "video['frames'].search_similarity(FRAME_ID, min_similarity = MIN_SIMILARITY, time_range = TIME_RANGE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c107959d-f991-4353-8158-986f59236e2b",
   "metadata": {
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "video['frames'].display_frames(start=0, end=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "859c9107-d4ce-4578-a3a3-1412902abe06",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Scenes library\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note</b>: The code for scene detection is located in the <b>lib</b> folder of this project if you want to dive deeper, but the objective of this exercise is to understand the concept of a scene.    \n",
    "</div>\n",
    "\n",
    "Click the link to open [lib/scenes.py](lib/scenes.py)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a88cbde-0af5-4026-b454-5574cd21260f",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Execute the Scene detection\n",
    "\n",
    "Now let's apply this similarity search to all the frames across shots to find similar shots.  If shots are similar within the TIME_RANGE, then they will be grouped to the same scene.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b780a526-5b4f-4d35-8143-0ed72041c388",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "MIN_SIMILARITY = .90\n",
    "\n",
    "TIME_RANGE = 30\n",
    "\n",
    "video['scenes'] = Scenes(video['frames'], video['shots'].shots, MIN_SIMILARITY, TIME_RANGE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2706434f-e5f0-4668-a731-336bdbec7636",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Visualize the scenes\n",
    "\n",
    "Now let's visualize some scenes using the generated composite images. Note that some scenes will have more than one composite image.\n",
    "\n",
    "\n",
    "🤔 Do you think the scenes grouped the shots in a way that makes sense?  Is there anything you would want to change?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "529fae5b-f633-4b6f-9d11-c19e27b7f937",
   "metadata": {
    "editable": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# visualize the scenes\n",
    "for counter, scene in enumerate(video[\"scenes\"].scenes):\n",
    "    print(f'\\nScene {counter}: frames {scene[\"start_frame_id\"] } to {scene[\"end_frame_id\"] } =======\\n')\n",
    "    video['frames'].display_frames(start=scene[\"start_frame_id\"], end=scene[\"end_frame_id\"]+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a15ee898-13cc-47fc-90dc-8f69ba3db4a7",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "💡 Make sure to use the scroll bar in the output box above to view the scenes.  You can also drag the lower right corner to increase the size of the output cell.  \n",
    "</div>\n",
    "\n",
    "\n",
    "Finally, let's play a couple of adjacent scenes and observe how they will look as a video clips. As you play the video segments, pay attention to the transition of the _video and the audio_ from one scene to the next.  Because the scenes are created based on only the visual information, the audio can get cut off if you clip on a scene boundary.  In the next notebook, we'll look at the audio segmentation of videos. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9206c3ae-4359-43d5-8745-5d7022aa2e3e",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "start_scene = 3\n",
    "start = video['scenes'].scenes[start_scene]['start_ms']/1000\n",
    "end = video['scenes'].scenes[start_scene]['end_ms']/1000\n",
    "print(f\"scene { start_scene } duration: {video['scenes'].scenes[start_scene]['duration_ms']/1000} seconds\\n\")\n",
    "print(f\"start time: { start } end time: {end} seconds\\n\")\n",
    "shot_url = f'{video[\"url\"]}#t={start},{end}'\n",
    "Video(url=shot_url, width=640, height=360)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7105f194-39f4-425b-aaea-a86b14cfe801",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Play the next scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d5223b5-ccc1-4828-81f7-3e549a4f572c",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "start_scene = start_scene + 4\n",
    "start = video['scenes'].scenes[start_scene]['start_ms']/1000\n",
    "end = video['scenes'].scenes[start_scene]['end_ms']/1000\n",
    "print(f\"scene { start_scene } duration: {video['scenes'].scenes[start_scene]['duration_ms']/1000} seconds\\n\")\n",
    "print(f\"start time: { start } end time: {end} seconds\\n\")\n",
    "shot_url = f'{video[\"url\"]}#t={start},{end}'\n",
    "Video(url=shot_url, width=640, height=360)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47ab5a23-e97c-4522-9b80-c359fde05bdf",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "🤔 We just played scenes 9 and 10.  You may have noticed that the scene changed on a visual cue when the focus of the video turns towards the beach and the ocean.  _However_, this scene change occurs in the middle of the audio of the police dispatcher speaking over the radio.  If we want to identify clean breakpoints in the video to make clips, we should probably consider not only the visual content but the audio as well. We'll explore this topic in the next part of the workshop [01B Combining Audio and Video Segments](./01B-combining-audio-and-video.ipynb).  But, before we move on, let's do our first exploration of prompt engineering with video segments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93ab3ae0-d099-470c-8630-207328ba7327",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Generate contextual metadata for video segments with Amazon Bedrock\n",
    "\n",
    "![Prompt engineering with frame sequences](./static/images/01-prompt-engineering-with-frame-sequences.jpg)\n",
    "TBD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63a997c-26a6-4dad-b888-6d1908934cc1",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# What's next?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98de755-2cd6-451a-9570-7b26474447e7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Store the video metadata so it can be used in the rest of the workshop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee6a12e-9e70-4120-b1de-1b23f827c2a7",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%store video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5768c9f6-c3a4-4539-afd7-0056d5604d97",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Continue to the next section of the workshop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59aa2234-0461-4488-a50b-dbfd8c8ea48a",
   "metadata": {
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "In the next sections of the workshop, you will use Amazon Transcribe to generate a transcript from the speech within the video.  The transcipt provides addtional information that can be used to find segements based on the context of the video narrative.  The transcript can also be used to provide more input context in prompts for video clips.\n",
    "\n",
    "Go to the next notebook [Audio Segments](./01B-audio-segments.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f85a7c8-241c-4b91-84f8-35ed3521dcb0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.m5.8xlarge",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}