async-inference/Transcription_on_SM_endpoint.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "62cd2cb0-d482-4e19-bc19-94acd1124ca5",
   "metadata": {},
   "source": [
    "## Transcription inference on Amazon SageMaker Inference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "812c1224",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "332b87d6",
   "metadata": {},
   "source": [
    "##### A near real-time inference for transcription using Whisper model\n",
    "\n",
    "#### Table of Contents\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Background\n",
    "Transcribe is the go-to service for transcription in AWS.\n",
    "However, for non-supported languages, we can use other models (in our case Whisper) that will be deployed in Amazon SageMaker for inference.\n",
    "For short audio files that the inference takes up to 60 seconds, we can use real-time inference.\n",
    "For inference that takes longer than 60 seconds, or in the case we want to save on costs by autoscaling the instance count to zero when there are no requests to process, asynchronous inference should be used.\n",
    "\n",
    "## Notebook scope\n",
    "This notebook provides 2 deployments options for the Whisper model - real-time and asynchronous inference - including auto-scaling setup and asynchronous inference invocation example \n",
    "\n",
    "We used Data Science image to execute the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afbb5b4f-604e-431e-b443-413c2bff7849",
   "metadata": {},
   "source": [
    "## 1. Prepare the model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddd3fc63-d062-4335-8877-6bb89ad60e24",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir model\n",
    "!mkdir model/code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a41cdd-e63f-42a2-a7af-f1296602676b",
   "metadata": {},
   "source": [
    "Create a customer inference code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4401a262-e873-447b-a450-1bd6806c58d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile model/code/inference.py\n",
    "import whisper\n",
    "import boto3\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "\n",
    "def model_fn(model_dir):\n",
    "    model = whisper.load_model(\"large-v2\")\n",
    "    return model\n",
    "\n",
    "\n",
    "def transcribe_from_s3(model, s3_file, language=None):\n",
    "    s3 = boto3.client(\"s3\")\n",
    "    o = urlparse(s3_file, allow_fragments=False)\n",
    "    bucket = o.netloc\n",
    "    key = o.path.lstrip(\"/\")\n",
    "\n",
    "    s3.download_file(bucket, key, \"tmp.wav\")\n",
    "    result = model.transcribe(\"tmp.wav\", language=language)\n",
    "\n",
    "    return result[\"language\"], result[\"text\"], result[\"segments\"]\n",
    "\n",
    "\n",
    "def predict_fn(data, model):\n",
    "    s3_file = data.pop(\"s3_file\")\n",
    "    language = data.pop(\"language\", None)\n",
    "\n",
    "    detected_language, transcription, segments = transcribe_from_s3(model, s3_file, language)\n",
    "\n",
    "    return {\n",
    "        \"detected_language\": detected_language,\n",
    "        \"transcription\": transcription,\n",
    "        \"segments\": segments,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a331280-7719-47e2-a9dd-3a528e024649",
   "metadata": {},
   "source": [
    "In requirements.txt file we put the libraries we will need to run the inference code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88ba6f59-1dc3-442a-a3fc-797dd2c95eca",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"model/code/requirements.txt\", \"w\") as f:\n",
    "    f.write(\"transformers==4.25.1\\n\")\n",
    "    f.write(\"git+https://github.com/openai/whisper.git\\n\")\n",
    "    f.write(\"boto3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "278cba87-f620-4538-9c46-751934b99995",
   "metadata": {},
   "source": [
    "### Uploading the model to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4377d64e-ea3e-47a7-883e-fe8291ed203d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8111a49-068a-4997-ade8-3d2b7c3949b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm model.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dd97fa9-bb42-4c8d-8cd4-a2eb57dafebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar zcvf model.tar.gz *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c91ac1c-3f08-4266-89e7-fa90a0d472e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "sagemaker_session_bucket = None\n",
    "if sagemaker_session_bucket is None and sess is not None:\n",
    "    sagemaker_session_bucket = sess.default_bucket()\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    iam = boto3.client(\"iam\")\n",
    "    role = iam.get_role(RoleName=\"sagemaker_execution_role\")[\"Role\"][\"Arn\"]\n",
    "\n",
    "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n",
    "print(f\"sagemaker session region: {sess.boto_region_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85560335-2f5d-4cc9-a3b7-ad5bac88231f",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_location = f\"s3://{sagemaker_session_bucket}/whisper/model/model.tar.gz\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52b85169-c847-4cea-b8fc-a02997f22474",
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp model.tar.gz $s3_location"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e966d7-55f8-4f13-9f25-e6964ff9ae7f",
   "metadata": {},
   "source": [
    "## 2. Real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35cb0a59-d392-4384-ac32-83006f8634b6",
   "metadata": {},
   "source": [
    "### Deploying the model to a real-time inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aeb50fd-359b-48f3-b22a-692431acd896",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "rt_endpoint_name = name_from_base(\"whisper-large-v2-custom\")\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "    model_data=s3_location,  # path to your model and script\n",
    "    role=role,  # iam role with permissions to create an Endpoint\n",
    "    transformers_version=\"4.17\",  # transformers version used\n",
    "    pytorch_version=\"1.10\",  # pytorch version used\n",
    "    py_version=\"py38\",  # python version used\n",
    ")\n",
    "\n",
    "# deploy the endpoint endpoint\n",
    "rl_predictor = huggingface_model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=\"ml.g4dn.xlarge\",\n",
    "    endpoint_name=rt_endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54ec7ca5-04cf-46bf-bbcb-1362b9b99bbb",
   "metadata": {},
   "source": [
    "### Execute inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ca4498d-8133-4701-98ec-7d41180a40e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with a path to audio object in S3\n",
    "# Comment out the language line if you want to specify the input language. Otherwise it will detect it automatically\n",
    "\n",
    "\n",
    "data = {\n",
    "    \"s3_file\": \"REPLACE WITH A PATH TO AUDIO OBJECT IN S3\"\n",
    "    # \"language\": \"pl\"\n",
    "}\n",
    "\n",
    "res = rl_predictor.predict(data=data)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29186047-de59-4061-acff-fec088689210",
   "metadata": {},
   "source": [
    "## 3. Asynchronous inference\n",
    "\n",
    "For inference that takes longer than 60 seconds, or in the case we want to save on costs by autoscaling the instance count to zero when there are no requests to process, asynchronous inference should be used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d09305-6413-4f25-8c19-8ea25929c9c8",
   "metadata": {},
   "source": [
    "### Deploying the model to an asynchronous inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "682715ad-622b-4482-b197-5434dbfc00c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "from sagemaker.async_inference.async_inference_config import AsyncInferenceConfig\n",
    "from sagemaker.s3 import s3_path_join\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "async_endpoint_name = name_from_base(\"whisper-large-v2-custom-asyc\")\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "    model_data=s3_location,  # path to your model and script\n",
    "    role=role,  # iam role with permissions to create an Endpoint\n",
    "    transformers_version=\"4.17\",  # transformers version used\n",
    "    pytorch_version=\"1.10\",  # pytorch version used\n",
    "    py_version=\"py38\",  # python version used\n",
    ")\n",
    "\n",
    "# create async endpoint configuration\n",
    "async_config = AsyncInferenceConfig(\n",
    "    output_path=s3_path_join(\n",
    "        \"s3://\", sagemaker_session_bucket, \"async_inference/output\"\n",
    "    ),  # Where our results will be stored\n",
    "    # Add nofitication SNS if needed\n",
    "    notification_config={\n",
    "        # \"SuccessTopic\": \"PUT YOUR SUCCESS SNS TOPIC ARN\",\n",
    "        # \"ErrorTopic\": \"PUT YOUR ERROR SNS TOPIC ARN\",\n",
    "    },  #  Notification configuration\n",
    ")\n",
    "\n",
    "env = {\"MODEL_SERVER_WORKERS\": \"2\"}\n",
    "\n",
    "# deploy the endpoint endpoint\n",
    "async_predictor = huggingface_model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=\"ml.g4dn.xlarge\",\n",
    "    async_inference_config=async_config,\n",
    "    endpoint_name=async_endpoint_name,\n",
    "    env=env,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ae7f39-a422-4daa-809d-c205e8161e5f",
   "metadata": {},
   "source": [
    "### Execute inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f00b83f4-5a98-4770-99ac-5ab8256559d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with a path to audio object in S3\n",
    "# Comment out the language line if you want to specify the input language. Otherwise it will detect it automatically\n",
    "\n",
    "\n",
    "data = {\n",
    "    \"s3_file\": \"REPLACE WITH A PATH TO AUDIO OBJECT IN S3\"\n",
    "    # \"language\": \"pl\"\n",
    "}\n",
    "\n",
    "res = async_predictor.predict_async(data=data)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a771eb5b-9178-4c5e-9d59-e93e0af51128",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since it is async inference, get_results is looking for the output_path\n",
    "# If the inference completed, you'll get the results from the output path. Otherwise, you'll get error that the output_path file doesn't exist\n",
    "res.get_result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d4d3d97-2872-4542-8118-6e038312eaa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "res.output_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32976984-376f-4911-a058-8feafcbb89df",
   "metadata": {},
   "source": [
    "### Setting up Autoscale asynchronous endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbddef99-8df3-4e46-9fed-b6ef310de90b",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = boto3.client(\n",
    "    \"application-autoscaling\"\n",
    ")  # Common class representing Application Auto Scaling for SageMaker amongst other services\n",
    "\n",
    "resource_id = (\n",
    "    \"endpoint/\" + async_endpoint_name + \"/variant/\" + \"AllTraffic\"\n",
    ")  # This is the format in which application autoscaling references the endpoint\n",
    "\n",
    "response = client.register_scalable_target(\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    ResourceId=resource_id,\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",\n",
    "    MinCapacity=1,  # async endpoint can scale in to 0 if setting the MinCapacity=0\n",
    "    MaxCapacity=5,\n",
    ")\n",
    "\n",
    "response = client.put_scaling_policy(\n",
    "    PolicyName=\"Invocations-ScalingPolicy\",\n",
    "    ServiceNamespace=\"sagemaker\",  # The namespace of the AWS service that provides the resource.\n",
    "    ResourceId=resource_id,  # Endpoint name\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",  # SageMaker supports only Instance Count\n",
    "    PolicyType=\"TargetTrackingScaling\",  # 'StepScaling'|'TargetTrackingScaling'\n",
    "    TargetTrackingScalingPolicyConfiguration={\n",
    "        \"TargetValue\": 5.0,  # The target value for the metric.\n",
    "        \"CustomizedMetricSpecification\": {\n",
    "            \"MetricName\": \"ApproximateBacklogSizePerInstance\",\n",
    "            \"Namespace\": \"AWS/SageMaker\",\n",
    "            \"Dimensions\": [{\"Name\": \"EndpointName\", \"Value\": async_endpoint_name}],\n",
    "            \"Statistic\": \"Average\",\n",
    "        },\n",
    "        \"ScaleInCooldown\": 300,  # ScaleInCooldown - The amount of time, in seconds, after a scale-in activity completes before another scale in activity can start.\n",
    "        \"ScaleOutCooldown\": 300  # ScaleOutCooldown - The amount of time, in seconds, after a scale-out activity completes before another scale out activity can start.\n",
    "        # 'DisableScaleIn': True|False - indicates whether scale in by the target tracking policy is disabled.\n",
    "        # If the value is true, scale-in is disabled and the target tracking policy won't remove capacity from the scalable resource.\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35cf51d3-47a3-49b2-a01c-3421a50605ba",
   "metadata": {},
   "source": [
    "## 4. Invoke Whisper on SageMaker Endpoint for Asynchronous inference\n",
    "In this section we will demonstrate invocation of an Asynchronous inference endpoint by using the Asynchronous endpoint deployed in section #3 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30b928f-5887-4a41-8411-022c7126d98d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "# Create a low-level client representing Amazon SageMaker Runtime\n",
    "# Update the relevant region\n",
    "sagemaker_runtime = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "# Specify the location of the input. Should be JSON with the input audion file (example in 02_deploy_whisper-Async.ipynb notebook)\n",
    "input_location = \"REPLACE WITH A PATH TO AUDIO OBJECT IN S3\"\n",
    "\n",
    "# The name of the endpoint. The name must be unique within an AWS Region in your AWS account.\n",
    "async_endpoint_name = async_endpoint_name\n",
    "\n",
    "# After you deploy a model using SageMaker hosting\n",
    "# services, your client applications use this API to get inferences\n",
    "# from the model hosted at the specified endpoint.\n",
    "response = sagemaker_runtime.invoke_endpoint_async(\n",
    "    EndpointName=async_endpoint_name,\n",
    "    # ContentType='audio/mpeg',\n",
    "    InputLocation=input_location,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2127c894-6afa-44ae-9a62-e79a5096b808",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View invocation response\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e2e458-7bf2-48d8-9e2e-041ea1409c32",
   "metadata": {},
   "source": [
    "#### Check Output Location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06d23609-1419-448d-98d4-c9b33c7dd00d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A function that waiting for the async response\n",
    "\n",
    "import urllib, time\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "\n",
    "def get_output(output_location):\n",
    "    output_url = urllib.parse.urlparse(output_location)\n",
    "    bucket = output_url.netloc\n",
    "    key = output_url.path[1:]\n",
    "    while True:\n",
    "        try:\n",
    "            return sm_session.read_s3_file(bucket=output_url.netloc, key_prefix=output_url.path[1:])\n",
    "        except ClientError as e:\n",
    "            if e.response[\"Error\"][\"Code\"] == \"NoSuchKey\":\n",
    "                print(\"waiting for output...\")\n",
    "                time.sleep(2)\n",
    "                continue\n",
    "            raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29d494c8-7cbe-4d78-a1c9-7f858062faed",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sm_session = sagemaker.session.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58160d12-029d-4f4b-aa2a-bf0beaca83a0",
   "metadata": {},
   "source": [
    "#### Get Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62ac17ed-9965-4abf-bd9e-dd1dac39fed2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "output = get_output(response[\"OutputLocation\"])\n",
    "result = json.loads(output)\n",
    "\n",
    "print(f\"Output: {output}\")\n",
    "print(\"transcription: \", result.get(\"transcription\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c58fb4e-f5de-4f76-b5ff-aa69a892152d",
   "metadata": {},
   "source": [
    "### Example for multiple invocations (can be used to test the autoscaling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b97ecf2f-e9ea-435d-b1c2-32c05a2de930",
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_runtime = boto3.client(\"sagemaker-runtime\", region_name=\"eu-west-1\")\n",
    "inferences = []\n",
    "for i in range(10):\n",
    "    response = sm_runtime.invoke_endpoint_async(\n",
    "        EndpointName=async_endpoint_name, InputLocation=input_location\n",
    "    )\n",
    "    output_location = response[\"OutputLocation\"]\n",
    "    inferences += [(input_location, output_location)]\n",
    "    time.sleep(0.5)\n",
    "\n",
    "for input_location, output_location in inferences:\n",
    "    output = get_output(output_location)\n",
    "    print(f\"Input File: {input_location}, Output: {output}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2c397a2-7c3d-4e71-bcd6-d15a89881c4f",
   "metadata": {},
   "source": [
    "## 5. Clean up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "495941d9-d4ad-44ba-a053-90d6258794d7",
   "metadata": {},
   "source": [
    "Remember to delete your endpoints after use as you will be charged for the instances used in this Demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17b78f4a-2b9b-4181-9c45-67c4f0461c15",
   "metadata": {},
   "outputs": [],
   "source": [
    "rl_predictor.delete_endpoint(rt_endpoint_name)\n",
    "async_predictor.delete_endpoint(async_endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b128850d",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/async-inference|Transcription_on_SM_endpoint.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/async-inference|Transcription_on_SM_endpoint.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}