dataset_overview.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "data_path = \"./data/tom_in_amc.json\"\n",
    "with open(data_path) as f:\n",
    "    dataset = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Split:           \tdict_keys(['train', 'dev', 'test'])\n",
      "Example Movies (Dev Set):\t['constantine', 'nightbreed', 'the martian']\n"
     ]
    }
   ],
   "source": [
    "print(f\"Dataset Split:           \\t{dataset.keys()}\")\n",
    "print(f\"Example Movies (Dev Set):\\t{list(dataset['train'].keys())[:3]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take the first movie in the training set as an example movie data:\n",
    "- `movie_data[\"chars\"]`: character names and the their frequency of utterances throughout the movie\n",
    "- `movie_data[\"training_scenes\"]`: training scenes in the movie\n",
    "- `movie_data[\"testing_scenes\"]`: testing scenes in the movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie Data Components:\tdict_keys(['chars', 'training_scenes', 'testing_scenes'])\n",
      "Characters:           \t{'john': 40, 'chaz': 7, 'angela': 20}\n",
      "# Training Scenes:    \t49\n",
      "# Testing Scenes:     \t31\n"
     ]
    }
   ],
   "source": [
    "movie_data = dataset[\"train\"][\"constantine\"]\n",
    "print(f\"Movie Data Components:\\t{movie_data.keys()}\")\n",
    "print(f\"Characters:           \\t{movie_data['chars']}\")\n",
    "print(f\"# Training Scenes:    \\t{len(movie_data['training_scenes'])}\")\n",
    "print(f\"# Testing Scenes:     \\t{len(movie_data['testing_scenes'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take the first training scene as an example scene data:\n",
    "- `scene_data[\"scene\"]`: scene content composed by lines\n",
    "- `scene_data[\"chars\"]`: characters in the scene\n",
    "- `scene_data[\"char_map\"]`: character map from character name to anonymous ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scene Data Components:         \tdict_keys(['scene', 'chars', 'char_map'])\n",
      "The First Line of the Scene:   \t{'type': 'scene', 'title': 'INT. APARTMENT 7B', 'text': 'One scan of the situation is all it takes. The bed -- the child -- the panicked priest -- who rushes to P0.'}\n",
      "Characters in the Scene:       \t{'john': 1, 'chaz': 1}\n",
      "Character Name to Anonymous ID:\t{'john': 'P0', 'chaz': 'P1'}\n"
     ]
    }
   ],
   "source": [
    "scene_data = movie_data[\"training_scenes\"][0]\n",
    "print(f\"Scene Data Components:         \\t{scene_data.keys()}\")\n",
    "print(f\"The First Line of the Scene:   \\t{scene_data['scene'][0]}\")\n",
    "print(f\"Characters in the Scene:       \\t{scene_data['chars']}\")\n",
    "print(f\"Character Name to Anonymous ID:\\t{scene_data['char_map']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lines can be processed to text-form scene content by the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INT. APARTMENT 7B\n",
      "One scan of the situation is all it takes. The bed -- the child -- the panicked priest -- who rushes to P0.\n",
      "HENNESSEY: (whispering) Thank God you're here... \n",
      "P0 shoots him a disgusted look. Hennessey gives him a wide berth. P0 walks past the panic-stricken MOTHER without a glance, sets his cigarette on the nightstand, the glowing tip drooped over the edge. He puts a gloved hand to the child's face and it burns on contact. His demeanor instantly changes as he leans right next to the ear of the little girl and whispers --\n",
      "JOHN: This is Constantine. John Constantine, asshole. \n",
      "The girl JOLTS, bandages on her arms cut into her skin. Eyes snap open -- glare right through him.\n",
      "JOHN: How ya doing? \n",
      "JEANIE: Vamos juntos a matarla. \n",
      "P0 whips out a key chain crammed with medallions.\n",
      "JOHN: Let's see who we got here... \n",
      "He holds them up so they cast shadows across Jeanie's face. He flips through each of these sculptured SAINTS until the child suddenly reacts to one -- tries to look away. P0 stares at the Saint responsible, seems genuinely puzzled. He turns to the Mother -- very serious.\n",
      "JOHN: I need a mirror. Now. \n",
      "Shaken, the Mother produces a small compact. P0 pushes it away, turns to the doorway crowd.\n",
      "JOHN: A large mirror. At least -- (sizes up the child) -- three feet high. \n",
      "CONTINUED:\n",
      "The hallway crowd discusses possibilities.\n",
      "MOVE.\n",
      "Tenants run to retrieve a mirror. Hennessey looks worried. P0 grabs the bed -- shoves Hennessey aside as he swings the bed around, legs gouging hardwood all the way. Faces it toward the window.\n",
      "INTERCUT - APARTMENT 5G BELOW\n",
      "Several of the MEN from the hallway crowd storm through an OLD WOMAN'S stuffy apartment, locate a huge iron- framed mirror and rip it and half the plaster from the wall.\n",
      "P0\n",
      "centers on a drapery ROPE, pulls hard. Drapery HOOKS EXPLODE off the curtain rod as he swings open the window, looks down into the alley where that yellow taxi is parked.\n",
      "JOHN: Chaz? \n",
      "P1 (20s) looks up from his cab. Cool. Frustrated. You just know he'd love to be up there, not down here.\n",
      "JOHN: Move the car. \n",
      "CHAZ: Why? \n",
      "MOVE THE DAMN CAR.\n",
      "P0 frees the drapery rope, ducks back into the room.\n",
      "CHAZ: Park the car -- Move the car... \n",
      "P1 shifts into reverse, moves the car three feet.\n",
      "CHAZ: There, I moved the damn car. \n"
     ]
    }
   ],
   "source": [
    "def process_lines(lines):\n",
    "    scene_content = []\n",
    "    for line in lines:\n",
    "        type = line[\"type\"]\n",
    "        title = line[\"title\"]\n",
    "        content = line[\"text\"]\n",
    "        \n",
    "        assert type in (\"scene\", \"dialog\"), f\"Unknown type: {type}\"\n",
    "        if type == \"scene\":\n",
    "            if title != \"NULL\":\n",
    "                scene_content.append(title)\n",
    "            scene_content.append(content)\n",
    "        elif type == \"dialog\":\n",
    "            scene_content.append(f\"{title}: {content}\")\n",
    "\n",
    "    return \"\\n\".join(scene_content)\n",
    "\n",
    "scene_content = process_lines(scene_data[\"scene\"])\n",
    "print(scene_content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}