notebooks/02-Download 45,000 Adoptable Cat Images with petpy and multiprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combining the [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) package for concurrent use of multiple CPUs and the [petpy](https://github.com/aschleg/petpy) package for interacting with the [Petfinder API](https://www.petfinder.com/developers/api-docs) allows one to find and download a vast amount of animal images for use in other tasks, such as image classification. \n",
    "\n",
    "This post will introduce how to use the multiprocessing and petpy packages to quickly and easily download a large set of cat images of all the different breeds available in the Petfinder database. We will end up with a collection of just under 45,000 of cat images sorted by user-defined breed classifications. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by importing the various packages and modules that will be needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import petpy\n",
    "import os\n",
    "import pandas as pd\n",
    "import urllib.request\n",
    "import urllib.error\n",
    "from multiprocessing import Pool\n",
    "from multiprocessing.dummy import Pool as ThreadPool"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the available cat breeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a connection to the Petfinder API by calling the `Petfinder` class from the petpy package with your given API key. You can receive an API key by creating an account on the [Petfinder developer page](https://www.petfinder.com/developers/api-key)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "key = os.getenv('PETFINDER_KEY')\n",
    "pf = petpy.Petfinder(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `breed_list()` method to get the available cat breeds in the Petfinder database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_breeds = pf.breed_list('cat', return_df=True)['cat breeds'].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting cat breed records with multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To speed up the process of extracting the pet records in the Petfinder database for each breed, we will utilize the [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) library to spread out the task across multiple cores.\n",
    "\n",
    "The machine that I am working with has a quad-core CPU, thus I set the maximum amount of processes that can be running at one time to be twice that amount. It is generally not recommended to exceed the device's maximum number of cores by too much (I've heard more than double the amount of cores for longer tasks is the recommended max), as the process can slow down if the program has to switch between processes more than is required. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pool = ThreadPool(processes=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To leverage the concurrency provided by `multiprocessing`, we first define a worker function that wraps the `pet_find()` method. We will pull up to 1,500 pet records for each individual breed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_cat_breeds(cat):\n",
    "    breeds = pf.pet_find('US', animal='cat', breed=cat, count=500, pages=3, return_df=True)\n",
    "    return(breeds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the worker function and the pool initialized, we can begin extracting the pet records using the `pet_find()` method in `petpy` concurrently. We also time the duration of the process using the `%%time` magic function available in Jupyter Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time # use Jupyter Notebook time magic function for recording how long it takes to accumulate the results.\n",
    "cats = pool.map(get_cat_breeds, cat_breeds) \n",
    "pool.close()\n",
    "pool.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire process took just under 5 and a half minutes, likely due to the time taken to convert the JSON results from the API into pandas DataFrames. The completed pool process returns the collected results as a list, which we can convert to a DataFrame with `concat()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cats = pd.concat(cats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process compiled 32,000 individual pet records that matched the breeds we were looking for in the `cat_breeds` list. Admittedly, the prcoess of coercing the JSON results from the Petfinder API into pandas DataFrames could likely be much more efficient; however, we were able to find just under 32,000 adoptable cat records from animal shelters across the United States and return the results in a clean and tidy DataFrame, so this seems like an acceptable trade-off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31858"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning the data to get the image URLs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are only interested in the associated photo images for each cat, we can clean the data set we extracted to reshape and remove the data that is not needed for the task at hand.\n",
    "\n",
    "The following helper function is used to clean and reshape the data we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_images(df):\n",
    "    try:\n",
    "        del df['media.photos.photo'] # This column may need to be deleted manually.\n",
    "    except:\n",
    "        pass\n",
    "    \n",
    "    # Keep only the columns that contain 'id', 'breed', and 'photo'\n",
    "    photos = df[df.columns[df.columns.str.contains('id|breed|photo')]]\n",
    "    # Melt the data to reshape it from wide to long and remove any NAs introduced from empty photo records.\n",
    "    photos_melted = pd.melt(photos, id_vars=['id', 'breed0', 'breed1'])\n",
    "    photos_melted.dropna(subset=['value'], inplace=True)\n",
    "    del photos_melted['variable']\n",
    "    \n",
    "    # The Petfinder API gives two fields for breed, thus we want to split these into individual data sets \n",
    "    breeds1 = photos_melted.loc[:,['id', 'breed0', 'value']]\n",
    "    breeds2 = photos_melted.loc[:,['id', 'breed1', 'value']]\n",
    "    breeds2.dropna(inplace=True)\n",
    "    \n",
    "    # The columns of each breed dataset are renamed and appended with the index column deleted.\n",
    "    breeds1.rename(columns={'breed0':'breed', 'value':'img'}, inplace=True)\n",
    "    breeds2.rename(columns={'breed1':'breed', 'value':'img'}, inplace=True)\n",
    "    \n",
    "    breed_photos = breeds1.append(breeds2).drop_duplicates().reset_index()\n",
    "    del breed_photos['index']\n",
    "    \n",
    "    return breed_photos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cat_breed_images = get_images(cats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting DataFrame that is output from our helper function contains only the data that is of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>breed</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40181161</td>\n",
       "      <td>Abyssinian</td>\n",
       "      <td>http://photos.petfinder.com/photos/pets/401811...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40181148</td>\n",
       "      <td>Abyssinian</td>\n",
       "      <td>http://photos.petfinder.com/photos/pets/401811...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38018075</td>\n",
       "      <td>Abyssinian</td>\n",
       "      <td>http://photos.petfinder.com/photos/pets/380180...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>38017865</td>\n",
       "      <td>Domestic Short Hair</td>\n",
       "      <td>http://photos.petfinder.com/photos/pets/380178...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38017999</td>\n",
       "      <td>Abyssinian</td>\n",
       "      <td>http://photos.petfinder.com/photos/pets/380179...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                breed  \\\n",
       "0  40181161           Abyssinian   \n",
       "1  40181148           Abyssinian   \n",
       "2  38018075           Abyssinian   \n",
       "3  38017865  Domestic Short Hair   \n",
       "4  38017999           Abyssinian   \n",
       "\n",
       "                                                 img  \n",
       "0  http://photos.petfinder.com/photos/pets/401811...  \n",
       "1  http://photos.petfinder.com/photos/pets/401811...  \n",
       "2  http://photos.petfinder.com/photos/pets/380180...  \n",
       "3  http://photos.petfinder.com/photos/pets/380178...  \n",
       "4  http://photos.petfinder.com/photos/pets/380179...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_breed_images.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cleaning of the data left us with just over 20,000 unique cat records from the Petfinder database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20394"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cat_breed_images['id'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Petfinder API provides several different sizes of each uploaded image associated to a record for thumbnails, search results and individual pet profiles. We want to extract the size information from each image URL to filter the larger images that will be used for future tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_breed_images['image_width'] = cat_breed_images['img'].str.split('width=', 1).str[1].str.split('&', 0).str[0].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a quick count of each unique image width using the `value_counts()` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500    71959\n",
       "95     71959\n",
       "60     71959\n",
       "50     71959\n",
       "300    71959\n",
       "Name: image_width, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_breed_images['image_width'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As there are the same number of images for each size, we can go ahead and filter the data set to keep only the images that are listed with a 500 pixel width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_images_largest = cat_breed_images.groupby('id').apply(lambda x: x[x['image_width'] == 500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove some of the added columns resulting from the groupby and apply operations as well as reset the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "del cat_images_largest['id']\n",
    "cat_images_largest.reset_index(inplace=True)\n",
    "del cat_images_largest['level_1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, replace space and '/' characters with an underscore or space, respectively, to help clean up the breed names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_images_largest['breed'] = cat_images_largest['breed'].str.replace(' ', '_')\n",
    "cat_images_largest['breed'] = cat_images_largest['breed'].str.replace('/', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the `drop_duplicates()` method to the DataFrame to remove duplicate pet records and images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "breed_images = cat_images_largest.drop_duplicates(subset=['img', 'id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our cat image DataFrame is now reshaped into the format we need with only unique pet records, thus we should be almost ready to begin downloading the images. First, we can get a quick count of the number of breed images we have for each respective breed by using the `value_counts()` method as we did earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Domestic_Short_Hair                     9658\n",
       "Domestic_Medium_Hair                    3631\n",
       "Tabby                                   3630\n",
       "Domestic_Long_Hair                      3441\n",
       "American_Shorthair                      2935\n",
       "Calico                                  2911\n",
       "Siamese                                 2867\n",
       "Tortoiseshell                           2804\n",
       "Tuxedo                                  2538\n",
       "Maine_Coon                              1930\n",
       "Russian_Blue                            1438\n",
       "Tiger                                   1354\n",
       "Torbie                                  1259\n",
       "Dilute_Calico                           1211\n",
       "Dilute_Tortoiseshell                    1142\n",
       "Bombay                                  1135\n",
       "Manx                                     687\n",
       "Bengal                                   636\n",
       "Extra-Toes_Cat__Hemingway_Polydactyl     494\n",
       "Turkish_Van                              470\n",
       "Persian                                  407\n",
       "Snowshoe                                 383\n",
       "Bobtail                                  338\n",
       "Abyssinian                               324\n",
       "Ragdoll                                  243\n",
       "Oriental_Short_Hair                      224\n",
       "Himalayan                                186\n",
       "Turkish_Angora                           181\n",
       "British_Shorthair                        146\n",
       "Egyptian_Mau                             133\n",
       "                                        ... \n",
       "American_Curl                             46\n",
       "Balinese                                  40\n",
       "Nebelung                                  40\n",
       "Oriental_Tabby                            39\n",
       "Birman                                    36\n",
       "Selkirk_Rex                               26\n",
       "Ocicat                                    23\n",
       "Scottish_Fold                             23\n",
       "Tonkinese                                 22\n",
       "Siberian                                  20\n",
       "Chausie                                   19\n",
       "Munchkin                                  17\n",
       "Chartreux                                 14\n",
       "Japanese_Bobtail                          14\n",
       "Pixie-Bob                                 14\n",
       "Applehead_Siamese                         12\n",
       "Ragamuffin                                11\n",
       "Cornish_Rex                                9\n",
       "Devon_Rex                                  9\n",
       "Cymric                                     8\n",
       "American_Wirehair                          7\n",
       "Korat                                      7\n",
       "LaPerm                                     6\n",
       "Somali                                     6\n",
       "Javanese                                   6\n",
       "Burmilla                                   5\n",
       "Chinchilla                                 5\n",
       "Oriental_Long_Hair                         4\n",
       "Singapura                                  3\n",
       "Sphynx__Hairless_Cat                       1\n",
       "Name: breed, Length: 65, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "breed_images['breed'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we would expect, more common breeds such as the Domestic Short Hair, Medium Hair and Long Hair have the most images available. Although the Petfinder API lists the Tuxedo, Calico, and Tabby as breeds, they are actually just colorings and not genetically distinctive to be considered individual 'breeds'. As the API is user-input from shelters and organizations listing cats for adoption, this is to be expected. I decided to keep those images rather than filter them out. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the cat images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before downloading the images, we first need to coerce our results that are stored in a DataFrame into a list of lists for us to take advantage of the `multiprocessing` module. First, remove all but the first 5,000 images for each breed, which for our current dataset will only cut a few thousand images for the Domestic Short Hair breed. 5,000 is an admittedly arbitrary number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "breed_images_5000 = breed_images.groupby('breed').head(5000).reset_index()\n",
    "del breed_images_5000['index']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then take the columns of the DataFrame we need and convert them each to a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urls, breed, index = breed_images_5000['img'].tolist(), breed_images_5000['breed'].tolist(), breed_images_5000.index.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "breed_list = [index, breed, urls]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of now, our list is just a list of three lists containing the information we need. We must rearrange the list of lists to be in a format that allows us to easily input the values into the `Pool` process as it iterates through the values. Therefore, we create a new list and iterate through the `breed_list` collection and append the values of each list into the newly created list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "breed_list_new = []\n",
    "for i in range(0, len(breed_list[0])):\n",
    "    breed_list_new.append([breed_list[0][i], breed_list[1][i], breed_list[2][i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44987"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(breed_list_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see we have just under 45,000 images with URLs compiled in the new list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the images organized after downloading, we first create individual directories in the main directory where we will store the downloaded images. To do this, use the `unique()` attribute of a pandas Series and convert it to a list, as so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "breed_dirs = list(breed_images_5000['breed'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then create individual directories for the breed images by iterating through the list and using the `makedirs()` function in the [`os`](https://docs.python.org/3/library/os.html) package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in breed_dirs:\n",
    "    os.makedirs('cat_breeds/' + str(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the directories created, we can proceed to writing the worker function that wil be used to download the images in the `Pool` process as we did previously when compiling the pet record results. The downloaded image name will have a naming convention of BREEDNAME_INDEX. We also make sure to write an error exception with [`urllib`](https://docs.python.org/3/library/urllib.html) and the [`HTTPError`](https://docs.python.org/3/library/urllib.error.html#module-urllib.error) for grabbing the images from the URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def download_breed_images(breed_img):\n",
    "    try:\n",
    "        urllib.request.urlretrieve(breed_img[2], \n",
    "                                   os.path.join('cat_breeds/', \n",
    "                                                str(breed_img[1]), str(breed_img[1]) + str(breed_img[0]) + '.jpg'))\n",
    "    except urllib.error.HTTPError as err:\n",
    "        print(err.code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the process is I/O bound, in that all we are doing is calling a URL and downloading the stored image, we can increase the number of processes as each iteration should be quick. I chose a value of 5x the number of cores available on my machine, again an arbitrary choice that may or may not be the most efficient =). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pool = ThreadPool(processes=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to download the images to our machine! As before, we start the pool by using the `map()` method and track the amount of time the process takes to download the images. Any HTTPErrors that arise will also print with the error code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "415\n",
      "415\n",
      "Wall time: 6min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pool.map(download_breed_images, breed_list_new)\n",
    "pool.close()\n",
    "pool.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire process to download just under 45,000 images took about 6 and a half minutes and only had two HTTP errors! The images will be stored in a separate directory *cat_breeds* with subdirectories containing the respective breed images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope this post served as a fun and useful introduction to what is possible with the `multiprocessing` module and the `petpy` library. Please note that as the Petfinder API is a public API with users around the country inputting data, as well as records continually being added and removed as pets move through the shelter system to adoption that the results obtained above will likely be different when performed at different times.\n",
    "\n",
    "The images that were downloaded during this exercise can also be downloaded as a `tar.gz` file using the following [Dropbox link](https://www.dropbox.com/s/kzoffohjrpdxxjw/cat_breeds.tar.gz?dl=0) (warning: the file is about 1.5GB)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}