wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Report: Data Wrangling\n",
    "\n",
    "This report describes my data wrangling efforts for WeRateDogs' Twitter data.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gathering Data\n",
    "\n",
    "For this project, the gathering data phase involved three datasets: \n",
    "\n",
    "1. `archive` table: contains the information WeRateDogs sent Udacity and I downloaded directly.\n",
    "\n",
    "2. `fav_rt` table: contains the information I gathered from Tweepy. Namely, tweet ID, favorite count and retweet count. \n",
    "\n",
    "3. `image_prediction`table: WeRateDog's tweets were processed through a neural network that can identify dog breed. This dataset was downloaded programmatically.\n",
    "\n",
    "## Assessing Data\n",
    "\n",
    "After gathering data from multiple methods, I moved on to assessing and cleaning data.\n",
    "\n",
    "During the visual assessment, I checked every dataset for possible quality and tidiness issues. During the programmatic assessment, I used specific methods and functions to reveal something about the data's quality and tidiness, such as `.info()`, `notna()`, `.head()`, `.tail()`, `.sample()`, among others. \n",
    "\n",
    "I ended up with the following list of issues:\n",
    "\n",
    "### Compiled list of issues\n",
    "\n",
    "### Quality:\n",
    "\n",
    "##### `archive` table:\n",
    "1. `source` column includes the HTML tag\n",
    "2. Erroneus datatypes (`timestamp`, `in_reply_to_status_id`, `in_reply_to_user_id`, `retweeted_status_user_id`, `retweeted_status_id`, `retweeted_status_timestamp`)\n",
    "3. `name` column has 109 rows with invalid, lower case words\n",
    "4. `name` column has 705 rows in which a dog name wasn't listed but instead of a null value, there is a None string as an entry instead\n",
    "5. In the `text` column, because of the string quotation marks, the url link breaks if one tries to click on it\n",
    "6. `doggo`, `pupper`, `puppo`, `floofer`: columns have rows in which a dog stage wasn't listed but instead of a null value, there is a None string as an entry instead\n",
    "7. `doggo`, `pupper`, `puppo`, `floofer`: erroneous data type, string instead of bool\n",
    "8. Retweets should be dropped\n",
    "\n",
    "##### `image_prediction` table:\n",
    "\n",
    "9. `p1`, `p2`, `p3`: dog breeds are separated by '_' instead of spaces ' '\n",
    "10. `p1`, `p2`, `p3`: some dog breed are not capitalized\n",
    "11. The neural network apparently identified that some images are not from dogs\n",
    "\n",
    "### Tidiness:\n",
    "1. The dataset could be tidily represented with the columns `favorite_count` and `retweet_count` in the `archive` table\n",
    "2. Four variables in one column in the `archive` table (`dog_stage`, i.e., doggo, floofer, pupper, and puppo)\n",
    "3. The dataset could be tidily represented joining the `image_prediction` table into the `archive` table to create one master dataset\n",
    "\n",
    "## Cleaning Data\n",
    "\n",
    "> I started cleaning from the first two tidiness issues. \n",
    "\n",
    "#### Tidiness Issue #1 \n",
    "\n",
    "I used [`df.merge`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html) to merge the tables `archive` and `fav_rt`. \n",
    "\n",
    "#### Tidiness Issue #2\n",
    "\n",
    "I created a function to extract the dog stage from each of the four dog stage columns (i.e., `doggo`, `puppo`, `pupper`, and `floofer`). Then, I use [`df.apply`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) to apply the function and create the `dog_stage` column. \n",
    "\n",
    "> Note: I couldn't make `pd.melt` work, since there are many rows without any classification for dog stage at all. Dropping them would have meant losing a huge chunk of the dataset that is perfectly valid for other analises. Instead, I found help at [this discussion](https://knowledge.udacity.com/questions/80210) from the Udacity forum. \n",
    "\n",
    "> At this point, I moved on to the quality issues. I could have acted on tidiness issue #3 at this point, but I would have ended up with a huge dataset before I could safely drop any columns.\n",
    "\n",
    "#### Quality Issue #1:\n",
    "\n",
    "I used a regular expression to leave just the text for the HTML tag in the `source` column.\n",
    "\n",
    "#### Quality Issue #2:\n",
    "\n",
    "I dealt with various columns that were in erroneous datatypes using [`pd.datetime`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html) and [`pd.astype`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html).\n",
    "\n",
    "#### Quality Issue #3:\n",
    "\n",
    "I found the rows in which invalid, lowercase words were used in the `name` column. Then, I changed those to pd.NA. \n",
    "\n",
    "#### Quality Issue #4: \n",
    "\n",
    "First, I found the rows in which a dog name wasn't listed but instead of a null value, there is a `None` string as an entry instead. Then, I changed those to pd.NA.\n",
    "\n",
    "#### Quality Issue #5:\n",
    "\n",
    "For every entry in the `text` column, I added a trailing space. This way, if you click on the link, it won't assume the quotation mark belongs to the url and it'll open just fine. \n",
    "\n",
    "#### Quality Issues #6 and #7:\n",
    "\n",
    "While dealing with tidiness issue #2, these columns were repurposed as a single `dog_stage` column and the null values were already converted to `pd.NA`. Therefore, both of these issues were already addressed.\n",
    "\n",
    "#### Quality Issue #8:\n",
    "\n",
    "First, I used [`pd.index`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.index.html) to get the indices for the rows representing retweets. Then, I dropped those rows. Finally, since those columns weren't useful anymore, I dropped them.\n",
    "\n",
    "#### Quality Issues #9 and #10:\n",
    "\n",
    "I used [`str.replace`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.replace.html) and [`str.title`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.title.html#pandas.Series.str.title) to replace the underscores ('_') with spaces (' ') and converted the first character of each word to uppercase, respectively. \n",
    "\n",
    "#### Quality Issue #11:\n",
    "\n",
    "First, I used [`pd.index`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.index.html) to get the indices for the rows in which none of the predictions (`p1`, `p2`, `p3`) seemed to come from dogs. Then, I dropped those rows.\n",
    "\n",
    "> At this point, the list of issues was exhausted and I went back to tidiness issue #3. \n",
    "\n",
    "#### Tidiness Issue #3:\n",
    "\n",
    "I used [`df.merge`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html) to merge the tables `archive_copy` and `image_prediction_copy`. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}