label_text_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text analysis of issue labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, import all the necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pylab as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import nltk\n",
    "import re\n",
    "ps = nltk.PorterStemmer()\n",
    "from nltk.probability import FreqDist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in the data\n",
    "For this analysis, GitHub issue labels are not grouped by 'organization' or 'repository' instead, all 200+ issue labels are listed as a vectory under the 'issue_labels' column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "issue_data = pd.read_csv('data/whole_corpus.csv', header=0)\n",
    "issue_data.issue_labels = issue_data.issue_labels.astype(str)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>organization_or_project_name</th>\n",
       "      <th>issue_labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>whole_corpus</td>\n",
       "      <td>breaking-change,datamanager,dependencies,eml-p...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  organization_or_project_name  \\\n",
       "0                 whole_corpus   \n",
       "\n",
       "                                        issue_labels  \n",
       "0  breaking-change,datamanager,dependencies,eml-p...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "issue_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create and run the `tokenizer` function. This will take all of the issue labels, recognize that they are separated by commas, and then \"tokenize\" words found in the same GitHub repo into a list of terms that is much easier to analyze. You'll see the tokenized words under the `text_tokenized` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>organization_or_project_name</th>\n",
       "      <th>issue_labels</th>\n",
       "      <th>text_tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>whole_corpus</td>\n",
       "      <td>breaking-change,datamanager,dependencies,eml-p...</td>\n",
       "      <td>[breaking-change, datamanager, dependencies, e...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  organization_or_project_name  \\\n",
       "0                 whole_corpus   \n",
       "\n",
       "                                        issue_labels  \\\n",
       "0  breaking-change,datamanager,dependencies,eml-p...   \n",
       "\n",
       "                                      text_tokenized  \n",
       "0  [breaking-change, datamanager, dependencies, e...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    tokens = re.split(',', text)\n",
    "    return tokens\n",
    "\n",
    "issue_data['text_tokenized'] = issue_data['issue_labels'].apply(lambda x: tokenize(x.lower()))\n",
    "\n",
    "issue_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use the `PorterStemmer` function from the natural langue toolkit  and it's from the Natural Language Toolkit package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = nltk.PorterStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the function that will 'stem' the labels. Stemming  trims each of the words in our dataset down to a shortened or 'stemmed' version of that word to faciliate term grouping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>organization_or_project_name</th>\n",
       "      <th>issue_labels</th>\n",
       "      <th>text_tokenized</th>\n",
       "      <th>labels_stemmed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>whole_corpus</td>\n",
       "      <td>breaking-change,datamanager,dependencies,eml-p...</td>\n",
       "      <td>[breaking-change, datamanager, dependencies, e...</td>\n",
       "      <td>[breaking-chang, datamanag, depend, eml-pars, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  organization_or_project_name  \\\n",
       "0                 whole_corpus   \n",
       "\n",
       "                                        issue_labels  \\\n",
       "0  breaking-change,datamanager,dependencies,eml-p...   \n",
       "\n",
       "                                      text_tokenized  \\\n",
       "0  [breaking-change, datamanager, dependencies, e...   \n",
       "\n",
       "                                      labels_stemmed  \n",
       "0  [breaking-chang, datamanag, depend, eml-pars, ...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def stemming(tokenized_text):\n",
    "    text = [ps.stem(word) for word in tokenized_text]\n",
    "    return text\n",
    "\n",
    "issue_data['labels_stemmed'] = issue_data['text_tokenized'].apply(lambda x: stemming(x))\n",
    "\n",
    "issue_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, count the stemmed words. Note that some words (like variations on 'priority') are captured in a few different stems and it may require some further visual classification to group these words that appear in multiple stems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "counts_of_labels = pd.value_counts(issue_data.labels_stemmed.apply(pd.Series).stack())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Series.reorder_levels of high prior                                3\n",
       "type-defect                               3\n",
       "new class                                 3\n",
       "mix                                       2\n",
       "usabl                                     2\n",
       "low prior                                 2\n",
       "prioriti                                  2\n",
       "type-enhanc                               2\n",
       "depend                                    2\n",
       " docs - simple darwin cor                 2\n",
       "hierarchi                                 2\n",
       "annot                                     2\n",
       "in progress                               2\n",
       "type-review                               2\n",
       "typo                                      2\n",
       "portal requir                             1\n",
       "security-track                            1\n",
       "ua-src                                    1\n",
       "core standard                             1\n",
       "privacy-needs-resolut                     1\n",
       " docs - xml guid                          1\n",
       "web                                       1\n",
       "tag-needs-resolut                         1\n",
       "sdg ontology framework                    1\n",
       "food prepar                               1\n",
       "process - under public review             1\n",
       "defect                                    1\n",
       "tag-track                                 1\n",
       "minmark                                   1\n",
       "mim                                       1\n",
       "duration = day                            1\n",
       "task                                      1\n",
       "synonym                                   1\n",
       "format - owl                              1\n",
       "question - open                           1\n",
       "charter                                   1\n",
       "needs clarif                              1\n",
       "new term                                  1\n",
       "api                                       1\n",
       "datamanag                                 1\n",
       "thought                                   1\n",
       "ugdisagre                                 1\n",
       "privacy-track                             1\n",
       "roadmap                                   1\n",
       "delete/deprecate class                    1\n",
       "ugagre                                    1\n",
       "type-patch                                1\n",
       "question - document on dwc q&a            1\n",
       "sample_represent                          1\n",
       "mig                                       1\n",
       "possible domain conflict                  1\n",
       "process - prepare for executive review    1\n",
       "soil                                      1\n",
       "sdw community group item                  1\n",
       "ready for test                            1\n",
       "in editor's vers                          1\n",
       "duplicate class                           1\n",
       "cim                                       1\n",
       "unassign                                  1\n",
       "update document                           1\n",
       "gold/ebi-mgnifi                           1\n",
       "state                                     1\n",
       "i18n-track                                1\n",
       "google dataset                            1\n",
       "duration = fortnight                      1\n",
       "morpholog                                 1\n",
       "obc.id                                    1\n",
       "style                                     1\n",
       "nutrit                                    1\n",
       " docs - quick reference guid              1\n",
       "format - text                             1\n",
       "new term request                          1\n",
       "errata                                    1\n",
       "proposed decis                            1\n",
       "fair                                      1\n",
       "in editors' vers                          1\n",
       "mkdisagre                                 1\n",
       "duration = week                           1\n",
       "modify preferred label                    1\n",
       "mfdisagre                                 1\n",
       "modify definit                            1\n",
       "interesting discuss                       1\n",
       "govern                                    1\n",
       "general-mainten                           1\n",
       "process - in executive review             1\n",
       "sd goal                                   1\n",
       "patter                                    1\n",
       "interesting-discuss                       1\n",
       "axiomati                                  1\n",
       "document change propos                    1\n",
       "etl                                       1\n",
       "process - dismiss                         1\n",
       "community consult                         1\n",
       "controlled vocabulari                     1\n",
       "tech                                      1\n",
       "iri refactor                              1\n",
       "map                                       1\n",
       "i18n-needs-resolut                        1\n",
       "engineering issu                          1\n",
       "duration = hour                           1\n",
       "security-needs-resolut                    1\n",
       "import                                    1\n",
       "odm2cv                                    1\n",
       "priority-medium                           1\n",
       "termupd                                   1\n",
       "community of practic                      1\n",
       "accepted decis                            1\n",
       "ssn-ext                                   1\n",
       "best practic                              1\n",
       "leading practic                           1\n",
       "eml-pars                                  1\n",
       "bp                                        1\n",
       "cultivation and/or rear                   1\n",
       " docs - text guid                         1\n",
       "process- ready for public com             1\n",
       "3d                                        1\n",
       "docs - contribut                          1\n",
       "matter                                    1\n",
       "extens                                    1\n",
       "version                                   1\n",
       "refactor                                  1\n",
       " docs - namespace polici                  1\n",
       "cim 1.x                                   1\n",
       "fair rat                                  1\n",
       "weekly-digest                             1\n",
       "format - rdf                              1\n",
       "enhanc                                    1\n",
       "taxonomi                                  1\n",
       "urgent                                    1\n",
       " docs - rdf guid                          1\n",
       "issu                                      1\n",
       "rout                                      1\n",
       "guid                                      1\n",
       "bcodmo                                    1\n",
       "represent                                 1\n",
       "process - need evidence for demand        1\n",
       "auto-migr                                 1\n",
       "mapml                                     1\n",
       "edm2.1?                                   1\n",
       "pheno-workbench                           1\n",
       "type-task                                 1\n",
       "maintain                                  1\n",
       "definit                                   1\n",
       " format - cml                             1\n",
       "sweet                                     1\n",
       "wish list                                 1\n",
       "cityjson                                  1\n",
       "webvmt                                    1\n",
       "fixed in develop branch                   1\n",
       "backburn                                  1\n",
       "mkagre                                    1\n",
       "mfagre                                    1\n",
       "newterm                                   1\n",
       "cim 2.x                                   1\n",
       "process                                   1\n",
       "fix                                       1\n",
       "human act                                 1\n",
       "class available!                          1\n",
       "realm                                     1\n",
       "standards_map                             1\n",
       "bmdisagre                                 1\n",
       "eo-qb                                     1\n",
       "cmip6                                     1\n",
       "process chang                             1\n",
       "stat                                      1\n",
       "class released to address issu            1\n",
       "design feedback                           1\n",
       "executive approval pend                   1\n",
       "planet microb                             1\n",
       "ssn                                       1\n",
       "anatomi                                   1\n",
       "site-maintenan                            1\n",
       "term - chang                              1\n",
       "github problem                            1\n",
       "modify term                               1\n",
       "a11y-needs-resolut                        1\n",
       " documentation error                      1\n",
       "releas                                    1\n",
       "bmagre                                    1\n",
       "public com                                1\n",
       "term propos                               1\n",
       "priority - crit                           1\n",
       "a11y-track                                1\n",
       "phenomena                                 1\n",
       "fix definit                               1\n",
       "discuss                                   1\n",
       "priority - low                            1\n",
       "qb4st                                     1\n",
       "standard nam                              1\n",
       "pr submit                                 1\n",
       "review                                    1\n",
       "cryospher                                 1\n",
       "term - deprec                             1\n",
       "view                                      1\n",
       "meet                                      1\n",
       "auto-migrr                                1\n",
       "priorty-low                               1\n",
       "env_packag                                1\n",
       "curation_enhanc                           1\n",
       "breaking-chang                            1\n",
       "priority - medium                         1\n",
       "relat                                     1\n",
       "hacktoberfest                             1\n",
       "process-impl                              1\n",
       "question - answ                           1\n",
       "priority - high                           1\n",
       "feature request                           1\n",
       "propos                                    1\n",
       "dtype: int64>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_rows', None)\n",
    "counts_of_labels.reorder_levels"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sysrevenv",
   "language": "python",
   "name": "sysrevenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}