corpus_stats/frequency_stats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PELIC frequency statistics\n",
    "\n",
    "<br>\n",
    "\n",
    "**Authors:** Cassie Maz, John Starr, Ben Naismith  \n",
    "**Date:** 9 June 2020\n",
    "\n",
    "<br>\n",
    "\n",
    "The code in this notebook calculates the frequency of tokens (words and lemmas) in PELIC. The data come from [PELIC_compiled.csv](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/blob/master/PELIC_compiled.csv), resulting in frequency distributions of tokens based on students' L1 and Level. \n",
    "\n",
    "<br>\n",
    "\n",
    "**Notebook contents:**\n",
    "- [Initial setup](#Initial-setup)\n",
    "- [Total frequencies](#Total-frequencies)\n",
    "    * The output is a csv with a column for total frequency count and a column for frequency per million.\n",
    "- [Frequencies by level](#Frequencies-by-level): 2, 3, 4, 5\n",
    "    * Note: The majority of students in PELIC are in levels 3, 4, and 5.\n",
    "    * The output is an NLTK Frequency Distribution for each level, exported as a pickle file.\n",
    "- [Frequencies by L1](#Frequencies-by-L1) Arabic, Chinese, Korean, Japanese, Spanish\n",
    "    * Note: we are not creating frequency dictionaries for all L1's, but only the 5 most common.\n",
    "    * The output is an NLTK Frequency Distribution for each L1, exported as a pickle file.\n",
    "- [Conditional frequency distributions](#Conditional-frequency-distributions)\n",
    "    * There are two conditional frequency distributions, for level and L1, to allow access to word frequencies based on either Level or L1 using only one pickle file, rather than separate pickle files.\n",
    "- [Demonstration](#Demonstration)\n",
    "    * A short demonstration comparing the top 20 Arabic and Japanese lemmas\n",
    "    \n",
    "**Note:** Distributions do not take capitalization into account – a capitalized word and the same non-capitalized word will go towards the same count."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import pandas as pd\n",
    "import pickle as pkl\n",
    "from nltk import FreqDist\n",
    "from nltk.probability import ConditionalFreqDist\n",
    "import random\n",
    "from ast import literal_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 46230 entries, 1 to 48420\n",
      "Data columns (total 11 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   anon_id      46230 non-null  object\n",
      " 1   L1           46230 non-null  object\n",
      " 2   gender       46230 non-null  object\n",
      " 3   level_id     46230 non-null  int64 \n",
      " 4   class_id     46230 non-null  object\n",
      " 5   question_id  46230 non-null  int64 \n",
      " 6   version      46230 non-null  int64 \n",
      " 7   text_len     46230 non-null  int64 \n",
      " 8   text         46230 non-null  object\n",
      " 9   tokens       46230 non-null  object\n",
      " 10  tok_lem_POS  46230 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 4.2+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eq0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>177</td>\n",
       "      <td>I met my friend Nife while I was studying in a...</td>\n",
       "      <td>['I', 'met', 'my', 'friend', 'Nife', 'while', ...</td>\n",
       "      <td>(('I', 'i', 'PRP'), ('met', 'meet', 'VBD'), ('...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>am8</td>\n",
       "      <td>Thai</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>137</td>\n",
       "      <td>Ten years ago, I met a women on the train betw...</td>\n",
       "      <td>['Ten', 'years', 'ago', ',', 'I', 'met', 'a', ...</td>\n",
       "      <td>(('Ten', 'ten', 'CD'), ('years', 'year', 'NNS'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>64</td>\n",
       "      <td>In my country we usually don't use tea bags. F...</td>\n",
       "      <td>['In', 'my', 'country', 'we', 'usually', 'do',...</td>\n",
       "      <td>(('In', 'in', 'IN'), ('my', 'my', 'PRP$'), ('c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>I organized the instructions by time.</td>\n",
       "      <td>['I', 'organized', 'the', 'instructions', 'by'...</td>\n",
       "      <td>(('I', 'i', 'PRP'), ('organized', 'organize', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>59</td>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>['First', ',', 'prepare', 'a', 'port', ',', 'l...</td>\n",
       "      <td>(('First', 'first', 'RB'), (',', ',', ','), ('...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id       L1  gender  level_id class_id  question_id  version  \\\n",
       "answer_id                                                                     \n",
       "1             eq0   Arabic    Male         4        g            5        1   \n",
       "2             am8     Thai  Female         4        g            5        1   \n",
       "3             dk5  Turkish  Female         4        w           12        1   \n",
       "4             dk5  Turkish  Female         4        w           13        1   \n",
       "5             ad1   Korean  Female         4        w           12        1   \n",
       "\n",
       "           text_len                                               text  \\\n",
       "answer_id                                                                \n",
       "1               177  I met my friend Nife while I was studying in a...   \n",
       "2               137  Ten years ago, I met a women on the train betw...   \n",
       "3                64  In my country we usually don't use tea bags. F...   \n",
       "4                 6              I organized the instructions by time.   \n",
       "5                59  First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          ['I', 'met', 'my', 'friend', 'Nife', 'while', ...   \n",
       "2          ['Ten', 'years', 'ago', ',', 'I', 'met', 'a', ...   \n",
       "3          ['In', 'my', 'country', 'we', 'usually', 'do',...   \n",
       "4          ['I', 'organized', 'the', 'instructions', 'by'...   \n",
       "5          ['First', ',', 'prepare', 'a', 'port', ',', 'l...   \n",
       "\n",
       "                                                 tok_lem_POS  \n",
       "answer_id                                                     \n",
       "1          (('I', 'i', 'PRP'), ('met', 'meet', 'VBD'), ('...  \n",
       "2          (('Ten', 'ten', 'CD'), ('years', 'year', 'NNS'...  \n",
       "3          (('In', 'in', 'IN'), ('my', 'my', 'PRP$'), ('c...  \n",
       "4          (('I', 'i', 'PRP'), ('organized', 'organize', ...  \n",
       "5          (('First', 'first', 'RB'), (',', ',', ','), ('...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in PELIC_compiled which contains all the processed texts\n",
    "\n",
    "pelic_df = pd.read_csv('../pelic_compiled.csv', index_col = 'answer_id')\n",
    "pelic_df.info()\n",
    "pelic_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because lists are read in as strings, these need to be converted back to lists\n",
    "\n",
    "pelic_df.tokens = pelic_df.tokens.apply(literal_eval)\n",
    "pelic_df.tok_lem_POS = pelic_df.tok_lem_POS.apply(literal_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eq0</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>[I, met, my, friend, Nife, while, I, was, stud...</td>\n",
       "      <td>((I, i, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>am8</td>\n",
       "      <td>4</td>\n",
       "      <td>Thai</td>\n",
       "      <td>[Ten, years, ago, ,, I, met, a, women, on, the...</td>\n",
       "      <td>((Ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dk5</td>\n",
       "      <td>4</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>[In, my, country, we, usually, do, n't, use, t...</td>\n",
       "      <td>((In, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dk5</td>\n",
       "      <td>4</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>[I, organized, the, instructions, by, time, .]</td>\n",
       "      <td>((I, i, PRP), (organized, organize, VBD), (the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad1</td>\n",
       "      <td>4</td>\n",
       "      <td>Korean</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>((First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id  level_id       L1  \\\n",
       "answer_id                              \n",
       "1             eq0         4   Arabic   \n",
       "2             am8         4     Thai   \n",
       "3             dk5         4  Turkish   \n",
       "4             dk5         4  Turkish   \n",
       "5             ad1         4   Korean   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          [I, met, my, friend, Nife, while, I, was, stud...   \n",
       "2          [Ten, years, ago, ,, I, met, a, women, on, the...   \n",
       "3          [In, my, country, we, usually, do, n't, use, t...   \n",
       "4             [I, organized, the, instructions, by, time, .]   \n",
       "5          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "\n",
       "                                                 tok_lem_POS  \n",
       "answer_id                                                     \n",
       "1          ((I, i, PRP), (met, meet, VBD), (my, my, PRP$)...  \n",
       "2          ((Ten, ten, CD), (years, year, NNS), (ago, ago...  \n",
       "3          ((In, in, IN), (my, my, PRP$), (country, count...  \n",
       "4          ((I, i, PRP), (organized, organize, VBD), (the...  \n",
       "5          ((First, first, RB), (,, ,, ,), (prepare, prep...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Keep only the four relevant columns\n",
    "\n",
    "pelic_df = pelic_df[[\"anon_id\", \"level_id\", \"L1\", \"tokens\", \"tok_lem_POS\"]]\n",
    "pelic_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'met', 'my', 'friend', 'nife', 'while', 'i', 'was', 'studying', 'in']\n",
      "['i', 'meet', 'my', 'friend', 'nife', 'while', 'i', 'be', 'study', 'in']\n"
     ]
    }
   ],
   "source": [
    "# Create a list of all word and lemma tokens, flatten these lists, and make all tokens lowercase\n",
    "\n",
    "toks_list = [y.lower() for x in pelic_df.tokens for y in x]\n",
    "print(toks_list[:10])\n",
    "\n",
    "lemmas_list = [y[1].lower() for x in pelic_df.tok_lem_POS for y in x]\n",
    "print(lemmas_list[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PELIC total tokens: 4819157\n",
      "PELIC total lemmas: 4819157\n"
     ]
    }
   ],
   "source": [
    "# Calculate the total number of tokens\n",
    "\n",
    "total_toks = len(toks_list)\n",
    "total_lemmas = len(lemmas_list)\n",
    "print('PELIC total tokens:',total_toks)\n",
    "print('PELIC total lemmas:',total_lemmas)\n",
    "\n",
    "# These should match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Total frequencies\n",
    "This will be the general frequency distribution of all tokens for all students, i.e. a dictionary where each word or lemma is the key and its raw frequency is the value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw frequency counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FreqDist with 49241 samples and 4819157 outcomes>\n",
      "<FreqDist with 41362 samples and 4819157 outcomes>\n"
     ]
    }
   ],
   "source": [
    "# Create the distribution using our toks_list and lemmas_list\n",
    "\n",
    "freq_dist = FreqDist(toks_list)\n",
    "freq_dist_lemmas = FreqDist(lemmas_list)\n",
    "\n",
    "print(freq_dist) # 49,241 word types\n",
    "print(freq_dist_lemmas) # 41,448 lemma types - this is lower than word types as expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('youngstown', 10),\n",
       " ('sober-minded', 1),\n",
       " ('paticpates', 1),\n",
       " ('948', 1),\n",
       " ('fluctuation', 30)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check a random sample of 5 entries in the dictionaries\n",
    "\n",
    "random.sample(freq_dist.items(), 5)\n",
    "random.sample(freq_dist_lemmas.items(), 5)\n",
    "\n",
    "# As we can see here, spelling is not corrected in the texts, a factor which must always be considered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>frequency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>.</td>\n",
       "      <td>279067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>,</td>\n",
       "      <td>217804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the</td>\n",
       "      <td>193235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>to</td>\n",
       "      <td>135001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and</td>\n",
       "      <td>108331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>i</td>\n",
       "      <td>98387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>a</td>\n",
       "      <td>92882</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>in</td>\n",
       "      <td>92013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>of</td>\n",
       "      <td>88910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>is</td>\n",
       "      <td>75973</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  word  frequency\n",
       "0    .     279067\n",
       "1    ,     217804\n",
       "2  the     193235\n",
       "3   to     135001\n",
       "4  and     108331\n",
       "5    i      98387\n",
       "6    a      92882\n",
       "7   in      92013\n",
       "8   of      88910\n",
       "9   is      75973"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframes from these frequency distributions\n",
    "\n",
    "# words\n",
    "freq_df = pd.DataFrame.from_dict(freq_dist, orient='index', columns=[\"frequency\"]) # create df from dict\n",
    "freq_df = freq_df.sort_values(by=['frequency'], ascending = False) # sort by frequency\n",
    "freq_df = freq_df.reset_index() # make new index\n",
    "freq_df = freq_df.rename(columns = {\"index\":\"word\" }) # rename index\n",
    "freq_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemma</th>\n",
       "      <th>frequency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>.</td>\n",
       "      <td>279067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>,</td>\n",
       "      <td>217804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the</td>\n",
       "      <td>193235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>be</td>\n",
       "      <td>179888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>to</td>\n",
       "      <td>135001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>and</td>\n",
       "      <td>108331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>a</td>\n",
       "      <td>103373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>i</td>\n",
       "      <td>98387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>in</td>\n",
       "      <td>92013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>of</td>\n",
       "      <td>88910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  lemma  frequency\n",
       "0     .     279067\n",
       "1     ,     217804\n",
       "2   the     193235\n",
       "3    be     179888\n",
       "4    to     135001\n",
       "5   and     108331\n",
       "6     a     103373\n",
       "7     i      98387\n",
       "8    in      92013\n",
       "9    of      88910"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lemmas\n",
    "freq_df_lemmas = pd.DataFrame.from_dict(freq_dist_lemmas, orient='index', columns=[\"frequency\"]) # create df from dict\n",
    "freq_df_lemmas = freq_df_lemmas.sort_values(by=['frequency'], ascending = False) # sort by frequency\n",
    "freq_df_lemmas = freq_df_lemmas.reset_index() # make new index\n",
    "freq_df_lemmas = freq_df_lemmas.rename(columns = {\"index\":\"lemma\" }) # rename index\n",
    "freq_df_lemmas.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notes:** \n",
    "- Using NLTK tokens, punctuation is also included in the dictionary. If considering frequency ranking (for example for frequency bands), it is important to exclude punctuation.\n",
    "- Note the difference between the top 10 most frequent words and lemmas - in the lemma list the lemma 'be' is much higher as it combines 'is' and other verb forms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency per million\n",
    "In addition to raw frequencies, frequency per million is a useful measurement for comparion, both within and across corpora. The formula for the per/mil statistic is:  \n",
    "\n",
    "`per M factor = 1,000,000 / total corpus frequency  \n",
    "word per M = word frequency * per M factor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2075051715476379"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_M_factor = 1000000 / total_toks # total_lemmas would produce the same result\n",
    "per_M_factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>frequency</th>\n",
       "      <th>per_M</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>.</td>\n",
       "      <td>279067</td>\n",
       "      <td>57907.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>,</td>\n",
       "      <td>217804</td>\n",
       "      <td>45195.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the</td>\n",
       "      <td>193235</td>\n",
       "      <td>40097.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>to</td>\n",
       "      <td>135001</td>\n",
       "      <td>28013.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and</td>\n",
       "      <td>108331</td>\n",
       "      <td>22479.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  word  frequency    per_M\n",
       "0    .     279067  57907.8\n",
       "1    ,     217804  45195.5\n",
       "2  the     193235  40097.3\n",
       "3   to     135001  28013.4\n",
       "4  and     108331  22479.2"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new column for the per_M statistic for each word and lemma\n",
    "\n",
    "# words\n",
    "freq_df['per_M'] = freq_df.frequency.map(lambda x: round(x*per_M_factor,1)) # Round the results to one decimal\n",
    "freq_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemma</th>\n",
       "      <th>frequency</th>\n",
       "      <th>per_M</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>.</td>\n",
       "      <td>279067</td>\n",
       "      <td>57907.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>,</td>\n",
       "      <td>217804</td>\n",
       "      <td>45195.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the</td>\n",
       "      <td>193235</td>\n",
       "      <td>40097.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>be</td>\n",
       "      <td>179888</td>\n",
       "      <td>37327.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>to</td>\n",
       "      <td>135001</td>\n",
       "      <td>28013.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  lemma  frequency    per_M\n",
       "0     .     279067  57907.8\n",
       "1     ,     217804  45195.5\n",
       "2   the     193235  40097.3\n",
       "3    be     179888  37327.7\n",
       "4    to     135001  28013.4"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lemmas\n",
    "freq_df_lemmas['per_M'] = freq_df_lemmas.frequency.map(lambda x: round(x*per_M_factor,1)) # Round the results to one decimal\n",
    "freq_df_lemmas.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print a csv of these dataframes for easy access\n",
    "\n",
    "freq_df.to_csv(\"word_frequencies.csv\")\n",
    "freq_df_lemmas.to_csv(\"lemma_frequencies.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequencies by level<a name=\"levelfreqdict\"></a>\n",
    "The following code is the same basic process, only now we will have four dictionaries, one per level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>lemmas</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eq0</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>[I, met, my, friend, Nife, while, I, was, stud...</td>\n",
       "      <td>((I, i, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "      <td>[i, meet, my, friend, nife, while, i, be, stud...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>am8</td>\n",
       "      <td>4</td>\n",
       "      <td>Thai</td>\n",
       "      <td>[Ten, years, ago, ,, I, met, a, women, on, the...</td>\n",
       "      <td>((Ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "      <td>[ten, year, ago, ,, i, meet, a, woman, on, the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dk5</td>\n",
       "      <td>4</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>[In, my, country, we, usually, do, n't, use, t...</td>\n",
       "      <td>((In, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>[in, my, country, we, usually, do, not, use, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dk5</td>\n",
       "      <td>4</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>[I, organized, the, instructions, by, time, .]</td>\n",
       "      <td>((I, i, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[i, organize, the, instruction, by, time, .]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad1</td>\n",
       "      <td>4</td>\n",
       "      <td>Korean</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>((First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>[first, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id  level_id       L1  \\\n",
       "answer_id                              \n",
       "1             eq0         4   Arabic   \n",
       "2             am8         4     Thai   \n",
       "3             dk5         4  Turkish   \n",
       "4             dk5         4  Turkish   \n",
       "5             ad1         4   Korean   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          [I, met, my, friend, Nife, while, I, was, stud...   \n",
       "2          [Ten, years, ago, ,, I, met, a, women, on, the...   \n",
       "3          [In, my, country, we, usually, do, n't, use, t...   \n",
       "4             [I, organized, the, instructions, by, time, .]   \n",
       "5          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "\n",
       "                                                 tok_lem_POS  \\\n",
       "answer_id                                                      \n",
       "1          ((I, i, PRP), (met, meet, VBD), (my, my, PRP$)...   \n",
       "2          ((Ten, ten, CD), (years, year, NNS), (ago, ago...   \n",
       "3          ((In, in, IN), (my, my, PRP$), (country, count...   \n",
       "4          ((I, i, PRP), (organized, organize, VBD), (the...   \n",
       "5          ((First, first, RB), (,, ,, ,), (prepare, prep...   \n",
       "\n",
       "                                                      lemmas  \n",
       "answer_id                                                     \n",
       "1          [i, meet, my, friend, nife, while, i, be, stud...  \n",
       "2          [ten, year, ago, ,, i, meet, a, woman, on, the...  \n",
       "3          [in, my, country, we, usually, do, not, use, t...  \n",
       "4               [i, organize, the, instruction, by, time, .]  \n",
       "5          [first, ,, prepare, a, port, ,, loose, tea, ,,...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First, create a separate 'lemmas' column in pelic_df from the tok_lem_POS tuple\n",
    "pelic_df['lemmas'] = pelic_df.tok_lem_POS.apply(lambda row: [x[1] for x in row])\n",
    "pelic_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Next, split pelic_df into smaller dataframes of each level\n",
    "\n",
    "level2_df = pelic_df[pelic_df.level_id == 2]\n",
    "level3_df = pelic_df[pelic_df.level_id == 3]\n",
    "level4_df = pelic_df[pelic_df.level_id == 4]\n",
    "level5_df = pelic_df[pelic_df.level_id == 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the frequency distributions for each level\n",
    "\n",
    "# words\n",
    "level2_freq_dist = FreqDist([y.lower() for x in level2_df.tokens for y in x])\n",
    "level3_freq_dist = FreqDist([y.lower() for x in level3_df.tokens for y in x])\n",
    "level4_freq_dist = FreqDist([y.lower() for x in level4_df.tokens for y in x])\n",
    "level5_freq_dist = FreqDist([y.lower() for x in level5_df.tokens for y in x])\n",
    "\n",
    "# lemmas\n",
    "level2_freq_dist_lemmas = FreqDist([y.lower() for x in level2_df.lemmas for y in x])\n",
    "level3_freq_dist_lemmas = FreqDist([y.lower() for x in level3_df.lemmas for y in x])\n",
    "level4_freq_dist_lemmas = FreqDist([y.lower() for x in level4_df.lemmas for y in x])\n",
    "level5_freq_dist_lemmas = FreqDist([y.lower() for x in level5_df.lemmas for y in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FreqDist with 2641 samples and 34894 outcomes>\n",
      "[('.', 3305), ('i', 1328), (',', 1318), ('and', 945), ('to', 802), ('the', 795), ('is', 770), ('my', 757), ('a', 739), ('in', 668)]\n",
      "<FreqDist with 17219 samples and 729061 outcomes>\n",
      "[('.', 49617), (',', 30768), ('the', 26880), ('i', 21546), ('to', 20039), ('and', 15475), ('a', 14547), ('in', 14370), ('is', 13065), ('of', 11469)]\n",
      "<FreqDist with 28939 samples and 2213651 outcomes>\n",
      "[('.', 128822), (',', 101411), ('the', 82504), ('to', 63373), ('and', 50090), ('i', 48617), ('a', 41915), ('in', 40310), ('of', 38846), ('is', 33978)]\n",
      "<FreqDist with 30718 samples and 1841551 outcomes>\n",
      "[('.', 97323), (',', 84307), ('the', 83056), ('to', 50787), ('and', 41821), ('of', 38340), ('in', 36665), ('a', 35681), ('is', 28160), ('i', 26896)]\n"
     ]
    }
   ],
   "source": [
    "# Check the results\n",
    "\n",
    "# words\n",
    "print(level2_freq_dist)\n",
    "print(level2_freq_dist.most_common(10))\n",
    "print(level3_freq_dist)\n",
    "print(level3_freq_dist.most_common(10))\n",
    "print(level4_freq_dist)\n",
    "print(level4_freq_dist.most_common(10))\n",
    "print(level5_freq_dist)\n",
    "print(level5_freq_dist.most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FreqDist with 2258 samples and 34894 outcomes>\n",
      "[('.', 3305), ('be', 1420), ('i', 1328), (',', 1318), ('and', 945), ('to', 802), ('a', 796), ('the', 795), ('my', 757), ('in', 668)]\n",
      "<FreqDist with 14184 samples and 729061 outcomes>\n",
      "[('.', 49617), (',', 30768), ('be', 27537), ('the', 26880), ('i', 21546), ('to', 20039), ('a', 15677), ('and', 15475), ('in', 14370), ('of', 11469)]\n",
      "<FreqDist with 23456 samples and 2213651 outcomes>\n",
      "[('.', 128822), (',', 101411), ('the', 82504), ('be', 81296), ('to', 63373), ('and', 50090), ('i', 48617), ('a', 46545), ('in', 40310), ('of', 38846)]\n",
      "<FreqDist with 24677 samples and 1841551 outcomes>\n",
      "[('.', 97323), (',', 84307), ('the', 83056), ('be', 69635), ('to', 50787), ('and', 41821), ('a', 40355), ('of', 38340), ('in', 36665), ('i', 26896)]\n"
     ]
    }
   ],
   "source": [
    "# lemmas\n",
    "print(level2_freq_dist_lemmas)\n",
    "print(level2_freq_dist_lemmas.most_common(10))\n",
    "print(level3_freq_dist_lemmas)\n",
    "print(level3_freq_dist_lemmas.most_common(10))\n",
    "print(level4_freq_dist_lemmas)\n",
    "print(level4_freq_dist_lemmas.most_common(10))\n",
    "print(level5_freq_dist_lemmas)\n",
    "print(level5_freq_dist_lemmas.most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "279067\n",
      "279067\n",
      "279067\n",
      "279067\n"
     ]
    }
   ],
   "source": [
    "# Check that the sum of period frequency counts should add up to the total period frequency count\n",
    "\n",
    "# words\n",
    "print(level2_freq_dist['.'] + level3_freq_dist['.'] + level4_freq_dist['.'] + level5_freq_dist['.'])\n",
    "print(freq_dist['.'])\n",
    "\n",
    "# lemmas\n",
    "print(level2_freq_dist_lemmas['.'] + level3_freq_dist_lemmas['.'] + level4_freq_dist_lemmas['.'] + level5_freq_dist_lemmas['.'])\n",
    "print(freq_dist_lemmas['.'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export to pickle files\n",
    "\n",
    "# words\n",
    "level2file = open('word_frequency_pickles/level2_word_frequencies.pkl','wb')\n",
    "pkl.dump(level2_freq_dist, level2file)\n",
    "level2file.close()\n",
    "\n",
    "level3file = open('word_frequency_pickles/level3_word_frequencies.pkl','wb')\n",
    "pkl.dump(level3_freq_dist, level3file)\n",
    "level3file.close()\n",
    "\n",
    "level4file = open('word_frequency_pickles/level4_word_frequencies.pkl','wb')\n",
    "pkl.dump(level4_freq_dist, level4file)\n",
    "level4file.close()\n",
    "\n",
    "level5file = open('word_frequency_pickles/level5_word_frequencies.pkl','wb')\n",
    "pkl.dump(level5_freq_dist, level5file)\n",
    "level5file.close()\n",
    "\n",
    "# lemmas\n",
    "level2file_lemmas = open('lemma_frequency_pickles/level2_lemma_frequencies.pkl','wb')\n",
    "pkl.dump(level2_freq_dist_lemmas, level2file_lemmas)\n",
    "level2file_lemmas.close()\n",
    "\n",
    "level3file_lemmas = open('lemma_frequency_pickles/level3_lemma_frequencies.pkl','wb')\n",
    "pkl.dump(level3_freq_dist_lemmas, level3file_lemmas)\n",
    "level3file_lemmas.close()\n",
    "\n",
    "level4file_lemmas = open('lemma_frequency_pickles/level4_lemma_frequencies.pkl','wb')\n",
    "pkl.dump(level4_freq_dist_lemmas, level4file_lemmas)\n",
    "level4file_lemmas.close()\n",
    "\n",
    "level5file_lemmas = open('lemma_frequency_pickles/level5_lemma_frequencies.pkl','wb')\n",
    "pkl.dump(level5_freq_dist_lemmas, level5file_lemmas)\n",
    "level5file_lemmas.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequencies by L1<a name=\"langfreqdict\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the same process, but by L1. We will do this for the five most frequent languagaes: Arabic, Chinese, Korean, Japanese, and Spanish. However, this same process could be applied to create frequency dictionaries for any of the L1s in PELIC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Arabic               16831\n",
       "Korean                9226\n",
       "Chinese               8503\n",
       "Japanese              2782\n",
       "Spanish               1909\n",
       "Turkish               1538\n",
       "Thai                  1383\n",
       "Taiwanese              678\n",
       "Portuguese             603\n",
       "Other                  493\n",
       "French                 477\n",
       "Italian                393\n",
       "Russian                193\n",
       "Hebrew                 189\n",
       "English                154\n",
       "Farsi                  144\n",
       "Mongol                 144\n",
       "Vietnamese             104\n",
       "German                  85\n",
       "Indonesian              71\n",
       "Romanian                69\n",
       "Russian,Ukrainian       62\n",
       "Azerbaijani             45\n",
       "Suundi                  41\n",
       "Swedish                 31\n",
       "Montenegrin             27\n",
       "Zulu                    25\n",
       "Polish                  20\n",
       "Hindi                    6\n",
       "Swahili                  4\n",
       "Name: L1, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pelic_df.L1.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create sub-dataframes by language\n",
    "\n",
    "ARA_df = pelic_df[pelic_df.L1 == \"Arabic\"]\n",
    "CHI_df = pelic_df[pelic_df.L1 == \"Chinese\"]\n",
    "KOR_df = pelic_df[pelic_df.L1 == \"Korean\"]\n",
    "JAP_df = pelic_df[pelic_df.L1 == \"Japanese\"]\n",
    "SPA_df = pelic_df[pelic_df.L1 == \"Spanish\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create token lists (words and lemmas)\n",
    "\n",
    "#words \n",
    "ARA_freq_dist = FreqDist([y.lower() for x in ARA_df.tokens for y in x])\n",
    "CHI_freq_dist = FreqDist([y.lower() for x in CHI_df.tokens for y in x])\n",
    "KOR_freq_dist = FreqDist([y.lower() for x in KOR_df.tokens for y in x])\n",
    "JAP_freq_dist = FreqDist([y.lower() for x in JAP_df.tokens for y in x])\n",
    "SPA_freq_dist = FreqDist([y.lower() for x in SPA_df.tokens for y in x])\n",
    "\n",
    "# lemmas\n",
    "ARA_freq_dist_lemmas = FreqDist([y.lower() for x in ARA_df.lemmas for y in x])\n",
    "CHI_freq_dist_lemmas = FreqDist([y.lower() for x in CHI_df.lemmas for y in x])\n",
    "KOR_freq_dist_lemmas = FreqDist([y.lower() for x in KOR_df.lemmas for y in x])\n",
    "JAP_freq_dist_lemmas = FreqDist([y.lower() for x in JAP_df.lemmas for y in x])\n",
    "SPA_freq_dist_lemmas = FreqDist([y.lower() for x in SPA_df.lemmas for y in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arabic (words):\n",
      "<FreqDist with 26608 samples and 1534315 outcomes>\n",
      "[('.', 87084), ('the', 67846), (',', 62284), ('to', 43467), ('and', 36320), ('in', 32747), ('i', 30720), ('of', 27889), ('a', 27225), ('is', 24273)]\n",
      "\n",
      "Arabic (lemmas):\n",
      "<FreqDist with 21653 samples and 1534315 outcomes>\n",
      "[('.', 87084), ('the', 67846), (',', 62284), ('be', 56339), ('to', 43467), ('and', 36320), ('in', 32747), ('i', 30720), ('a', 30298), ('of', 27889)]\n",
      "\n",
      "Chinese (words):\n",
      "<FreqDist with 20488 samples and 950340 outcomes>\n",
      "[('.', 55369), (',', 46239), ('the', 37648), ('to', 26182), ('and', 20464), ('a', 18789), ('i', 18452), ('in', 16784), ('of', 16153), ('is', 14633)]\n",
      "\n",
      "Chinese (lemmas):\n",
      "<FreqDist with 16064 samples and 950340 outcomes>\n",
      "[('.', 55369), (',', 46239), ('the', 37648), ('be', 33764), ('to', 26182), ('a', 20842), ('and', 20464), ('i', 18452), ('in', 16784), ('of', 16153)]\n",
      "\n",
      "Korean (words):\n",
      "<FreqDist with 20480 samples and 987135 outcomes>\n",
      "[('.', 61690), (',', 48546), ('the', 32939), ('to', 27218), ('i', 22767), ('and', 20394), ('a', 20041), ('of', 18041), ('in', 16835), ('is', 15863)]\n",
      "\n",
      "Korean (lemmas):\n",
      "<FreqDist with 16324 samples and 987135 outcomes>\n",
      "[('.', 61690), (',', 48546), ('be', 38449), ('the', 32939), ('to', 27218), ('i', 22767), ('a', 22304), ('and', 20394), ('of', 18041), ('in', 16835)]\n",
      "\n",
      "Japanese (words):\n",
      "<FreqDist with 12037 samples and 352115 outcomes>\n",
      "[('.', 20439), (',', 16542), ('the', 13197), ('to', 10009), ('i', 7993), ('and', 7290), ('of', 6983), ('a', 6929), ('in', 6576), ('is', 5213)]\n",
      "\n",
      "Japanese (lemmas):\n",
      "<FreqDist with 9279 samples and 352115 outcomes>\n",
      "[('.', 20439), (',', 16542), ('the', 13197), ('be', 13054), ('to', 10009), ('i', 7993), ('a', 7705), ('and', 7290), ('of', 6983), ('in', 6576)]\n",
      "\n",
      "Spanish (words):\n",
      "<FreqDist with 10111 samples and 216400 outcomes>\n",
      "[('.', 10380), ('the', 10033), (',', 9896), ('to', 6226), ('and', 5550), ('a', 4447), ('i', 4427), ('in', 4393), ('of', 4371), ('is', 3622)]\n",
      "\n",
      "Spanish (lemmas):\n",
      "<FreqDist with 7879 samples and 216400 outcomes>\n",
      "[('.', 10380), ('the', 10033), (',', 9896), ('be', 8751), ('to', 6226), ('and', 5550), ('a', 5048), ('i', 4427), ('in', 4393), ('of', 4371)]\n"
     ]
    }
   ],
   "source": [
    "# Check data\n",
    "\n",
    "print(\"Arabic (words):\")\n",
    "print(ARA_freq_dist)\n",
    "print(ARA_freq_dist.most_common(10))\n",
    "\n",
    "print(\"\\nArabic (lemmas):\")\n",
    "print(ARA_freq_dist_lemmas)\n",
    "print(ARA_freq_dist_lemmas.most_common(10))\n",
    "\n",
    "print(\"\\nChinese (words):\")\n",
    "print(CHI_freq_dist)\n",
    "print(CHI_freq_dist.most_common(10))\n",
    "\n",
    "print(\"\\nChinese (lemmas):\")\n",
    "print(CHI_freq_dist_lemmas)\n",
    "print(CHI_freq_dist_lemmas.most_common(10))\n",
    "\n",
    "print(\"\\nKorean (words):\")\n",
    "print(KOR_freq_dist)\n",
    "print(KOR_freq_dist.most_common(10))\n",
    "\n",
    "print(\"\\nKorean (lemmas):\")\n",
    "print(KOR_freq_dist_lemmas)\n",
    "print(KOR_freq_dist_lemmas.most_common(10))\n",
    "\n",
    "print(\"\\nJapanese (words):\")\n",
    "print(JAP_freq_dist)\n",
    "print(JAP_freq_dist.most_common(10))\n",
    "\n",
    "print(\"\\nJapanese (lemmas):\")\n",
    "print(JAP_freq_dist_lemmas)\n",
    "print(JAP_freq_dist_lemmas.most_common(10))\n",
    "\n",
    "print(\"\\nSpanish (words):\")\n",
    "print(SPA_freq_dist)\n",
    "print(SPA_freq_dist.most_common(10))\n",
    "\n",
    "print(\"\\nSpanish (lemmas):\")\n",
    "print(SPA_freq_dist_lemmas)\n",
    "print(SPA_freq_dist_lemmas.most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export to pickle files\n",
    "\n",
    "# words\n",
    "ARAfile = open('word_frequency_pickles/arabic_word_frequencies.pkl','wb') \n",
    "pkl.dump(ARA_freq_dist, ARAfile)\n",
    "ARAfile.close()\n",
    "\n",
    "CHIfile = open('word_frequency_pickles/chinese_word_frequencies.pkl','wb') \n",
    "pkl.dump(CHI_freq_dist, CHIfile)\n",
    "CHIfile.close()\n",
    "\n",
    "KORfile = open('word_frequency_pickles/korean_word_frequencies.pkl','wb') \n",
    "pkl.dump(KOR_freq_dist, KORfile)\n",
    "KORfile.close()\n",
    "\n",
    "JAPfile = open('word_frequency_pickles/japanese_word_frequencies.pkl','wb') \n",
    "pkl.dump(JAP_freq_dist, JAPfile)\n",
    "JAPfile.close()\n",
    "\n",
    "SPAfile = open('word_frequency_pickles/spanish_word_frequencies.pkl','wb') \n",
    "pkl.dump(SPA_freq_dist, SPAfile)\n",
    "SPAfile.close()\n",
    "\n",
    "\n",
    "# lemmas\n",
    "ARAfile_lemmas = open('lemma_frequency_pickles/arabic_lemma_frequencies.pkl','wb') \n",
    "pkl.dump(ARA_freq_dist_lemmas, ARAfile_lemmas)\n",
    "ARAfile_lemmas.close()\n",
    "\n",
    "CHIfile_lemmas = open('lemma_frequency_pickles/chinese_lemma_frequencies.pkl','wb') \n",
    "pkl.dump(CHI_freq_dist_lemmas, CHIfile_lemmas)\n",
    "CHIfile_lemmas.close()\n",
    "\n",
    "KORfile_lemmas = open('lemma_frequency_pickles/korean_lemma_frequencies.pkl','wb') \n",
    "pkl.dump(KOR_freq_dist_lemmas, KORfile_lemmas)\n",
    "KORfile_lemmas.close()\n",
    "\n",
    "JAPfile_lemmas = open('lemma_frequency_pickles/japanese_lemma_frequencies.pkl','wb') \n",
    "pkl.dump(JAP_freq_dist_lemmas, JAPfile_lemmas)\n",
    "JAPfile_lemmas.close()\n",
    "\n",
    "SPAfile_lemmas = open('lemma_frequency_pickles/spanish_lemma_frequencies.pkl','wb') \n",
    "pkl.dump(SPA_freq_dist_lemmas, SPAfile_lemmas)\n",
    "SPAfile_lemmas.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional frequency distributions\n",
    "In the previous section, we created frequency counts of overall word frequencies, as well as separate frequency distributions for the levels: 2, 3, 4, 5 and the languages: Arabic, Chinese, Korean, Japanese, Spanish.\n",
    "\n",
    "Here, we create two additional _Conditional_ Frequency Distributions: one conditional on all four levels, and one conditional on all five languages above. These distributions allow access to word frequencies based on either Level or L1 using only one pickle file, rather then separate pickle files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of tuples so that the level is associated with each word or lemma\n",
    "\n",
    "# words\n",
    "level2_cond_list = [(2, y.lower()) for x in level2_df.tokens for y in x]\n",
    "level3_cond_list = [(3, y.lower()) for x in level3_df.tokens for y in x]\n",
    "level4_cond_list = [(4, y.lower()) for x in level4_df.tokens for y in x]\n",
    "level5_cond_list = [(5, y.lower()) for x in level5_df.tokens for y in x]\n",
    "\n",
    "# lemmas\n",
    "level2_cond_list_lemmas = [(2, y.lower()) for x in level2_df.lemmas for y in x]\n",
    "level3_cond_list_lemmas = [(3, y.lower()) for x in level3_df.lemmas for y in x]\n",
    "level4_cond_list_lemmas = [(4, y.lower()) for x in level4_df.lemmas for y in x]\n",
    "level5_cond_list_lemmas = [(5, y.lower()) for x in level5_df.lemmas for y in x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The level dataframes are no longer needed and can be deleted to save memory.\n",
    "\n",
    "del level2_df\n",
    "del level3_df\n",
    "del level4_df\n",
    "del level5_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine the above lists\n",
    "\n",
    "level_cond_list = level2_cond_list + level3_cond_list + level4_cond_list + level5_cond_list\n",
    "level_cond_list_lemmas = level2_cond_list_lemmas + level3_cond_list_lemmas + level4_cond_list_lemmas + level5_cond_list_lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create conditional frequency distributions with the above lists\n",
    "\n",
    "level_cond_dist = ConditionalFreqDist()\n",
    "level_cond_dist_lemmas = ConditionalFreqDist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making level the condition (i.e., the first tuple entry and adding frequencies)\n",
    "\n",
    "for x in level_cond_list:\n",
    "    cond = x[0]\n",
    "    level_cond_dist[cond][x[1]] += 1\n",
    "\n",
    "for x in level_cond_list_lemmas:\n",
    "    cond = x[0]\n",
    "    level_cond_dist_lemmas[cond][x[1]] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('.', 97323),\n",
       " (',', 84307),\n",
       " ('the', 83056),\n",
       " ('be', 69635),\n",
       " ('to', 50787),\n",
       " ('and', 41821),\n",
       " ('a', 40355),\n",
       " ('of', 38340),\n",
       " ('in', 36665),\n",
       " ('i', 26896)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the resulting conditional distributions\n",
    "\n",
    "# words\n",
    "print(level_cond_dist[2].most_common(10))\n",
    "print(level_cond_dist[3].most_common(10))\n",
    "print(level_cond_dist[4].most_common(10))\n",
    "level_cond_dist[5].most_common(10))\n",
    "\n",
    "# lemmas\n",
    "print(level_cond_dist_lemmas[2].most_common(10))\n",
    "print(level_cond_dist_lemmas[3].most_common(10))\n",
    "print(level_cond_dist_lemmas[4].most_common(10))\n",
    "print(level_cond_dist_lemmas[5].most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "279067"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check that the counts for period add up to the total period count\n",
    "# found to be 279067 earlier in this notebook\n",
    "\n",
    "level_cond_dist[2]['.'] + level_cond_dist[3]['.'] + level_cond_dist[4]['.'] + level_cond_dist[5]['.']\n",
    "level_cond_dist_lemmas[2]['.'] + level_cond_dist_lemmas[3]['.'] + level_cond_dist_lemmas[4]['.'] + level_cond_dist_lemmas[5]['.']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L1 \n",
    "Repeating the process above but with L1s instead of levels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of tuples so that the L1 is associated with each word or lemma\n",
    "\n",
    "# words\n",
    "ARA_cond_list = [('Arabic', y.lower()) for x in ARA_df.tokens for y in x]\n",
    "CHI_cond_list = [('Chinese', y.lower()) for x in CHI_df.tokens for y in x]\n",
    "KOR_cond_list = [('Korean', y.lower()) for x in KOR_df.tokens for y in x]\n",
    "JAP_cond_list = [('Japanese', y.lower()) for x in JAP_df.tokens for y in x]\n",
    "SPA_cond_list = [('Spanish', y.lower()) for x in SPA_df.tokens for y in x]\n",
    "\n",
    "# lemmas\n",
    "ARA_cond_list_lemmas = [('Arabic', y.lower()) for x in ARA_df.lemmas for y in x]\n",
    "CHI_cond_list_lemmas = [('Chinese', y.lower()) for x in CHI_df.lemmas for y in x]\n",
    "KOR_cond_list_lemmas = [('Korean', y.lower()) for x in KOR_df.lemmas for y in x]\n",
    "JAP_cond_list_lemmas = [('Japanese', y.lower()) for x in JAP_df.lemmas for y in x]\n",
    "SPA_cond_list_lemmas = [('Spanish', y.lower()) for x in SPA_df.lemmas for y in x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The dataframes are no longer needed and can be deleted to save memory.\n",
    "\n",
    "del ARA_df\n",
    "del CHI_df\n",
    "del KOR_df\n",
    "del JAP_df\n",
    "del SPA_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine the above lists\n",
    "\n",
    "lang_cond_list = ARA_cond_list + CHI_cond_list + KOR_cond_list + JAP_cond_list + SPA_cond_list\n",
    "lang_cond_list_lemmas = ARA_cond_list_lemmas + CHI_cond_list_lemmas + KOR_cond_list_lemmas + JAP_cond_list_lemmas + SPA_cond_list_lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create conditional frequency distributions with the above lists\n",
    "\n",
    "lang_cond_dist = ConditionalFreqDist()\n",
    "lang_cond_dist_lemmas = ConditionalFreqDist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making L1 the condition (i.e., the first tuple entry and adding frequencies)\n",
    "\n",
    "for x in lang_cond_list:\n",
    "    cond = x[0]\n",
    "    lang_cond_dist[cond][x[1]] += 1\n",
    "    \n",
    "for x in lang_cond_list_lemmas:\n",
    "    cond = x[0]\n",
    "    lang_cond_dist_lemmas[cond][x[1]] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arabic (words)\n",
      " [('.', 87084), ('the', 67846), (',', 62284), ('to', 43467), ('and', 36320), ('in', 32747), ('i', 30720), ('of', 27889), ('a', 27225), ('is', 24273)]\n",
      "\n",
      "Chinese (words)\n",
      " [('.', 55369), (',', 46239), ('the', 37648), ('to', 26182), ('and', 20464), ('a', 18789), ('i', 18452), ('in', 16784), ('of', 16153), ('is', 14633)]\n",
      "\n",
      "Korean (words)\n",
      " [('.', 61690), (',', 48546), ('the', 32939), ('to', 27218), ('i', 22767), ('and', 20394), ('a', 20041), ('of', 18041), ('in', 16835), ('is', 15863)]\n",
      "\n",
      "Japanese (words)\n",
      " [('.', 20439), (',', 16542), ('the', 13197), ('to', 10009), ('i', 7993), ('and', 7290), ('of', 6983), ('a', 6929), ('in', 6576), ('is', 5213)]\n",
      "\n",
      "Spanish (words)\n",
      " [('.', 10380), ('the', 10033), (',', 9896), ('to', 6226), ('and', 5550), ('a', 4447), ('i', 4427), ('in', 4393), ('of', 4371), ('is', 3622)]\n",
      "\n",
      "Arabic (lemmas)\n",
      " [('.', 87084), ('the', 67846), (',', 62284), ('be', 56339), ('to', 43467), ('and', 36320), ('in', 32747), ('i', 30720), ('a', 30298), ('of', 27889)]\n",
      "\n",
      "Chinese (lemmas)\n",
      " [('.', 55369), (',', 46239), ('the', 37648), ('be', 33764), ('to', 26182), ('a', 20842), ('and', 20464), ('i', 18452), ('in', 16784), ('of', 16153)]\n",
      "\n",
      "Korean (lemmas)\n",
      " [('.', 61690), (',', 48546), ('be', 38449), ('the', 32939), ('to', 27218), ('i', 22767), ('a', 22304), ('and', 20394), ('of', 18041), ('in', 16835)]\n",
      "\n",
      "Japanese (lemmas)\n",
      " [('.', 20439), (',', 16542), ('the', 13197), ('be', 13054), ('to', 10009), ('i', 7993), ('a', 7705), ('and', 7290), ('of', 6983), ('in', 6576)]\n",
      "\n",
      "Spanish (lemmas)\n",
      " [('.', 10380), ('the', 10033), (',', 9896), ('be', 8751), ('to', 6226), ('and', 5550), ('a', 5048), ('i', 4427), ('in', 4393), ('of', 4371)]\n"
     ]
    }
   ],
   "source": [
    "# Check the resulting conditional distributions\n",
    "\n",
    "# words\n",
    "print('Arabic (words)\\n',lang_cond_dist['Arabic'].most_common(10))\n",
    "print('\\nChinese (words)\\n',lang_cond_dist['Chinese'].most_common(10))\n",
    "print('\\nKorean (words)\\n',lang_cond_dist['Korean'].most_common(10))\n",
    "print('\\nJapanese (words)\\n',lang_cond_dist['Japanese'].most_common(10))\n",
    "print('\\nSpanish (words)\\n',lang_cond_dist['Spanish'].most_common(10))\n",
    "\n",
    "# lemmas\n",
    "print('\\nArabic (lemmas)\\n',lang_cond_dist_lemmas['Arabic'].most_common(10))\n",
    "print('\\nChinese (lemmas)\\n',lang_cond_dist_lemmas['Chinese'].most_common(10))\n",
    "print('\\nKorean (lemmas)\\n',lang_cond_dist_lemmas['Korean'].most_common(10))\n",
    "print('\\nJapanese (lemmas)\\n',lang_cond_dist_lemmas['Japanese'].most_common(10))\n",
    "print('\\nSpanish (lemmas)\\n',lang_cond_dist_lemmas['Spanish'].most_common(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Unlike with level, the counts for \".\" will not add up to the total because we are only using five languages, not all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export to pickle files\n",
    "\n",
    "# words\n",
    "levelfile = open('word_frequency_pickles/level_conditional_word_frequencies.pkl', 'wb')\n",
    "pkl.dump(level_cond_dist, levelfile)\n",
    "levelfile.close()\n",
    "\n",
    "langfile = open('word_frequency_pickles/L1_conditional_word_frequencies.pkl', 'wb')\n",
    "pkl.dump(lang_cond_dist, langfile)\n",
    "langfile.close()\n",
    "\n",
    "# lemmas\n",
    "levelfile_lemmas = open('lemma_frequency_pickles/level_conditional_lemma_frequencies.pkl', 'wb')\n",
    "pkl.dump(level_cond_dist_lemmas, levelfile_lemmas)\n",
    "levelfile_lemmas.close()\n",
    "\n",
    "langfile_lemmas = open('lemma_frequency_pickles/L1_conditional_lemma_frequencies.pkl', 'wb')\n",
    "pkl.dump(lang_cond_dist_lemmas, langfile_lemmas)\n",
    "langfile_lemmas.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demonstration\n",
    "The following small example showcases how the frequency information above might be useful. Here we contrast the Top 20 Arabic lemmas with the top 20 Spanish lemmas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove punctuation items from the lemma freq dictionary we are using and make into lists\n",
    "Arabic_top20 = [x for x in list(lang_cond_dist_lemmas['Arabic'].most_common(30)) if x[0].isalpha()][:20]\n",
    "Spanish_top20 = [x for x in list(lang_cond_dist_lemmas['Spanish'].most_common(30)) if x[0].isalpha()][:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Arabic</th>\n",
       "      <th>Spanish</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(the, 67846)</td>\n",
       "      <td>(the, 10033)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(be, 56339)</td>\n",
       "      <td>(be, 8751)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(to, 43467)</td>\n",
       "      <td>(to, 6226)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(and, 36320)</td>\n",
       "      <td>(and, 5550)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(in, 32747)</td>\n",
       "      <td>(a, 5048)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(i, 30720)</td>\n",
       "      <td>(i, 4427)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(a, 30298)</td>\n",
       "      <td>(in, 4393)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(of, 27889)</td>\n",
       "      <td>(of, 4371)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(have, 20827)</td>\n",
       "      <td>(that, 3098)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(that, 16282)</td>\n",
       "      <td>(have, 3055)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(for, 14397)</td>\n",
       "      <td>(my, 2044)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(it, 14391)</td>\n",
       "      <td>(for, 1877)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(my, 14188)</td>\n",
       "      <td>(it, 1808)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(you, 12505)</td>\n",
       "      <td>(you, 1808)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(not, 11778)</td>\n",
       "      <td>(not, 1697)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>(they, 9541)</td>\n",
       "      <td>(with, 1446)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>(do, 9191)</td>\n",
       "      <td>(this, 1428)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>(people, 8342)</td>\n",
       "      <td>(do, 1388)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>(with, 8127)</td>\n",
       "      <td>(can, 1351)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>(will, 7484)</td>\n",
       "      <td>(because, 1137)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Arabic          Spanish\n",
       "0     (the, 67846)     (the, 10033)\n",
       "1      (be, 56339)       (be, 8751)\n",
       "2      (to, 43467)       (to, 6226)\n",
       "3     (and, 36320)      (and, 5550)\n",
       "4      (in, 32747)        (a, 5048)\n",
       "5       (i, 30720)        (i, 4427)\n",
       "6       (a, 30298)       (in, 4393)\n",
       "7      (of, 27889)       (of, 4371)\n",
       "8    (have, 20827)     (that, 3098)\n",
       "9    (that, 16282)     (have, 3055)\n",
       "10    (for, 14397)       (my, 2044)\n",
       "11     (it, 14391)      (for, 1877)\n",
       "12     (my, 14188)       (it, 1808)\n",
       "13    (you, 12505)      (you, 1808)\n",
       "14    (not, 11778)      (not, 1697)\n",
       "15    (they, 9541)     (with, 1446)\n",
       "16      (do, 9191)     (this, 1428)\n",
       "17  (people, 8342)       (do, 1388)\n",
       "18    (with, 8127)      (can, 1351)\n",
       "19    (will, 7484)  (because, 1137)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top20_df = pd.DataFrame({'Arabic': Arabic_top20, 'Spanish': Spanish_top20})\n",
    "top20_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These look very similar, so let's see if there are any differences in terms of inclusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 20 lemmas used by L1 Arabic but not L1 Spanish students: ['they', 'people', 'will']\n",
      "Top 20 lemmas used by L1 Spanish but not L1 Arabic students: ['this', 'can', 'because']\n"
     ]
    }
   ],
   "source": [
    "Arabic_top20_list = [x[0] for x in top20_df.Arabic]\n",
    "Spanish_top20_list = [x[0] for x in top20_df.Spanish]\n",
    "\n",
    "print('Top 20 lemmas used by L1 Arabic but not L1 Spanish students:',\n",
    "      [x for x in Arabic_top20_list if x not in Spanish_top20_list])\n",
    "print('Top 20 lemmas used by L1 Spanish but not L1 Arabic students:',\n",
    "      [x for x in Spanish_top20_list if x not in Arabic_top20_list])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above, we see that overall, as would be expected, the most frequent lemmas are very consistent regardless of the learners' L1. However, we might be interested to look deeper into the use of the items which did differ. Could it be that Spanish learners use more discourse markers or complex sentences with _because_? Do Spanish and Arabic learners show different patterns of usage for modal verbs like _can_ and _will_? These questions can now be explored empirically in PELIC using the full texts in the [`PELIC_compiled.csv`](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/blob/master/PELIC_compiled.csv).\n",
    "\n",
    "For more detailed tutorials, please see the [tutorials folder](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/tree/master/tutorials) and description in the [`README.md`](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/blob/master/README.md). For, example, you may wish to create concordances of the items in these to 20 list in order to see them in context. The concordancing function is described in the [`PELIC_concordancing_tutorial`](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/blob/master/tutorials/PELIC_concordancing_tutorial.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to top](#PELIC-frequency-statistics)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}