Script 1. Corpus Creation and Word2Vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Script 1. Corpus Creation and Word2Vec\n",
    "\n",
    "This script takes the original Diorisis xml files, builds the corpus for the VSM, trains the data with the Word2Vec model, and performs a dimensionality reduction using t-SNE. It outputs a csv file containing a list of Greek models from the corpus, with corresponding x and y coordinates.\n",
    "\n",
    "This script only needs to be run once, unless you make changes to which xml files are used for the corpus, or to the parameters for the Word2Vec model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, load the dependencies. If any dependencies are not downloaded onto your computer, use pip to install."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## !pip install cltk ##\n",
    "\n",
    "from cltk.corpus.greek.beta_to_unicode import Replacer\n",
    "from cltk.corpus.utils.formatter import tonos_oxia_converter\n",
    "from cltk.stop.greek.stops import STOPS_LIST\n",
    "from glob import glob\n",
    "from xml.etree.ElementTree import parse\n",
    "\n",
    "import re\n",
    "import os\n",
    "import os.path\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Construct the Corpus\n",
    "This code takes the Koine Greek texts sourced from Diorisis Corpus, taking the lemma entries and appending them into a list. The relevant xml files can be found at https://figshare.com/articles/dataset/The_Diorisis_Ancient_Greek_Corpus/6187256. The texts used in this project are speicifed in Appendix A of the written thesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stopwords\n",
    "\n",
    "This csv file lists all the stopwords we wish to exclude from the corpus. The file can be downloaded from the Word2Vec_koine_greek GitHub repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "new_stops = os.path.join(\"Desktop/Word2Vec_koine_greek-master\", \"new_stops.csv\")\n",
    "\n",
    "f = open(new_stops)\n",
    "\n",
    "X = pd.read_csv(f, delimiter=\",\", )\n",
    "\n",
    "X.head()\n",
    "df = pd.DataFrame(X, columns=['Add Stops'])\n",
    "new_list = df['Add Stops'].values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## for testing purposes ##\n",
    "print(new_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XML Parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parsing XML\n",
    "\n",
    "xml_files = glob('Desktop/greek_corpus/*.xml')\n",
    "replacer = Replacer()\n",
    "corpus = []\n",
    "for xml in xml_files:\n",
    "    with open(xml, 'r') as x:\n",
    "        tree = parse(x)\n",
    "        root = tree.getroot()\n",
    "        for sentence in root.iter('sentence'):\n",
    "            sentences = []\n",
    "            for word in sentence.iter('word'):\n",
    "                for lemma in word.iter('lemma'):\n",
    "                    entry = lemma.get('entry')\n",
    "                    if entry is None:\n",
    "                        entry = replacer.beta_code(word.get('form'))\n",
    "                        sentences.append(entry)\n",
    "                    elif tonos_oxia_converter(entry) not in new_list:\n",
    "                        sentences.append(entry)\n",
    "            if len(sentences) > 0:\n",
    "                corpus.append(sentences)\n",
    "    x.close()\n",
    "\n",
    "\n",
    "## print(corpus) ## Testing purposes ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Run the Word2Vec Model\n",
    "The following scripts takes the preprocessed corpus and trains the data with Word2Vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dependencies\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import codecs\n",
    "import glob\n",
    "import multiprocessing\n",
    "import gensim.models.word2vec as w2v\n",
    "import sklearn.manifold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the model's parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code defines the hyperparameter\n",
    "# Dimensionality of the resulting word vectors. \n",
    "# The more vectors, the more computaionally extensive to train, but also more accurate.\n",
    "num_features = 500\n",
    "\n",
    "# Minimum word count threshold.\n",
    "min_word_count = 10\n",
    "\n",
    "# Number of threads to run in parallel.\n",
    "num_workers = multiprocessing.cpu_count()\n",
    "\n",
    "# Context window length. Note that Munson (2017: 17) says context_size is optimized at 12 for Greek.\n",
    "context_size = 2\n",
    "\n",
    "# Downsample setting for frequent words.\n",
    "#rate 0 and 1e-5 \n",
    "#how often to use\n",
    "downsampling = 1e-3\n",
    "\n",
    "# Seed for the RNG, to make the results reproducible. This is a random number generator\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "greek2vec = w2v.Word2Vec(\n",
    "    sg=1,\n",
    "    seed=seed,\n",
    "    workers=num_workers,\n",
    "    size=num_features,\n",
    "    min_count=min_word_count,\n",
    "    window=context_size,\n",
    "    sample=downsampling\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "greek2vec.build_vocab(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_count = sum([len(sentence) for sentence in corpus])\n",
    "print('The corpus contains {0:,} tokens'.format(token_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "#train model on sentneces, this may take a while to process\n",
    "greek2vec.train(corpus, total_examples=len(corpus), epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save and Load Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save model\n",
    "if not os.path.exists(\"Desktop/Word2Vec_koine_greek-master\"):\n",
    "    os.makedirs(\"Desktop/Word2Vec_koine_greek-master\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "greek2vec.save(os.path.join(\"Desktop/Word2Vec_koine_greek-master\", \"greek2vec.w2v\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load model\n",
    "greek2vec = w2v.Word2Vec.load(os.path.join(\"Desktop/Word2Vec_koine_greek-master\", \"greek2vec.w2v\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Perform Dimensionality Reduction with t-SNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#squash dimensionality to 2-dimensions\n",
    "#https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm\n",
    "tsne = sklearn.manifold.TSNE(n_components=2, random_state=0)\n",
    "#, perplexity=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#put it all into a giant matrix\n",
    "all_word_vectors_matrix = greek2vec.wv.syn0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "#train t sne\n",
    "all_word_vectors_matrix_2d = tsne.fit_transform(all_word_vectors_matrix)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}