NameDisambiguation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author Name Disambiguation Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Nickname on **biendata.com**: **Dracarys**\n",
    "+ Rank: 4 (by 2020/02/22)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Report\n",
    "\n",
    "---\n",
    "### introduction\n",
    "This project focus on author name disambiguation in academic publications. The target is to disambiguate the author names by publicated papers' attributes, such as title, co-author, organization and keywords.\n",
    "\n",
    "### method\n",
    "The flow of this project is quite straightforward: select suitable attuributes of papers to construct feature vectors, the do clustering.  \n",
    "\n",
    " + **data analysis and feature selection**: \n",
    "In the given data set of papers, several attributes are provided as string: title, co-authors and their organizations, keywords, abstract, venue, publication year .,etc. All these properties may provide useful infomation to seprate authors with the same name. In this project, some of these attributes are selected to construct the feature vectors. \n",
    " \n",
    " + **feature vectorization**: \n",
    "In this experiment, [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) algorithm is used to convert features into numerical vectors. This algorithm is proved to be effective in extracting unique infomations across different documents, which is very similar to this project. \n",
    "\n",
    "+ **clustering**\n",
    "Since the fact that how many differnt authors shared the same name is not known, th unsupervised clustering method [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN) is used. This algorithm does not require user to set the number of clusters, make it suitable for this experiment.\n",
    "\n",
    "### experiment\n",
    "\n",
    "Data prepreocessing is necessary for this project. Some of the attributes are missing, some of them are not in English. For the author's name, different format also co-exists, **\"-\", \"_\", \".\" or \" \"** all appeared in authors name as seprator. For the origanization name, some abbreviation also used along its full name. All these problems need to be addressed. Therefore, some of the preprocess techniques are used to clean the data. Most of the data clean functions are from these [shared notebooks](https://biendata.com/models/index_category/97/). \n",
    "\n",
    "The cleaned attributes(co-author, organization, title, keywords) is then concatenate into a long feature string. All these feature strings are added to a list, then the list is converted to numerical feature vectors. The feature vectors are feeded into DBSCAN to generate the cluster results.\n",
    "\n",
    "### result\n",
    "In the experiment we tried different combinations of  paper's attirbutes(title, co-authors and their organizations, keywords, abstract, venue, publication year), and the final score are around 0.25 ~ 0.3. The best combination is co-authors, their organizations and abstract. \n",
    "\n",
    "### summary\n",
    "Due to limted knowledge in NLP, this experiment result is not very good overall. New methods are expected to be tested.\n",
    "\n",
    "### reference:\n",
    "[shared notebook](https://biendata.com/models/category/3000/L_notebook/)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import json\n",
    "import re\n",
    "import string\n",
    "from collections import defaultdict\n",
    "from pprint import pprint\n",
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn.metrics.pairwise import pairwise_distances\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## global variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_AUTHOR = \"./data/train/train_author.json\"\n",
    "TRAIN_PUB = \"./data/train/train_pub.json\"\n",
    "SNA_VALID_AUTHOR_RAW = \"./data/sna_data/sna_valid_author_raw.json\"\n",
    "SNA_VALID_EXAMPLE = \"./data/sna_data/sna_valid_example_evaluation_scratch.json\"\n",
    "SNA_VALID_PUB = \"./data/sna_data/sna_valid_pub.json\"\n",
    "\n",
    "\n",
    "train_author_data = load_json(TRAIN_AUTHOR)\n",
    "train_pub_data = load_json(TRAIN_PUB)\n",
    "test_author_data = load_json(SNA_VALID_AUTHOR_RAW)\n",
    "test_pub_data = load_json(SNA_VALID_PUB)\n",
    "example_output = load_json(SNA_VALID_EXAMPLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_json(file_path):\n",
    "    with open(file_path, \"r\") as f:\n",
    "        return json.load(f)\n",
    "\n",
    "def preprocess_name(name):   \n",
    "    name = name.lower().replace(' ', '_')\n",
    "    name = name.replace('.', '_')\n",
    "    name = name.replace('-', '')\n",
    "    name = re.sub(r\"_{2,}\", \"_\", name) \n",
    "    return name\n",
    "\n",
    "\n",
    "def preproces_org(org):\n",
    "    if org != \"\":\n",
    "        org = org.replace('Sch.', 'School')\n",
    "        org = org.replace('Dept.', 'Department')\n",
    "        org = org.replace('Coll.', 'College')\n",
    "        org = org.replace('Inst.', 'Institute')\n",
    "        org = org.replace('Univ.', 'University')\n",
    "        org = org.replace('Lab ', 'Laboratory ')\n",
    "        org = org.replace('Lab.', 'Laboratory')\n",
    "        org = org.replace('Natl.', 'National')\n",
    "        org = org.replace('Comp.', 'Computer')\n",
    "        org = org.replace('Sci.', 'Science')\n",
    "        org = org.replace('Tech.', 'Technology')\n",
    "        org = org.replace('Technol.', 'Technology')\n",
    "        org = org.replace('Elec.', 'Electronic')\n",
    "        org = org.replace('Engr.', 'Engineering')\n",
    "        org = org.replace('Aca.', 'Academy')\n",
    "        org = org.replace('Syst.', 'Systems')\n",
    "        org = org.replace('Eng.', 'Engineering')\n",
    "        org = org.replace('Res.', 'Research')\n",
    "        org = org.replace('Appl.', 'Applied')\n",
    "        org = org.replace('Chem.', 'Chemistry')\n",
    "        org = org.replace('Prep.', 'Petrochemical')\n",
    "        org = org.replace('Phys.', 'Physics')\n",
    "        org = org.replace('Phys.', 'Physics')\n",
    "        org = org.replace('Mech.', 'Mechanics')\n",
    "        org = org.replace('Mat.', 'Material')\n",
    "        org = org.replace('Cent.', 'Center')\n",
    "        org = org.replace('Ctr.', 'Center')\n",
    "        org = org.replace('Behav.', 'Behavior')\n",
    "        org = org.replace('Atom.', 'Atomic')\n",
    "        #org = org.split(';')[0]\n",
    "        org = ' '.join(org.split(';'))\n",
    "        org = org.lower()\n",
    "    return org\n",
    "\n",
    "\n",
    "def remove_stopwords(content):\n",
    "    use_stopwords = set(stopwords.words('english'))\n",
    "    stemmer = WordNetLemmatizer()\n",
    "    content = [stemmer.lemmatize(word) for word in content.split()\n",
    "                   if word not in use_stopwords and len(word) > 1]\n",
    "    return ' '.join(content)\n",
    "\n",
    "def remove_non_english_words(content):\n",
    "    return \" \".join(word for word in nltk.wordpunct_tokenize(content) \\\n",
    "         if word.lower() in ENGLISH_WORDS or not word.isalpha())\n",
    "\n",
    "\n",
    "def remove_non_printable_words(content):\n",
    "    result = \"\"\n",
    "    printable_set = set(string.printable)\n",
    "    for c in content:\n",
    "        if c in printable_set:\n",
    "            result += c\n",
    "    \n",
    "    return result\n",
    "\n",
    "    \n",
    "#remove seprators by regular expression\n",
    "def etl(content):\n",
    "    content = re.sub(\"[\\s+\\.\\!\\/,;$%^*(+\\\"\\')]+|[+——()?【】“”！，。？、~@#￥%……&*（）]+\", \" \", content)\n",
    "    content = re.sub(r\" {2,}\", \" \", content)\n",
    "    return content\n",
    "\n",
    "def get_org(co_authors, author_name):\n",
    "    for au in co_authors:\n",
    "        name = precessname(au['name'])\n",
    "        name = name.split('_')\n",
    "        if ('_'.join(name) == author_name or '_'.join(name[::-1]) == author_name) and 'org' in au:\n",
    "            return au['org']\n",
    "    return ''\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### disambiguation: TF-IDF "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [01:52<00:00,  2.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result saved.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "def disambiguate():\n",
    "    result_dict = {}\n",
    "    \n",
    "    for author in tqdm(test_author_data.keys()):\n",
    "        #print(f\"{author}, \", end = \"\")\n",
    "        \n",
    "        feature_string_list = []\n",
    "        papers = test_author_data[author]\n",
    "        if len(papers) == 0:\n",
    "            result_dict[author] = []\n",
    "            continue\n",
    "        \n",
    "        #print(\"paper count: \", len(papers))\n",
    "        paper_dict = {}\n",
    "        for paper in papers:\n",
    "           \n",
    "            feature_list = []\n",
    "            \n",
    "            for k in test_pub_data[paper][\"authors\"]:\n",
    "                if \"name\" in k:\n",
    "                    feature_list.append(preprocess_name(k[\"name\"]))\n",
    "                \n",
    "                if \"org\" in k:\n",
    "                    feature_list.append(preproces_org(k[\"org\"]))                    \n",
    "\n",
    "                    \n",
    "            '''\n",
    "            if \"venue\" in k:\n",
    "                    feature_list.append(k[\"venue\"])\n",
    "            '''\n",
    "            \n",
    "            '''\n",
    "            if \"title\" in k:\n",
    "                if train_pub_data[paper][\"title\"] is not None: \n",
    "                    feature_list.append(train_pub_data[paper][\"title\"].lower())\n",
    "            '''\n",
    "            \n",
    "            if \"abstract\" in test_pub_data[paper]:\n",
    "                feature_list.append(test_pub_data[paper][\"abstract\"].lower())            \n",
    "            \n",
    "            '''\n",
    "            if \"keywords\" in test_pub_data[paper]:\n",
    "                feature_list += test_pub_data[paper][\"keywords\"]\n",
    "            '''\n",
    "            \n",
    "            feature_string = etl(\" \".join(feature_list)).lower()\n",
    "            #print(feature_string)\n",
    "            feature_string = remove_stopwords(feature_string)\n",
    "            #print(\"before:\\n\", feature_string)\n",
    "            feature_string = remove_non_printable_words(feature_string)\n",
    "            \n",
    "            feature_string_list.append(feature_string)\n",
    "            \n",
    "        \n",
    "        #print(\"feature string list length:\", len(feature_string_list))\n",
    "        vectorizer = TfidfVectorizer()\n",
    "        tfidf = vectorizer.fit_transform(feature_string_list)\n",
    "       \n",
    "        clf = DBSCAN(metric=\"cosine\", min_samples = 3)\n",
    "        s = clf.fit_predict(tfidf)\n",
    "      \n",
    "        \n",
    "        for label, paper in zip(clf.labels_, papers):\n",
    "            if str(label) not in paper_dict:\n",
    "                paper_dict[str(label)] = [test_pub_data[paper][\"id\"]]\n",
    "            else:\n",
    "                paper_dict[str(label)].append(test_pub_data[paper][\"id\"])\n",
    "            \n",
    "        #pprint(paper_dict)\n",
    "        result_dict[author] = list(paper_dict.values())\n",
    "    \n",
    "    f = open(\"./result/result_0223.json\", \"w\", encoding='utf-8')\n",
    "    json.dump(result_dict, f, indent=4)\n",
    "    f.close()\n",
    "    print(\"result saved.\")\n",
    "                \n",
    "               \n",
    "                    \n",
    "                \n",
    "disambiguate()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}