PHI augmentation-i2b2-2006 dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PHI augmentation : i2b2-2006 dataset processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The i2b2 2014 de-identification dataset can be accessed from https://portal.dbmi.hms.harvard.edu.\n",
    "\n",
    "The data processing mainly refers the guidance to this link:\n",
    "https://github.com/juand-r/entity-recognition-datasets/tree/master/data/i2b2_2006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.Combining all the XML files, then randomly splitting the records into train, validation, test set with the proportion of 7:1:2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.Follow the guidance from above link to convert records from XML fromat to CONLL format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.Double check the sentences in file should be seperated by a single empty line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.Convert records from CONLL fromat to BIO format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = 'C:/Users/zhou/Desktop/i2b2_2006_train.xml.conll'\n",
    "file_path2 = 'C:/Users/zhou/Desktop/i2b2_2006_train_bio.txt'\n",
    "\n",
    "# The sentences in input file should be seperated by a single empty line.\n",
    "# CONLL fromat -->  BIO format\n",
    "\n",
    "reserve_content = []\n",
    "with open(file_path, 'r') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for row in range(1, len(content)):\n",
    "        new_line = ''\n",
    "        new_line2 = ''\n",
    "        \n",
    "        if len(content[row]) == 1:  # cope with empty line\n",
    "        \t# print('this a non ')\n",
    "        \tnext_line = '\\n'\n",
    "        \treserve_content.append(next_line)\n",
    "        else:\n",
    "            all_token_Previous = content[row-1].split()\n",
    "            all_token = content[row].split()\n",
    "      \n",
    "            if len(content[row-1]) != 1 and all_token[-1] != 'O':   # non-empty line & non-empty in previous line & this label not 'O'\n",
    "                if all_token[-1] != all_token_Previous[-1]:  # whether these two lines have same label \n",
    "                    new_label = 'B-' + all_token[-1]\n",
    "                if  all_token[-1] == all_token_Previous[-1]: \n",
    "            \t    new_label = 'I-' + all_token[-1]\n",
    "                all_token[-1] = new_label\n",
    "                for i in all_token:\n",
    "                    new_line = new_line+' '+ i\n",
    "                new_line = new_line+'\\n'\n",
    "                reserve_content.append(new_line)\n",
    "\n",
    "            if len(content[row-1]) != 1 and all_token[-1] == 'O':  # non-empty line & non-empty in previous line & this label is 'O'\n",
    "                    for i in all_token:\n",
    "                        new_line2 = new_line2+' '+ i\n",
    "                    new_line2 = new_line2 + '\\n'\n",
    "                    reserve_content.append(new_line2)\n",
    "\n",
    "            elif len(content[row-1]) == 1:   # non-empty line & empty line in previous\n",
    "                if all_token[-1] == 'O':   # this label is 'O'\n",
    "                    for i in all_token:\n",
    "                        new_line2 = new_line2+' '+ i\n",
    "                    new_line2 = new_line2 + '\\n'\n",
    "                    reserve_content.append(new_line2)\n",
    "                else:      # this label not 'O'\n",
    "                    new_label = 'B-' + all_token[-1]\n",
    "                    all_token[-1] = new_label\n",
    "                    for i in all_token:\n",
    "                        new_line2 = new_line2+' '+ i\n",
    "                    new_line2 = new_line2 + '\\n'\n",
    "                    reserve_content.append(new_line2)\n",
    "\n",
    "with open(file_path2, 'w') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.Replace the label of low frequency PHI type, namely 'B-AGE' and 'I-AGE', with 'O'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If want to conduct PHI augmentation, run the step 6,7; If not, skip to step 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.Conduct PHI augmentation, run the following code.\n",
    "Notice that we replace PHI entity in fine-grainity (in PHI type level)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "file_path_write = 'C:/Users/zhou/Desktop/1.all_PHI_from_internet/'\n",
    "labels_in_2006 = ['PATIENT', 'DOCTOR', 'HOSPITAL', 'ID', 'DATE', 'LOCATION', 'PHONE']\n",
    "label_content = [[] for i in range(len(labels_in_2006))]\n",
    "for k in range(len(labels_in_2006)):\n",
    "    file_path = file_path_write + labels_in_2006[k] + '.txt'\n",
    "    with open(file_path, 'r', encoding='utf-8') as f_w:\n",
    "        content = f_w.readlines()\n",
    "        for j in content:\n",
    "            label_content[k].append(j.strip())\n",
    "\n",
    "\n",
    "def replace_PHI(label, line_number):\n",
    "    preserve_list = []\n",
    "    if label in labels_in_2006:\n",
    "        label_index = labels_in_2006.index(label)\n",
    "        words_list = label_content[label_index]\n",
    "\n",
    "        random.seed(global_seed)  # every epoch, it produces different dataset, also reimplementable.\n",
    "        random.shuffle(words_list)\n",
    "        words_num = len(words_list)\n",
    "\n",
    "        random.seed(line_number)  # line_number is defined for specific PHI instance\n",
    "        random_index = random.randint(0, words_num - 1)\n",
    "        random_w = words_list[random_index].strip()\n",
    "\n",
    "        words_num = random_w.split()\n",
    "        if len(words_num) == 1:\n",
    "            new_line = random_w+'   '+'B-'+label\n",
    "            preserve_list.append(new_line)\n",
    "        if len(words_num) > 1:\n",
    "            for i in range(len(words_num)):\n",
    "                if i == 0:\n",
    "                    new_line = words_num[i]+'   '+'B-'+label\n",
    "                    preserve_list.append(new_line)\n",
    "                if i > 0:\n",
    "                    new_line = words_num[i]+'   '+'I-'+label\n",
    "                    preserve_list.append(new_line)\n",
    "    return preserve_list\n",
    "\n",
    "\n",
    "file_path_2014 = 'C:/Users/zhou/Desktop/i2b2_2006_train_bio.txt'\n",
    "global_seed = 2\n",
    "repalced_file_path = 'C:/Users/zhou/Desktop/new_replaced_2006_train_seed'+str(global_seed)+'.txt'\n",
    "\n",
    "\n",
    "reserve_content = []\n",
    "with open(file_path_2014, 'r', encoding='utf-8') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for row in range(0, len(content)-1):\n",
    "        if len(content[row]) == 1:  # cope with empty line\n",
    "            reserve_content.append('')\n",
    "        if len(content[row]) != 1 and len(content[row+1]) != 1:\n",
    "            all_token_Next = content[row + 1].split()\n",
    "            all_token = content[row].split()\n",
    "            this_line_label = all_token[-1]\n",
    "            next_line_label = all_token_Next[-1]\n",
    "            if this_line_label.startswith('O'):\n",
    "                new_line = all_token[0]+'   '+all_token[-1]\n",
    "                reserve_content.append(new_line)\n",
    "                continue\n",
    "            if this_line_label.startswith('B-') and next_line_label.startswith('I-'):\n",
    "                continue\n",
    "            if this_line_label.startswith('B-') and not next_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                new_words = replace_PHI(label, row)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "            if this_line_label.startswith('I-') and next_line_label.startswith('I-'):\n",
    "                continue\n",
    "            if this_line_label.startswith('I-') and not next_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                # print(label)\n",
    "                new_words = replace_PHI(label, row)\n",
    "                # print(new_words)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "\n",
    "        if len(content[row]) != 1 and len(content[row+1]) == 1:\n",
    "            all_token = content[row].split()\n",
    "            this_line_label = all_token[-1]\n",
    "            if this_line_label.startswith('B-') or this_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                new_words = replace_PHI(label, row)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "            if this_line_label.startswith('O'):\n",
    "                new_line = all_token[0]+'   '+all_token[-1]\n",
    "                reserve_content.append(new_line)\n",
    "                continue\n",
    "\n",
    "\n",
    "with open(repalced_file_path, 'w', encoding='utf-8') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.writelines(line+'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.Combine PHI into five PHI categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_contact = 'PHONE'\n",
    "label_location = ['LOCATION', 'HOSPITAL']\n",
    "label_name = ['PATIENT', 'DOCTOR']\n",
    "\n",
    "reserve_content = []\n",
    "file_path = 'C:/Users/zhou/Desktop/new_replaced_2006_train_seed2.txt'\n",
    "file_path_write = 'C:/Users/zhou/Desktop/2006_train.txt'\n",
    "\n",
    "with open(file_path, 'r', encoding='utf-8') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for line in content:\n",
    "        if len(line.split()) == 0 :  # preserve empty line\n",
    "            reserve_content.append(line)\n",
    "        else:\n",
    "            words = line.split()\n",
    "            label = words[-1]\n",
    "            if label == 'O':\n",
    "                reserve_content.append(line)\n",
    "            else:\n",
    "                label_type = label.split('-')[1]\n",
    "                if label_type in label_location:\n",
    "                    new_label = label.replace(label_type, 'LOCATION')  # B-HOSPITAL --> B-LOCATION\n",
    "                    new_line = line.replace(label, new_label)  # XXX B-LOCATION\n",
    "                    reserve_content.append(new_line)\n",
    "                elif label_type == label_contact:\n",
    "                    line = line.replace(label_type,'CONTACT')\n",
    "                    reserve_content.append(line)\n",
    "                elif label_type in label_name:\n",
    "                    new_label = label.replace(label_type, 'NAME')\n",
    "                    new_line = line.replace(label, new_label) \n",
    "                    reserve_content.append(new_line)\n",
    "                else:\n",
    "                    assert label_type in ['DATE', 'ID']\n",
    "                    reserve_content.append(line)\n",
    "\n",
    "with open(file_path_write, 'w', encoding='utf-8') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.If want to conduct context augmentation, turn to relevant step."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}