ir_phase-3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cpOo1QnuBKbF",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div style=\"direction:rtl;line-height:300%;\">\n",
        "<font face=\"XB Zar\" size=5>\n",
        "<div align=center>\n",
        "<font face=\"B Titr\" size=5>\n",
        "<p></p><p></p>\n",
        "<p></p>\n",
        "</font>\n",
        "<p></p>\n",
        "<font>\n",
        "<br>\n",
        "Modern Information Retrieval course\n",
        "<br>\n",
        "lecturer: Dr. Soleymani\n",
        "</font>\n",
        "<p></p>\n",
        "<br>\n",
        "<font>\n",
        "<b>\n",
        "project third phase\n",
        "</b>\n",
        "</font>\n",
        "<br>\n",
        "<br>\n",
        "<br>\n",
        "<font>\n",
        "<b>Amir Hossein Rahmati</b>\n",
        "<br>\n",
        "<font>\n",
        "Sharif University of Technology\n",
        "<br>\n",
        "Computer Engineering department\n",
        "<br>\n",
        "<br>\n",
        "</font>\n",
        "</div>\n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poFt-P2DBKbI",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div>\n",
        "<font size=5>\n",
        "    <h1>\n",
        "    <b>introduction</b>\n",
        "    </h1>\n",
        "    <p></p>\n",
        "    <p></p>\n",
        "</font>\n",
        "<font size=3>\n",
        "    <br>\n",
        "    In this phase of the project, our focus will be on crawling and analyzing articles extracted from the Internet. We will start by examining various web crawling techniques to extract articles and other relevant information from the web.\n",
        "    <br>\n",
        "    In the next step, we will apply link analysis algorithms such as PageRank and HITS to determine the importance of these articles based on quotes, references, or other forms of links. We will also learn how to implement a personalized PageRank algorithm that takes user preferences into account to provide more relevant results.\n",
        "    <br>\n",
        "</font>\n",
        "</div>\n",
        "   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "HG8NPZNLBKbJ",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div >\n",
        "<font size=4>\n",
        "    <h1>\n",
        "    <b>\n",
        "    Crawler Implementation\n",
        "    </b>\n",
        "    </h1>\n",
        "</font>\n",
        "    <br>\n",
        "\n",
        "<font size=3>\n",
        "   translate to English:\n",
        "    In this section, you need to implement a Crawler to extract information from a number of articles from the <a href=\"https://www.semanticscholar.org/\">Semantic Scholar</a> website.\n",
        "    The extracted information must contain the following items.\n",
        "</font>\n",
        "</div>\n",
        "<br>\n",
        "<table dir=\"ltr\" style=\"width: 100%; border-collapse: collapse;\">\n",
        "  <tr>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">ID</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Title</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Abstract</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Publication Year</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Authors</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Related Topics</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Citation Count</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Reference Count</th>\n",
        "    <th style=\"padding: 8px; text-align: justify; border: 1px solid black;\">References</th>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Unique ID of the paper</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Title of the paper</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Abstract of the paper</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Publication year</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">Name of the first author, ..., Name of the last author</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">topic1, topic2, ...</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">number of citations of the paper</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">number of references of the paper</td>\n",
        "    <td style=\"padding: 8px; text-align: justify; border: 1px solid black;\">ID of the first reference, ..., ID of the tenth reference</td>\n",
        "  </tr>\n",
        "</table>\n",
        "    <br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "Pq95xPLbBKbK",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div>\n",
        "<font size=3>\n",
        "    translate to English:\n",
        "    Start the extraction process from 5 articles of each professor and add the first 10 references of each article to the article queue.\n",
        "    Continue the extraction process until you have information on 2000 articles.\n",
        "    Save the article information in the crawled_paper_profName.json file.\n",
        "\n",
        "</font>\n",
        "</div>\n",
        "\n",
        "<div >\n",
        "\n",
        "<font size=3>\n",
        "In implementing the Crawler, pay attention to the following.\n",
        "<ul>\n",
        "<li>You do not have the right to use the semantic scholar site api.</li>\n",
        "<li>For crawling, you can use packages such as <a href=\"https://www.selenium.dev/selenium/docs/api/py/\">Selenium</a> or <a href=\"https://gihub.com/scrapy/scrapy\">Scrapy</a>. The use of other packages is also allowed. You can also use the <a href=\"https://pypi.org/project/beautifulsoup4/\">Beautiful Soup</a> package to parse the extracted information.</li>\n",
        "<li>Give a few seconds of delay between each request to the site.</li>\n",
        "<li>At the time of delivery, your Crawler will be executed and its correctness will be checked.</li>\n",
        "<li>If your Crawler encounters an error such as request timeout, it should not stop its work.</li>\n",
        "</ul>\n",
        "\n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "dAgtJiGmhLUR"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "def get_html(url):\n",
        "    payload = {'api_key': 'c30935129c107b958a0ff0096be3994a' , 'url' : url}\n",
        "    return requests.get('http://api.scraperapi.com', params=payload)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "0pHKj5tDh4xN"
      },
      "outputs": [],
      "source": [
        "from bs4 import BeautifulSoup\n",
        "import re\n",
        "def get_data(explored_urls, urls, url, html):\n",
        "  ID = url.split(\"/\")[-1]\n",
        "\n",
        "  # title , year , authors\n",
        "  doc = BeautifulSoup(html.text, 'html.parser')\n",
        "  text = doc.find('pre', {'class': 'bibtex-citation'}).text\n",
        "  title = re.search('title={(.+)}', text).group(1)\n",
        "  year = int(re.search('year={(.+)}', text).group(1))\n",
        "  authors = \" , \".join(re.search('author={(.+)}', text).group(1).split(' and '))\n",
        "  # abstract\n",
        "  try:\n",
        "      abstract = doc.find('span', {'data-test-id': 'text-truncator-text'}).string\n",
        "  except:\n",
        "      abstract = \"\"\n",
        "\n",
        "  # related_topics\n",
        "  try:\n",
        "      related_topics = doc.find_all('li', {'class':'paper-meta-item'})[2].string\n",
        "  except:\n",
        "      related_topics = \"\"\n",
        "\n",
        "  # citation_count and references_count\n",
        "  counts_data = doc.findAll('h2', {'class': 'dropdown-filters__result-count__header dropdown-filters__result-count__citations'})\n",
        "  citations_count = 0\n",
        "  references_count = 0\n",
        "  for data in counts_data:\n",
        "    try:\n",
        "      data_splits = data.string.split()\n",
        "      if data_splits[1] == \"Citations\":\n",
        "          citations_count = int(data_splits[0])\n",
        "      elif data_splits[1] == \"References\":\n",
        "        references_count = int(data_splits[0])\n",
        "    except:\n",
        "      pass\n",
        "\n",
        "  # references\n",
        "    references = doc.findAll(\n",
        "      'a', {\n",
        "          'data-test-id': 'citation-paper-title',\n",
        "          'data-heap-citation-type': 'citedPapers',\n",
        "      })\n",
        "\n",
        "    for reference in references:\n",
        "        if ('https://www.semanticscholar.org' + reference['href']) not in explored_urls:\n",
        "          if ('https://www.semanticscholar.org' + reference['href']) not in urls:\n",
        "            urls.append('https://www.semanticscholar.org' + reference['href'])\n",
        "\n",
        "    paper = {'ID': ID, 'Title': title, 'Abstract': abstract, 'Publication Year': year, 'Authors': authors, 'Related Topics': related_topics\n",
        "               , 'Citation Count': citations_count, 'Reference Count':references_count, 'References':[r['data-heap-paper-id'] for r in references]}\n",
        "\n",
        "    return paper"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "RmnV07kbBL0o"
      },
      "outputs": [],
      "source": [
        "prof_names = [\"Kasaei\", \"Rabiee\", \"Rohban\", \"Sharifi\", \"Soleymani\"]\n",
        "URLS = {}\n",
        "for prof_name in prof_names:\n",
        "  with open(prof_name + \".txt\") as file:\n",
        "      URLS[prof_name] = file.readlines()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9P2PoSf5b3R1",
        "outputId": "55c31c21-a095-4461-b9c1-e05310195a2c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 25/25 [02:03<00:00,  4.94s/it]\n",
            "100%|██████████| 25/25 [03:52<00:00,  9.32s/it]\n",
            "100%|██████████| 25/25 [03:22<00:00,  8.09s/it]\n",
            "100%|██████████| 25/25 [02:54<00:00,  6.99s/it]\n",
            "100%|██████████| 25/25 [03:36<00:00,  8.67s/it]\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "from tqdm import tqdm\n",
        "for prof_name in prof_names:\n",
        "    crawled_papers = []\n",
        "    urls = URLS[prof_name]\n",
        "    explored_urls = []\n",
        "    for i in tqdm(range(25)):\n",
        "      if len(urls) == 0:\n",
        "        break\n",
        "      url = urls.pop(0)\n",
        "      html = get_html(url)\n",
        "      paper = get_data(explored_urls, urls, url, html)\n",
        "      crawled_papers.append(paper)\n",
        "      explored_urls.append(url)\n",
        "    with open('crawled_paper_' + prof_name + '.json', 'w') as j:\n",
        "        json.dump(crawled_papers, j)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "EpL8mu0FBKbK",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div >\n",
        "<font  size=4>\n",
        "    <h1>\n",
        "    <b>\n",
        "    personalized PageRank\n",
        "    </b>\n",
        "    </h1>\n",
        "</font>\n",
        "    <br>\n",
        "<font size=3>\n",
        "In this section, we implement the personalized PageRank algorithm, an extension of the PageRank algorithm that takes user preferences into account. The personalized PageRank algorithm ranks nodes in a graph based on their importance to the user, not based on their overall importance in the graph.\n",
        "\n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "id": "6dUuvXjqBKbL",
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from typing import List , Dict\n",
        "def compute_p(graph):\n",
        "    p = np.zeros((len(graph),len(graph)))\n",
        "    for i in range(len(graph)):\n",
        "      ref_n = len(graph[list(graph.keys())[i]])\n",
        "      if ref_n == 0 :\n",
        "        p[i] = np.ones(len(graph)) / len(graph)\n",
        "      else:\n",
        "        p[i] = np.array(list(map(lambda x: 1 if x else 0, [j in graph[list(graph.keys())[i]] for j in graph.keys()]))) / ref_n\n",
        "\n",
        "    return p\n",
        "\n",
        "def find_steady_a(a,p):\n",
        "    dif = 1\n",
        "    while dif > 0.000000001:\n",
        "      a_hat = np.dot(a,p)\n",
        "      dif = np.linalg.norm(a_hat - a)\n",
        "      a = a_hat\n",
        "\n",
        "    return a_hat\n",
        "\n",
        "\n",
        "def pagerank(graph: Dict[str, List[str]]) -> Dict[str, float]:\n",
        "    \"\"\"\n",
        "    Returns the personalized PageRank scores for the nodes in the graph, given the user's preferences.\n",
        "\n",
        "    Parameters:\n",
        "    graph (Dict[str, List[str]]): The graph represented as a dictionary of node IDs and their outgoing edges.\n",
        "\n",
        "    Returns:\n",
        "    Dict[str, float]: A dictionary of node IDs and their personalized PageRank scores.\n",
        "    \"\"\"\n",
        "\n",
        "    a = np.zeros(len(graph))\n",
        "    a[0:5] = 1/5\n",
        "    p = compute_p(graph)\n",
        "    a = find_steady_a(a,p)\n",
        "    return dict(zip(list(graph.keys()) , a))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "9wX4NpIsBKbM",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div >\n",
        "<font size=3>\n",
        "In this section, we use the personalized PageRank algorithm implemented in the previous section to identify important articles related to a specific professor's field of work. This function takes a field as input and outputs the top articles that have the most connections to that field.\n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "id": "OxVUBCYgBKbM",
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "def important_articles(Professor: str) -> List[str]:\n",
        "    \"\"\"\n",
        "    Returns the most important articles in the field of given professor, based on the personalized PageRank scores.\n",
        "\n",
        "    Parameters:\n",
        "    Professor (str): Professor's name.\n",
        "\n",
        "    Returns:\n",
        "    List[str]: A list of article IDs representing the most important articles in the field of given professor.\n",
        "    \"\"\"\n",
        "    graph = {}\n",
        "    with open(f'crawled_paper_{Professor}.json','r') as f:\n",
        "      crawled_papers = json.load(f)\n",
        "    for paper in crawled_papers:\n",
        "      graph[paper['ID']] = paper['References']\n",
        "      for ref in paper['References']:\n",
        "        graph.setdefault(ref, [])\n",
        "\n",
        "    im_papers = sorted(pagerank(graph).items(), key = lambda x: x[1], reverse=True)[:10]\n",
        "    return im_papers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "32Zkdy8uU_19",
        "outputId": "51bc8062-8e26-422f-a066-e767def94cde"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "paper_id = 4b1a47709d0546e5bc614bf9a521c550e6881d04 score = 0.00529375052118805\n",
            "paper_id = 505f48d8236eb25f871da272c2ac2fe4b41ea289 score = 0.005206013837088638\n",
            "paper_id = eda3368a5198ca55768b07b6f5667aea28baf2cd score = 0.005107287548373918\n",
            "paper_id = 201631fbc3f7d7cb2c1ddfaf82278cad5e44f2f9 score = 0.004370070110121608\n",
            "paper_id = b7d540cd0de72e984cdec44afa4a4d039cfd5eea score = 0.004301336099850455\n",
            "paper_id = bfba194dfd9c7c27683082aa8331adc4c5963a0d score = 0.003690278561280304\n",
            "paper_id = 9926020dda21874dc7a5ef1511bae6c4cef5ecb9 score = 0.0036640691401314874\n",
            "paper_id = 6767812e114c426d45ea83894b156f7906e525cd score = 0.0033284967523998037\n",
            "paper_id = c63a34ac6a4e049118070e707ca7679fbb132d33 score = 0.003275420181468577\n",
            "paper_id = 966aad492f75b17f698e981e008b73b51816c6aa score = 0.0031814144385978135\n"
          ]
        }
      ],
      "source": [
        "im_papers = important_articles(\"Kasaei\")\n",
        "for i in im_papers:\n",
        "  print(\"paper_id = \" + i[0] + \" score = \" + str(i[1]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "KEFesAONBKbO",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<div  style=\"text-align: justify\">\n",
        "<font  size=4>\n",
        "    <h1>\n",
        "    <b>\n",
        "    autohrs ranking\n",
        "    </b>\n",
        "    </h1>\n",
        "</font>\n",
        "    <br>\n",
        "<font size=3>  \n",
        "The concept of authors referring to each other is used to rank authors. When author A refers to article P, which author B is one of the authors of that article, i.e., article P, we say that author A has referred to author B. With this relationship, we can create a graph of references between authors and then use the HITS algorithm to rank authors. To rank, we need to use hub and authority indices.\n",
        "\n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UBdaoPEaBKbO",
        "outputId": "e59d5931-dfcb-4432-b668-3800e2396b9f",
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[('B. Mirheidari', 0.33333333333333476), ('H. Christensen', 0.33333333333333476), ('Yilin Pan', 0.23385403957214246), ('D. Blackburn', 0.09947929376119231), ('S. Orimaye', 2.862767636958744e-15), ('Jojo Sze-Meng Wong', 2.3707264061142116e-15), ('Judyanne Sharmini Gilbert Fernandez', 7.947068100655231e-16), ('Ireneous N. Soyiri', 7.947068100655231e-16), ('Karen J. Golden', 7.813127859831653e-16), ('Zeynep Akata', 6.107033129827701e-16)]\n"
          ]
        }
      ],
      "source": [
        "import networkx as nx\n",
        "import networkx as nx\n",
        "\n",
        "\n",
        "def hit_algorithm(papers, n):\n",
        "    \"\"\"\n",
        "        Implementing the HITS algorithm to score authors based on their papers and co-authors.\n",
        "\n",
        "        Parameters\n",
        "        ---------------------------------------------------------------------------------------------------\n",
        "        papers: A list of paper dictionaries with the following keys:\n",
        "                \"id\": A unique ID for the paper\n",
        "                \"title\": The title of the paper\n",
        "                \"abstract\": The abstract of the paper\n",
        "                \"date\": The year in which the paper was published\n",
        "                \"authors\": A list of the names of the authors of the paper\n",
        "                \"related_topics\": A list of IDs for related topics (optional)\n",
        "                \"citation_count\": The number of times the paper has been cited (optional)\n",
        "                \"reference_count\": The number of references in the paper (optional)\n",
        "                \"references\": A list of IDs for papers that are cited in the paper (optional)\n",
        "        n: An integer representing the number of top authors to return.\n",
        "\n",
        "        Returns\n",
        "        ---------------------------------------------------------------------------------------------------\n",
        "        List\n",
        "        list of the top n authors based on their hub scores.\n",
        "    \"\"\"\n",
        "    # Create a graph of authors and papers (all of the authors and papers represented as nodes, and all of the authors who wrote each paper connected to the corresponding paper node by an edge)\n",
        "    G = nx.DiGraph()\n",
        "\n",
        "    # Run the HITS algorithm\n",
        "    hubs, authorities = nx.hits(G)\n",
        "\n",
        "    # Create a list of top n authors based on their hub scores\n",
        "\n",
        "    for paper in papers:\n",
        "      authors = paper['Authors']\n",
        "      for author in authors:\n",
        "        G.add_edge(author, paper['ID'])\n",
        "\n",
        "    hubs, authorities = nx.hits(G)\n",
        "\n",
        "    top_authors = sorted(hubs.items(), key=lambda x: x[1], reverse=True)[:n]\n",
        "    return top_authors\n",
        "\n",
        "# call the hit_algorithm function\n",
        "with open('crawled_paper_' + prof_name + '.json') as f:\n",
        "  papers = json.load(f)\n",
        "  print(hit_algorithm(papers, 10))\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "vscode": {
      "interpreter": {
        "hash": "08ac30a6a1fd2e576b33e03f7d61c3a285d7ee0582c2dd23dde6343ef303ebe9"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}