Skip to content

Version 2 of the previous graph mapper. This is a web application designed to visualise the links between different Wikipedia pages. Upon being provided with a single source link, this application scrapes the relevant hyperlinks on the webpage, in a manner specified by the user, and displays all the links in an interactive and dynamic graph.

Notifications You must be signed in to change notification settings

deadlycoconuts/wikiweb2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

44 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Wikipedia Graph Mapper

This is a web application designed to visualise the links between different Wikipedia pages. Try it out at: http://www.wikiweb.appspot.com

Background

This project is initiated in 2019 at CentraleSupélec, and has been undertaken by Zi Yi EWE and supervised by Professor Christian CHABRERIE. It has been carried out to fulfil the Projet Innovation (Innovation Project) course requirement in Semester 8 of the second year of the Ecole Centrale Engineering programme.

Table of Contents

How to use

  1. Begin by entering the URL of your desired source Wikipedia page into the the first field.
  2. Enter the maximum search depth level desired*.
  3. Select a search mode. There are 3 search modes available:
    • Search for links from the 1st paragraph
      Uses links found within the 1st paragraph (of the introduction) of each Wikipedia page to generate a graph.
    • Search for links from the introduction
      Uses links found within the introduction (may contain multiple paragraphs) of each Wikipedia page to generate a graph.
    • Search for links from the entire page
      Uses links found within the entire page to generate a graph.
  4. Click on 'Create my graph!'.
  5. Voilà!

*Unforunately, the maximum search level that works on the server which hosts this application is 2. Nonetheless, you should be able to run the programme to your heart's content at any search depth on a development server.

Features

  • Scrapes URLs found within the first paragraph/introduction/entire document of a Wikipedia page, and repeats the process for each subsequent link found
  • Offers customisation of the search depth and breadth level of the scraping
    • Breadth: Select the relevant section of each Wikipedia page to look for URLs
    • Depth: Enter the maximum search depth of the graph from the source node
  • Displays the interconnections between these URLs found in an interactive graph
    • Click on a node to display only its neighbours
    • Click on the same node again to revert back to the original display
    • Drag the nodes around to rearrange and interact with them
  • Supported by mobile browsers on touch devices with smaller screen resolutions
    • Resizing of webpages
    • Modification of the graph to pack nodes closer to one another to fit narrower screen dimensions

Installation

To run the application locally, follow the steps below:

  1. Clone the repository on your local device.
    git clone https://github.com/DeadlyCoconuts/wikiweb2/

  2. Create a local environment to install the dependencies defined in requirements.txt
    If you do not already have virtualenv installed:
    pip install virtualenv
    pip install virtualenvwrapper-win

    To create a virtual environment, go to the project directory and enter:
    virtualenv [name of your virtual environment]

    To activate your virtual environment:
    source [name of your virtual environment]/bin/activate

    To install all dependencies indicated in requirements.txt:
    pip install -r requirements.txt

  3. Ensure that line 99 of your local copy of process_link.py has been commented out.
    #requests_toolbelt.adapters.appengine.monkeypatch()

  4. Run your local copy of main.py.
    python main.py

  5. Visit your local application on your browser at http://127.0.0.1:5000.

Full Documentation

Refer to the wiki of this repository at https://github.com/DeadlyCoconuts/wikiweb2/wiki for more details.

About

Version 2 of the previous graph mapper. This is a web application designed to visualise the links between different Wikipedia pages. Upon being provided with a single source link, this application scrapes the relevant hyperlinks on the webpage, in a manner specified by the user, and displays all the links in an interactive and dynamic graph.

Resources

Stars

Watchers

Forks

Packages

No packages published