Skip to content

Overview

DeadlyCoconuts edited this page Jun 13, 2019 · 7 revisions

How does this all work?

overview

This programme essentially takes in a user request from the client, and uses the input parameters such as the source URL to begin scraping pages on Wikipedia on the server-side. Once all the web scraping has been done, the programme then processes the data it has collected, filtering out unnecessary or repeated information. These data is then channeled back to the client in the form of JSON objects, which would be used to generate an interactive graph on the browser.

On a deeper level...

overview2

Back-end

The main bulk (if not all) of the back-end is written in Python. The main web scraping functionality of this programme uses libraries such as Requests and Beautiful Soup for making HTTP requests and parsing HTML documents, whereas Flask is used to as a framework to handle almost everything that concerns server back-end tasks.

Front-end

The front-end, on the other hand, is written in the usual HTML-Javascript-CSS mix. In particular, the creation of graphs is handled by the D3.js library. Jinja is also used sparingly to generate templates for the web application.