Skip to content

INN/current-ltw-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Current Local That Works database scraper

This aims to scrape this page (embedded here) to turn it into a CSV.

Uses https://mircozeiss.com/json2csv/ for creating the CSV.

Issues and time-tracking

Running this

  1. Clone this repo
  2. Run make
  3. In the browser window opened by the scrape target, open your browser's dev tools.
  4. Find the console log item that is the contents of a CSV. Save it.

Other make targets:

  • make clean removes all files generated by make
  • make download redownloads the database
  • make injected inserts our scraping JS into the downloaded page and performs some text replacements that are undone by injection.js
  • make scrape opens the post-injection page in the browser

Assumptions

  • you're on a Mac with all the Homebrew dependencies, including make and open
    • Linux users may be able to alias open=xdg-open