Git scraping: track changes over time by scraping to a Git repository
Simon Willison on gitscraping git-scraping (Github Topics) track changes over time by scraping to a Git repository | Hacker News Raspando dados com o GitHub Actions e analisando com Datasette
- simonw/scrape-open-data
- simonw/ca-fires-history
- simonw/cdc-vaccination-history
- simonw/disaster-scrapers
- simonw/scrape-roads-dot-ca-gov
reads through the entire history of a file and generates a SQLite database reflecting changes to that file over time. git-history: a tool for analyzing scraped data collected using Git and SQLite git-history - a tool for Datasette
datadesk/california-coronavirus-scrapers The open-source web scrapers that feed the Los Angeles Times California coronavirus tracker.
githubocto/flat-demo-bitcoin-price: A Flat Data GitHub Action demo repo pierrotsmnrd/flat_data_py_example: How to use Python in Github's Flat Data https://github.com/githubocto?q=flat-demo
swyxio/gh-action-data-scraping
- cmteb see also: web.archive.org/ .. harta_stare_sistem_termoficare_bucuresti.php
- situatie-drumuri
- http://posturi.gov.ro/
- https://data.gov.ro/dataset watch news?
- https://data.gov.ro/dataset/mecanismul-de-feed-back-al-pacientului-2023 read latest json - check weekly
- CKAN analysis see fetch data.gov.ro.docx'
- https://extranet.brasovcity.ro/MapServer2/WebGis2/wgd/getmap.aspx https://extranet.brasovcity.ro/MapServer2/WebGis2/wgd/
Structured data.gov.ro monthly dumps versioned as: https://data.gov.ro/dataset/activity/mecanismul-de-feed-back-al-pacientului-2023 check docs w multiple versions
Prometeu
- read target into local (to repository) json
- cmteb
- andnet
- add meaningful commit msgs
- save as csv
- check if changed before saving
- write to external repo
- build to datasette, FlatGithub
- get historical data