Skip to content

Commit

Permalink
Fix start script (#9)
Browse files Browse the repository at this point in the history
* added some more docs and removed strike through
* fixed bug in start up script
  • Loading branch information
LouiseMcMahon authored Jan 14, 2020
1 parent 1eb947c commit b943aa8
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 7 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ For the following registrys:
- [ ] [Ireland](http://www.ubss.org.uk/search_irishcaves.php)

# Download
~~To download the output from this repository go to the releases page and download the file for the region you are interested in.~~
To download the output from this repository go to the releases page and download the file for the region you are interested in.

## Accuracy
All efforts have been made to ensure accuracy of the output files but not every registry entry is acurate to begin with. Not every registry entry lists WGS84 or NGR which means it has to be derived from the other if availaible which can introduce an error. If you chose to use this data for navigation its not my fault if you end up wandering around on leck fell lost in the mist!
Expand All @@ -33,8 +33,8 @@ Docker is used manage the python enviroment.
1. First build the docker image `docker-compose build`
1. Then run the spiders with `docker-compose up`

To run scrapy commands run `docker-compose run scrapy scrapy <command>`.
To run scrapy commands run `docker-compose run scrapy scrapy <command>` such as docker-compose run scrapy scrapy crawl <registry>` to crawl a single registry.

To have access to the containers shell run `docker-compose run scrapy /bin/bash`

You may need to reset the owner and permissions of files created by docker/scrapy using `sudo chmod` & `sudo chown`
You may need to reset the owner and permissions of files created by docker/scrapy using `sudo chmod` & `sudo chown` on linux.
8 changes: 4 additions & 4 deletions app/start.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
import scrapy
from scrapy.crawler import CrawlerProcess
from ukcavegis.spiders import dcaregistry, mendipregistry, scotlandregistry, cnccregistry
from ukcavegis.spiders import dcaregistry, mendipregistry, scotlandregistry, cnccregistry, cccregistry, fodccagregistry, ducregistry
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.crawl(dcaregistry.DcaRegistry)
process.crawl(mendipregistry.MendipRegistry)
process.crawl(scotlandregistry.ScotlandRegistry)
process.crawl(cnccregistry.CnccRegistry)
process.crawl(cnccregistry.DucRegistry)
process.crawl(cnccregistry.FodccagRegistry)
process.crawl(cnccregistry.CccRegistry)
process.crawl(ducregistry.DucRegistry)
process.crawl(fodccagregistry.FodccagRegistry)
process.crawl(cccregistry.CccRegistry)
process.start()

0 comments on commit b943aa8

Please sign in to comment.