Offline Collector is a scraper used to collect Offline Raleigh events. It uses a proxy-rotator to gather these events to keep from having a single IP Address tracked/blocked.
This setup requires go version 1.11+
- Clone and enter the repository
- run
go get -u
- run
go build
- run
OFFLINE_SERVER=<Address> OFFLINE_PORT=<PORT> ./offline-collector
. TheOFFLINE_SERVER
andOFFLINE_PORT
should point to the running instance of Offline Server. Excluding these variables will default tohttp://localhost:3000
.
Options:
--start
: Used to indicate the start date to collect from (formatYYYY-MM-DD
). Defaults to today--end
: Used to indicate the end date to collect to (formatYYYY-MM-DD
). Defaults to value ofstart
.
To run with docker, you first have to create the docker image by executing:
run `docker build -t offline-collector`
Once the image is created, you can create/run the container by executing:
docker run \
--env OFFLINE_SERVER=<URL> OFFLINE_PORT=<PORT> \
-d \
--name offline-collector \
offline-collector
Leave of -d
if you want to run the container in the same shell (keep it from detaching).
A docker-compose.yaml
file is also included which will spin up a network, start the server and then kick of the collector.
To start:
ELASTIC_SEARCH_URL=<ES_URL> docker-compose up
To shutdown:
docker-compose down