Skip to content

mneedham/real-time-analytics-book

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

81 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Real-Time Analytics Book Code Repository

The code for Dunith Dhanushka and Mark Needham's upcoming book on Real-Time Analytics.

Put the architecture diagram here

Install components

docker-compose \
  -f docker-compose-base.yml \
  -f docker-compose-pinot.yml \
  -f docker-compose-dashboard-enriched.yml \
  -f docker-compose-dashboard.yml up
docker-compose \
  -f docker-compose-base.yml \
  -f docker-compose-pinot-m1.yml \
  -f docker-compose-dashboard.yml \
  -f docker-compose-enrich-streams.yml \
  -f docker-compose-dashboard-enriched.yml \
  up

Once that's run, you can navigate to the following:

Add enriched table

docker run -v $PWD/pinot/config:/config \
  --network rta \
  apachepinot/pinot:0.11.0 \
  AddTable -schemaFile /config/orders_enriched/schema.json \
           -tableConfigFile /config/orders_enriched/table.json \
           -controllerHost pinot-controller -exec

(Optional) Downloading products

Setup Python environment:

python -m venv venv
source venv/bin/activate

Install dependencies:

pip install -r scripts/requirements.txt

Downloading product pages:

python scripts/download_products.py

Scrape product data:

python scripts/scrape_products.py

The products will be written to simulator/data/products.json