Snowplow Realtime Docker

Composition

All snowplow images are taken from their repository: snowplow-docker

• Scala Stream Collector
• Stream Enrich
• Elasticsearch Loader

Other components:

• Elasticsearch - for storage
• Kibana - for visualization
• nsq - for realtime messaging
• elasticsearch-head - for interacting with elasticsearch
• iglu repository - for hosting your snowplow schemas

Usage

• If necessary, put your iglu schemas into the 'iglu' folder.
• Run docker-compose up -d to pull and start up all containers.
• Set up a tracker to point to DOCKER_HOST:8080

Exposed ports:

• Scala stream collector: 8080

• NSQ admin: 4171

• Kibana: 5601

• Elasticsearch head: 9100

• Elasticsearch: 9200

• Iglu repository: 80

Example use case

This is useful as a Snowplow Mini alternative, demonstrating the whole Snowplow realtime pipeline stack and allowing instant event visualization in Kibana. Example use case would be debugging Snowplow Javascript script enrichment without pushing your script to production or even waiting for the batch pipeline to run to test the results of this enrichment.

1. Add your JSON enrichment config file to the config/enrichments folder.
2. Add your custom self-describing JSON for the derived contexts into the iglu/schemas folder.
3. Restart stream-enrich container either via CLI or Portainer UI.
4. Hit the testing page, that has your tracker set up to stream-collector.
5. View the enrichment results directly in Kibana.

Debugging

• Loglevels for snowplow compontents can be easily changed via environment variables passed on runtime in docker-compose.yml config.
• Printing values to stdout in custom Javascript enrichment can be viewed in docker logs: docker logs -f stream-enrich.
• Bad events will land in elasticsearch with bad type.