This project serves as a POC for using Kafka Streams and Faust to aggregate API request metrics segmented by high-cardinality client attributes (e.g. user ID).
- Python 3.6+
- Zookeeper instance and Kafka server running on
localhost:9092
- Install Faust:
pip install faust - Start the Faust worker:
faust -A windowed_requests worker -l info - In a new terminal window, produce a message to the topic:
faust -A windowed_requests send windowed_requests '{"user_id": "7", "path": "/api/v1/data"}' - Fetch the current usage metrics for a given user:
curl localhost:6066/metrics/users/7
Prometheus is a poor fit for storing high cardinality metrics as discussed in their own documentation and this more in-depth article.