This is an app for connecting to kafka and sending webhooks.
- Language should be mature and have a mature kafka client
Kafka Benchmarking:
Client Type | Throughput |
---|---|
Java | 40,000 - 50,0000 |
Go | 28,000 - 30,0000 |
Node | 6,000 - 8,000 |
Kafka-pixy | 700 - 800 |
Logstash | 250 |
- The Language should be performant
- Language should support concurrency
Language Go or Node are preferable.
Document Architecture decisions using ADR TOOLS If you cannot install the tools then use the provided template
- single receiving topic enter the persist-and-validate consumer:
- This consumer persists all webhooks and validates them afterward.
- Then if it is invalid it will not send it on to the next consumer.
- Valid uuid's are passed onto the next topic
- Unique URL's should be persisted and monitored
- When a message arrived with a new URL this URL is added to the ENDPOINTS table
- This URL is monitored periodically for availability
- single delivering topic from the webhooks received consumer will be dispatched here:
- This consumer will poll the URL provided for the webhooks and send the webhooks on to the next stage
git clone git@github.com:simonireilly/kafka-webhook-dispatcher.git
# Build app, compose up and tail logs
make build up tail
# Stop the app
make stop
- Create exported functions to be passed to a consumer and create unit tests for these functions.
- Spin up the app with a consumer and producer and connect to a kafka cluster
- Receive messages and process them into the database
- Retrieve successfully stored messages and validate the contents
- Is valid url for urls in collection
- Is a valid json body (can be json parsed - all we can do really)
- Post successfully validated webhooks concurrently for all urls provided
- Create a container to receive webhooks and test response
- Status - 200
- Status - 404
- Status - 401
- Status - 301
- Status - 500