Distributed Key/Value count(s) workers, flushing entries to the database at least every 10 seconds or less.
The flow starts with a client action sending a POST request with a JSON encoded body.
The first layer to take action in the system is Caddy, a reverse proxy. It will distribute the requests from the clients between all available HTTP servers.
Rate limiting runs at this step, before passing the data downstream. It is implemented with a Caddy extension that returns a HTTP 429 if the amount of requests is larger than a threshold value.
HTTP server instances that will validate the JSON payload and, if valid, will push a message to a messaging queue implemented in top of Redis using a library called RMQ. It uses the publish-subscribe pattern.
If by this moment the client submitted a valid request, it will get a HTTP 200. Otherwise a HTTP 400 is returned.
A predefined set of workers are subscribed to this queue. These will pickup and distribute batches of up to N items from it.
After validating the message(s), it will send the new data to a channel. This is used to share/sync new messages to another goroutine running in the same worker which is in charge of maintaining a synchronized map with the amount of hits for each key.
In the aforementioned goroutine, a ticker that runs every nine seconds will send the counts held in the worker's memory, to the database using a single query that includes all keys and their values.
Nine seconds is a value chosen after one of the requirements of the problem that states data available in the database should be no older than 10 seconds.
Metrics are pulled from Caddy every 15 seconds by Prometheus.
{ 'key': "uuid" 'value': int64 }
keymust be a validUUIDstring.valuemust be equal or higher than number 1.- No extra fields are present in the request Payload.
make test runs Go tests
make build builds all the images
make db-clear clears the table increments
make all Build the images, create the table in Postgres and runs the app.
make dev hot reload dev environment
make bench runs a simple benchmark using hey
- /Increment endpoint
- JSON payload Validation
- Requests incremented by given key
- The persisted state must be, at most, ten seconds out of date.
- Rate limit
- Benchmarking
- Proxy Metrics + Prometheus + Grafana
- CI: CircleCI
- Auth: Suggested implementation
- Graceful shutdown
- Retry mechanism when connecting to DB, publish messages, DLQs, transport layer,etc
TestHitsServer_CreateIncrement: Simple test issuing a valid request expecting a valid response.
TestHitsServer_Concurrent: Integration test. Concurrent stress test issuing multiple concurrent queries using different keys and counting the available keys who made to the database.
Test_incrementRequestValid: Input and expected output tests
Test_fillMask: Useful UUID generators and their testing.
Exercise was fun to work on and allowed me to refresh on some concepts. One of the issues I found was the fact that introducing multiple consumers pulling from the same worker would create potential deadlocks in Postgres.
The solution applied was to use a single connection per host using a single query which compacts all available keys( after every tick ) into a single query containing all of them using UNNEST.
A potential improvement would be limiting the ammount of entries a single INSERT may contain which is currently unlimited.
Before going to production with the current solution I would invest some extra time making sure there are no single point of failures. For example, setting Redis to HA maybe by using a managed service like Elasticache.
As a final note, I lost many precious minutes with this issue I end up submitted a PR to
- https://medium.com/avitotech/how-to-work-with-postgres-in-go-bad2dabd13e4
- https://www.reddit.com/r/golang/comments/h7ontk/how_to_use_connection_pooling_with_pgxgolang/
- https://rafaelcn.github.io/2020/03/07/an-advice-about-postgresql-drivers-and-go.html
- https://medium.com/@amoghagarwal/insert-optimisations-in-golang-26884b183b35> (https://blogtitle.github.io/go-advanced-concurrency-patterns-part-2-timers/)
- https://medium.com/@jeremieshaker/golang-ticker-best-practices-using-tickers-in-a-multi-threaded-program-without-losing-your-mind-dfc307c6de62
- https://devandchill.com/posts/2020/05/go-lib/pq-or-pgx-which-performs-better/
- https://www.postgresonline.com/journal/archives/347-LATERAL-WITH-ORDINALITY---numbering-sets.html
- https://stackoverflow.com/questions/41717935/preserve-the-order-of-items-in-array-when-doing-join-in-postgres
- https://stackoverflow.com/questions/8760419/postgresql-unnest-with-element-number