This is a Golang (go) program to read a specific JSON Payload message from a Message Queue System and Store into Database using concurrency
This is a close image of how it works:
- The number of queue consumers could be different from the numbers of storage workers, see config-sample.yaml
- The process (job) of consuming one message from the queue and store into the database is synchronous because every message needs to be acknowledged (confirmed as stored).
- Golang
pprof
enabled via--profile
command line when starting the service - Prometheus metrics for consumers, storage workers, go statistics, and database
- Grafana dashboard for Prometheus.io metrics
- Dockerfile multi-stage build
- Makefile to facilitate the project builds
- docker-compose file and configuration to testing all elements
- docker images at docker-hub and Github Packages
- CI/CD Github Action pipeline workflow
There are many ways to do it, but always is necessary PostgreSQL
and RabbitMQ
dependencies, the easy way to see how it works is using containers.
The program and all dependencies and visibility systems at once
docker-compose up --build
docker-compose down -v
After docker-compose start all the services, you have the following links ready to be curious, I prepared a simple grafana dashboard
to show you part of the prometheus metrics
implemented
- mq-to-db-01 home page
- mq-to-db-02 home page
- Prometheus Dashboard
- Grafana Dashboard
- RabbitMQ Dashboard
# terminal 1, for mq-to-db-01 inside the docker-compose-file
go tool pprof http://127.0.0.1:8080/debug/pprof/goroutine
# terminal 2, for mq-to-db-02 inside the docker-compose-file
go tool pprof http://127.0.0.1:8081/debug/pprof/goroutine
# once you are into tool pprof, execute the command web
(pprof) web
docker-compose logs mq-to-db-01
docker-compose logs mq-to-db-02
First install dependencies
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
NOTES:
- RabbitMQ web console: http://localhost:15672
- Username: guest
- Password: guest
docker run --rm --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
# logs
docker logs postgresql -f
# remember to stop and remove (--rm in docker run do it for you)
docker stop postgresql
git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help
# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yaml
NOTE: the parameter -race
is to check race conditions because we are using Go Concurrency
compiling for your ARCH and OS
# make executable first
make
# check the available options
./build/mq-to-db --help
# execute
./build/mq-to-db --configFile config-sample.yaml
Cross-compiling
make build-dist
# cross-compiling files
ls -l ./dist/
NOTES related to make
-
- This create a cross-compiling binaries and also Docker Image (linux 64bits)
-
- Check the Makefile to see
The make available targets options
- Check the Makefile to see
-
- Remember to start dependencies first
Here I use latest tag
, but you can see all releases here
# pull the image first
docker pull christiangda/mq-to-db:latest
# see available option
docker run --rm --name mq-to-db christiangda/mq-to-db:latest - --help
# run with a config file mapped and with profile option
docker run --rm -v <path to config file>:/etc/mq-to-db/config.yaml --name mq-to-db christiangda/mq-to-db:latest - --profile
NOTES:
- Remember to start dependencies first
The application expose different endpoints via http server
- http://localhost:8080/
- http://localhost:8080/metrics
- http://localhost:8080/health
- http://localhost:8080/debug/pprof
Internals
Externals
This module is released under the GNU General Public License Version 3: