GoCrawler

This service is a crawler for social platform data.

---

Run service locally

Deploy infra services

cd deploy && docker-compose -f infra.yml

Initialize postgresql databases

create admin databases

docker exec -it postgres bash
psql -U postgres
CREATE DATABASE backend_admin;

---

Build images

Build go-crawler-service images

GOCRAWLER_IMAGE_TAGS=latest make build-gocrawler-service-image

Build ptt-crawler-consumer images

CONSUMER_IMAGE_TAGS=latest make build-consumer-image  

Deploy service with docker-compose

Add domain name to /etc/hosts

echo "127.0.0.1 gocrawler2.microservice.com\n127.0.0.1 gocrawler1.microservice.com" >> /etc/hosts

---

Deploy traefik as reverse proxy

cd deploy
docker create network traefik_net
docker-compose -f traefik-docker-compose.yml up -d

Deploy app microservices with docker-compose

docker-compose up -d

Test

http://gocrawler1.microservice.com/ping
http://gocrawler2.microservice.com/ping
http://gocrawler1.microservice.com/swagger/index.html
http://gocrawler2.microservice.com/swagger/index.html

Reference

• traefik docs
• traefik github
• traefik 2.0 Example
• cassandra docker
• cassandra cqlsh tutorial
• rabbitmq docker
• rabbitmq getstarted
• mongodb docker
• mongodb tutorial
• fluent example
• fluent-logger-golang github
• jaeger & gin example
• opentracing-jaeger
• gin+gorm+router 快速搭建 crud restful API 接口
• go-gin-example