micro-starter-kit

Microservices starter kit for Golang, aims to be developer friendly.

Overview

What you get

• Monorepo - Sharing Code Between Microservices
• gRPC microservices with REST Gateway
• Input Validation with protoc-gen-validate (PGV)
• Config - Pluggable Sources, Mergeable Config, Safe Recovery
• Customizable Logging
• CRUD Example using GORM
• GORM code gen via protoc-gen-gorm
• Dependency injection Container
• One Step build/publish/deploy with ko
• BuildInfo with govvv
• Linting with GolangCI linters aggregator
• CICD Pipelines with GitHub Actions
• Kubernetes Matrix Deployment with Kustomize
• Observability
• Service Mesh with Istio
• GraphQL Gateway with gqlgen, rejoiner

Getting Started

Prerequisite

run following go get ... commands outside this project root and $GOPATH
if you get error, try setting export GO111MODULE=on befor running go get ...

# fetch micro into $GOPATH
go get -u github.com/micro/micro
go get -u github.com/micro/go-micro
# go lang  build/publish/deploy tool
go get -u github.com/google/ko/cmd/ko
go get -u sigs.k8s.io/kustomize
# go better build tool
go get -u github.com/ahmetb/govvv
# for static check/linter
go get -u github.com/golangci/golangci-lint/cmd/golangci-lint
# for mac, use brew to install protobuf
brew install protobuf
# GUI Client for GRPC Services
brew cask install bloomrpc

# fetch protoc plugins into $GOPATH
go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
go get -u github.com/micro/protoc-gen-micro
# go get -u github.com/envoyproxy/protoc-gen-validate
# go get -u github.com/infobloxopen/protoc-gen-gorm

Installing PGV can currently only be done from source:

go get -d github.com/envoyproxy/protoc-gen-validate
cd ~/go/src/github.com/envoyproxy/protoc-gen-validate
make build

Installing protoc-gen-gorm can currently only be done from source:

go get -d github.com/infobloxopen/protoc-gen-gorm
cd ~/go/src/github.com/infobloxopen/protoc-gen-gorm
make install

Initial Setup

(optional) setup your workspace from scratch

go mod init github.com/xmlking/micro-starter-kit
mkdir srv api fnc

# scaffold modules
micro new --fqdn="account-srv" --type="srv" --gopath=false \
--alias="account" --plugin=registry=kubernetes srv/account

micro new --fqdn="emailer-srv" --type="srv" --gopath=false \
--alias="emailer"  --plugin=registry=kubernetes:broker=nats srv/emailer

micro new --fqdn="greeter-srv" --type="srv" --gopath=false \
--alias="greeter"  --plugin=registry=kubernetes srv/greeter

micro new --fqdn="account-api" --type="api" --gopath=false \
--alias="account" --plugin=registry=kubernetes api/account

Build

make proto
# silence
make -s proto

make build

Run

Database

By default this project use embedded sqlite3 database. if you want to use postgreSQL,

• start postgres via docker-compose command provided below
• uncommend postgres import statement and comment sqlite in plugin.go
• start micro server with --configFile=config.dev.postgres.yaml flag
  i.e., go run srv/account/main.go srv/account/plugin.go --configFile=config.dev.postgres.yaml

# to start postgres in foreground
docker-compose up postgres
# to stop postgres
docker-compose down
# if needed, remove `postgres_data` volume to recreate database next time, when you start.
docker system prune --volumes

Services

Node: --server_address=localhost:5501x --broker_address=localhost:5502x required only when you are behind VPN a.k.a Work From Home

# dev mode
make run-account
# or
make run-account ARGS="--server_address=localhost:55012 --broker_address=localhost:55022"
# or
go run srv/account/main.go srv/account/plugin.go \
--configDir deploy/bases/account-srv/config \
--server_address=localhost:55011 --broker_address=localhost:55021

make run-emailer
# or
make run-emailer ARGS="--server_address=localhost:55011 --broker_address=localhost:55021"

# integration tests for config module via CMD
make run TARGET=demo TYPE=cmd
go run cmd/demo/main.go --help
go run cmd/demo/main.go --database_host=1.1.1.1 --database_port=7777

export APP_ENV=production
go run cmd/demo/main.go

Services with gRPC transport

go-micro by default use transport=http and protocol=mucp. If you want to switch to gRPC, do this

# run account micro with gRPC transport, NOTE: we also have to add --server_name as it will remove server_name
go run srv/account/main.go srv/account/plugin.go --client=grpc --server=grpc --server_name=account-srv
go run srv/emailer/main.go srv/emailer/plugin.go --client=grpc --server=grpc --server_name=emailer-srv
# we also has to use grpc for gateway and `micro call` cli
go run cmd/micro/main.go cmd/micro/plugin.go --client=grpc --server=grpc api
# go run cmd/micro/main.go cmd/micro/plugin.go --client=grpc --server=grpc api --enable_rpc=true
go run cmd/micro/main.go cmd/micro/plugin.go  --api_handler=rpc  api  --enable_rpc=true

micro --client=grpc call account-srv UserService.List '{ "limit": 10, "page": 1}'

Test

# Run only Unit tests:
make test-emailer
go test -v -short
go test -v -short ./srv/emailer/service
# Run only Integration Tests: Useful for smoke testing canaries in production.
make inte-emailer
go test -v -run Integration ./srv/emailer/service

UAT Test

using micro CLI

micro list services
micro get service account-srv
micro get service emailer-srv

# how to start proxy
micro proxy --protocol=grpc

Test gRPC Directly

remember to use micro --client=grpc when microservices and gateway are using grpc transport

# micro --client=grpc call account-srv UserService.Create \
# '{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call  account-srv UserService.Create \
'{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call account-srv UserService.Create \
'{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call account-srv UserService.List '{}'
micro call account-srv UserService.List '{ "limit": 10, "page": 1}'
micro call account-srv UserService.Get '{"id": "UserIdFromList"}'
micro call account-srv UserService.Exist '{"username": "sumo", "email": "sumo@demo.com"}'
micro call account-srv UserService.Update \
'{"id": "UserIdFromGet", "firstName": "sumoto222","email": "sumo222@demo.com"}'
micro call account-srv UserService.Delete '{ "id": "UserIdFromGet" }'

Test via Micro Web UI

# Start Web UI for testing
micro web

open http://localhost:8082

create new user from Micro Web UI and see if an email is send

{
  "username": "sumo",
  "firstName": "sumo",
  "lastName": "demo",
  "email": "sumo@demo.com"
}

Test via Micro API Gateway

Start API Gateway

Start API Gateway and run REST Client tests

# start API Gateway to test via REST-Client
micro api --enable_rpc=true

GitOps

Deploy

Use ko. If you are new to ko check out the ko-demo

Set a registry and make sure you can push to it:

export PROJECT_ID=ngx-starter-kit
export KO_DOCKER_REPO=gcr.io/${PROJECT_ID}

to publish locally set: export KO_DOCKER_REPO=ko.local

Then apply like this:

ko apply -f deploy/

To deploy in a different namespace:

ko -n nondefault apply -f deploy/

Release

This will publish all of the binary components as container images and create a release.yaml

# publish to  docker repo ar KO_DOCKER_REPO
ko resolve -P -f deploy/ > release.yaml
# publish to local docker repo
ko resolve -P -L -f deploy/ > release.yaml

run local image

docker run -it \
-e MICRO_SERVER_ADDRESS=0.0.0.0:8080 \
-e MICRO_BROKER_ADDRESS=0.0.0.0:10001 \
-e MICRO_REGISTRY=mdns \
-e CONFIG_DIR=/var/run/ko \
-e CONFIG_FILE=config.yaml \
-p 8080:8080 -p 10001:10001 ko.local/github.com/xmlking/micro-starter-kit/srv/account

Docker

Docker Build

# build
TYPE=srv
TARGET=account
VERSION=0.0.5-SNAPSHOT
# DOCKER_REGISTRY=gcr.io
DOCKER_CONTEXT_PATH=xmlking
# docker build --force-rm=true --rm=true --no-cache \
docker build --rm \
--build-arg VERSION=$VERSION \
--build-arg TYPE=${TYPE} \
--build-arg TARGET=${TARGET} \
--build-arg DOCKER_REGISTRY=${DOCKER_REGISTRY} \
--build-arg DOCKER_CONTEXT_PATH=${DOCKER_CONTEXT_PATH} \
--build-arg VCS_REF=$(shell git rev-parse --short HEAD) \
--build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
-t ${DOCKER_REGISTRY:+${DOCKER_REGISTRY}/}${DOCKER_CONTEXT_PATH}/${TARGET}-${TYPE}:${VERSION} .

IMANGE_NAME=${DOCKER_REGISTRY:+${DOCKER_REGISTRY}/}${DOCKER_CONTEXT_PATH}/${TARGET}-${TYPE}:${VERSION}

# push
docker push $IMANGE_NAME

# check
docker inspect  $IMANGE_NAME
# remove temp images after build
docker image prune -f
# Remove dangling images
docker rmi $(docker images -f "dangling=true" -q)
# Remove images tagged with vendor=sumo
docker rmi $(docker images -f "label=org.label-schema.vendor=sumo"  -q)
# Remove all <none> layers
docker rmi $(docker images -a|grep "<none>"|awk '$1=="<none>" {print $3}')

Docker Run

run just for testing image...

docker run -it \
-e MICRO_SERVER_ADDRESS=0.0.0.0:8080 \
-e MICRO_BROKER_ADDRESS=0.0.0.0:10001 \
-e MICRO_REGISTRY=mdns \
-p 8080:8080 -p 10001:10001 $IMANGE_NAME

Docker Compose Run

Run complete app suite with docker-compose

docker-compose up consul
docker-compose up account-srv
docker-compose up emailer-srv
docker-compose up gateway
docker-compose up account-api
docker-compose up gateway-api
curl "http://localhost:8081/account/AccountService/list?limit=10"

Kubernetes Run

run just for testing image in k8s...

# account-srv
kubectl run --rm mytest --image=xmlking/account-srv:latest \
--env="MICRO_REGISTRY=kubernetes" \
--env="MICRO_SELECTOR=static" \
--env="MICRO_SERVER_ADDRESS=0.0.0.0:8080" \
--env="MICRO_BROKER_ADDRESS=0.0.0.0:10001" \
--restart=Never -it

# gateway
kubectl run --rm mygateway --image=microhq/micro:kubernetes \
--env="MICRO_REGISTRY=kubernetes" \
--env="MICRO_SELECTOR=static" \
--restart=Never -it \
--command ./micro api

Make

using Makefile

use -n flag for dry-run, -s or '--silent' flag to suppress echoing

# codegen from proto
make proto
make proto TARGET=account
make proto TARGET=account TYPE=api
make proto-account
make proto-account-api
## generate for protos in shared package
make proto TARGET=shared TYPE=.

# unit tests
make test-account
make test-emailer
make test-account-api
make test-config-shared
make test-demo-cmd

# integration tests
make inte-account
make inte-emailer

# run
make run-account
make run-emailer
make run-account-api
make run-micro-cmd ARGS="--api_address=0.0.0.0:8088 api"
make run-demo-cmd

# build
make build VERSION=v0.1.1
make build TARGET=account VERSION=v0.1.1
make build TARGET=account TYPE=srv VERSION=v0.1.1
make build TARGET=emailer TYPE=srv VERSION=v0.1.1
make build TARGET=account TYPE=api VERSION=v0.1.1
make build-account VERSION=v0.1.1
make build-account-api VERSION=v0.1.1

# push tag to git
make release VERSION=v0.1.1

# build docker image
make docker-account VERSION=v0.1.1
make docker-account-srv VERSION=v0.1.1
make docker TARGET=account VERSION=v0.1.1
make docker TARGET=account TYPE=srv VERSION=v0.1.1
make docker TARGET=account DOCKER_REGISTRY=gcr.io DOCKER_CONTEXT_PATH=micro-starter-kit

make docker-emailer-srv
make docker-account-api

# publish all microservices images
make docker_push
# remove all previous microservices images and any dangling images
make docker_clean

Reference

1. examples - example usage code for micro
2. microhq - a place for prebuilt microservices
3. explorer - which aggregates micro based open source projects
4. micro-plugins extensible micro plugins