This is an Idempotent Scala service that persists user-events from kinesis stream to Aurora RDS.
The main purpose of the project is to demonstrate
- http4s
- kamon
- AWS kinesis & KCL
- scalaz streams & Task implementation for AWS Kinesis stream
- scala & docker
- doobie
- circe
- statsd
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
What things you need to install the software and how to install them
- you have
- [docker-id]
- [AWS account]
-
set up development environment
- scala 2.11+
- sbt
- docker
- docker-compose
- [JDK 8+]
- direnv
- *aws-cli*
- *emacs ensime*
-
set up environmnet variables
export DOCKER_USER='docker_userid'
export DOCKER_PASS='docker-password'
export AWS_REGION='us-east-1'
export AWS_ACCESS_KEY_ID=XXXXXX
export SECRET_ACCESS_KEY=XXXXXX
export JDBC_USER=xxxxx
export JDBC_PASSWORD=xxxxx
export JDBC_DB=mytestdb
export JDBC_URL='mysql://$JDBC_USER:$JDBC_PASSWORD@0.0.0.0:3306/$JDBC_DB'
git clone git@github.com:kayvank/kinesis-to-aurora.git
cd kinesis-to-aurora
sbt clean compile ## to build prject
The following will
- test
- dockerize
- publish docker images
- create executable scripts
sbt test
sbt clean compile universal:packageBin
sbt clean compile docker:stage docker:publishLocal
docker images | grep 'kinesis-to-aurora' ## verify your docker image was published locally
sbt clean compile docker:stage docker:publish
Instruction for running the project locally. We are going to use mysql docker image to simulate AWS Aurora RDS
-
Assumptions
- you have created a AWS kinesis stream
-
Setup database
cd ./scripts/sql
docker-compose -f ./docker-compose-mysql.yml up -d
mysql -h 0.0.0.0 -u $JDBC_USER -p $JDBC_DB < ./like_events.sql
- Setup statsd & graphite
docker run -d\
--name graphite\
--restart=always\
-p 80:80\
-p 2003-2004:2003-2004\
-p 2023-2024:2023-2024\
-p 8125:8125/udp\
-p 8126:8126\
hopsoft/graphite-statsd
Put the various docker images into a docker-compose
sbt clean run
- local deployment
sbt docker clean compile docker:stage docker:publishLocal
docker run \
-p9000:9000 \
-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
-e KINESIS_STREAM_NAME=${KINESIS_STREAM_NAME} \
-e JDBC_URL=${JDBC_URL} \
-e JDBC_PASSWORD=${JDBC_PASSWORD} \
-e JDBC_DIRVER=${JDBC_DIRVER} \
q2io/kinesis-to-aurora
{
"entity_type": "USER",
"entity_id": "USUV71400762",
"user_id": "26985937",
"action": "UNLIKE"
}
mysql> describe like_events;
+-------------|--------------|------|-----|-------------------|-------+
| Field | Type | Null | Key | Default | Extra |
+-------------|--------------|------|-----|-------------------|-------+
| id | varchar(40) | YES | MUL | NULL | |
| user_id | varchar(40) | YES | MUL | NULL | |
| entity_id | varchar(140) | YES | MUL | NULL | |
| entity_type | varchar(20) | YES | MUL | NULL | |
| ts | timestamp | NO | MUL | CURRENT_TIMESTAMP | |
| created_at | timestamp | NO | | CURRENT_TIMESTAMP | |
+-------------|--------------|------|-----|-------------------|-------+
- Kayvan Kazeminejad - Intitial work