Skip to content
/ cete Public

Cete is a distributed key value store server written in Go built on top of BadgerDB.

License

Notifications You must be signed in to change notification settings

mosuka/cete

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

75 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Cete

Cete is a distributed key value store server written in Go built on top of BadgerDB.
It provides functions through gRPC (HTTP/2 + Protocol Buffers) or traditional RESTful API (HTTP/1.1 + JSON).
Cete implements Raft consensus algorithm by hashicorp/raft. It achieve consensus across all the instances of the nodes, ensuring that every change made to the system is made to a quorum of nodes, or none at all.
Cete makes it easy bringing up a cluster of BadgerDB (a cete of badgers) .

Features

  • Easy deployment
  • Bringing up cluster
  • Database replication
  • An easy-to-use HTTP API
  • CLI is also available
  • Docker container image is available

Building Cete

When you satisfied dependencies, let's build Cete for Linux as following:

$ mkdir -p ${GOPATH}/src/github.com/mosuka
$ cd ${GOPATH}/src/github.com/mosuka
$ git clone https://github.com/mosuka/cete.git
$ cd cete
$ make build

If you want to build for other platform, set GOOS, GOARCH environment variables. For example, build for macOS like following:

$ make GOOS=darwin build

Binaries

You can see the binary file when build successful like so:

$ ls ./bin
cete

Testing Cete

If you want to test your changes, run command like following:

$ make test

Packaging Cete

Linux

$ make GOOS=linux dist

macOS

$ make GOOS=darwin dist

Configure Cete

CLI Flag Environment variable Configuration File Description
--config-file - - config file. if omitted, cete.yaml in /etc and home directory will be searched
--id CETE_ID id node ID
--raft-address CETE_RAFT_ADDRESS raft_address Raft server listen address
--grpc-address CETE_GRPC_ADDRESS grpc_address gRPC server listen address
--http-address CETE_HTTP_ADDRESS http_address HTTP server listen address
--data-directory CETE_DATA_DIRECTORY data_directory data directory which store the key-value store data and Raft logs
--peer-grpc-address CETE_PEER_GRPC_ADDRESS peer_grpc_address listen address of the existing gRPC server in the joining cluster
--certificate-file CETE_CERTIFICATE_FILE certificate_file path to the client server TLS certificate file
--key-file CETE_KEY_FILE key_file path to the client server TLS key file
--common-name CETE_COMMON_NAME common_name certificate common name
--log-level CETE_LOG_LEVEL log_level log level
--log-file CETE_LOG_FILE log_file log file
--log-max-size CETE_LOG_MAX_SIZE log_max_size max size of a log file in megabytes
--log-max-backups CETE_LOG_MAX_BACKUPS log_max_backups max backup count of log files
--log-max-age CETE_LOG_MAX_AGE log_max_age max age of a log file in days
--log-compress CETE_LOG_COMPRESS log_compress compress a log file

Starting Cete node

Starting cete is easy as follows:

$ ./bin/cete start --id=node1 --raft-address=:7000 --grpc-address=:9000 --http-address=:8000 --data-directory=/tmp/cete/node1

You can get the node information with the following command:

$ ./bin/cete node | jq .

or the following URL:

$ curl -X GET http://localhost:8000/v1/node | jq .

The result of the above command is:

{
  "node": {
    "raft_address": ":7000",
    "metadata": {
      "grpc_address": ":9000",
      "http_address": ":8000"
    },
    "state": "Leader"
  }
}

Health check

You can check the health status of the node.

$ ./bin/cete healthcheck | jq .

Also provides the following REST APIs

Liveness prove

This endpoint always returns 200 and should be used to check Cete health.

$ curl -X GET http://localhost:8000/v1/liveness_check | jq .

Readiness probe

This endpoint returns 200 when Cete is ready to serve traffic (i.e. respond to queries).

$ curl -X GET http://localhost:8000/v1/readiness_check | jq .

Putting a key-value

To put a key-value, execute the following command:

$ ./bin/cete set 1 value1

or, you can use the RESTful API as follows:

$ curl -X PUT 'http://127.0.0.1:8000/v1/data/1' --data-binary value1
$ curl -X PUT 'http://127.0.0.1:8000/v1/data/2' -H "Content-Type: image/jpeg" --data-binary @/path/to/photo.jpg

Getting a key-value

To get a key-value, execute the following command:

$ ./bin/cete get 1

or, you can use the RESTful API as follows:

$ curl -X GET 'http://127.0.0.1:8000/v1/data/1'

You can see the result. The result of the above command is:

value1

Deleting a key-value

Deleting a value by key, execute the following command:

$ ./bin/cete delete 1

or, you can use the RESTful API as follows:

$ curl -X DELETE 'http://127.0.0.1:8000/v1/data/1'

Bringing up a cluster

Cete is easy to bring up the cluster. Cete node is already running, but that is not fault tolerant. If you need to increase the fault tolerance, bring up 2 more data nodes like so:

$ ./bin/cete start --id=node2 --raft-address=:7001 --grpc-address=:9001 --http-address=:8001 --data-directory=/tmp/cete/node2 --peer-grpc-address=:9000
$ ./bin/cete start --id=node3 --raft-address=:7002 --grpc-address=:9002 --http-address=:8002 --data-directory=/tmp/cete/node3 --peer-grpc-address=:9000

Above example shows each Cete node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host.

This instructs each new node to join an existing node, each node recognizes the joining clusters when started. So you have a 3-node cluster. That way you can tolerate the failure of 1 node. You can check the cluster with the following command:

$ ./bin/cete cluster | jq .

or, you can use the RESTful API as follows:

$ curl -X GET 'http://127.0.0.1:8000/v1/cluster' | jq .

You can see the result in JSON format. The result of the above command is:

{
  "cluster": {
    "nodes": {
      "node1": {
        "raft_address": ":7000",
        "metadata": {
          "grpc_address": ":9000",
          "http_address": ":8000"
        },
        "state": "Leader"
      },
      "node2": {
        "raft_address": ":7001",
        "metadata": {
          "grpc_address": ":9001",
          "http_address": ":8001"
        },
        "state": "Follower"
      },
      "node3": {
        "raft_address": ":7002",
        "metadata": {
          "grpc_address": ":9002",
          "http_address": ":8002"
        },
        "state": "Follower"
      }
    },
    "leader": "node1"
  }
}

Recommend 3 or more odd number of nodes in the cluster. In failure scenarios, data loss is inevitable, so avoid deploying single nodes.

The above example, the node joins to the cluster at startup, but you can also join the node that already started on standalone mode to the cluster later, as follows:

$ ./bin/cete join --grpc-addr=:9000 node2 127.0.0.1:9001

or, you can use the RESTful API as follows:

$ curl -X PUT 'http://127.0.0.1:8000/v1/cluster/node2' --data-binary '
{
  "raft_address": ":7001",
  "metadata": {
    "grpc_address": ":9001",
    "http_address": ":8001"
  }
}
'

To remove a node from the cluster, execute the following command:

$ ./bin/cete leave --grpc-addr=:9000 node2

or, you can use the RESTful API as follows:

$ curl -X DELETE 'http://127.0.0.1:8000/v1/cluster/node2'

The following command indexes documents to any node in the cluster:

$ ./bin/cete set 1 value1 --grpc-address=:9000 

So, you can get the document from the node specified by the above command as follows:

$ ./bin/cete get 1 --grpc-address=:9000

You can see the result. The result of the above command is:

value1

You can also get the same document from other nodes in the cluster as follows:

$ ./bin/cete get 1 --grpc-address=:9001
$ ./bin/cete get 1 --grpc-address=:9002

You can see the result. The result of the above command is:

value1

Cete on Docker

Building Cete Docker container image on localhost

You can build the Docker container image like so:

$ make docker-build

Pulling Cete Docker container image from docker.io

You can also use the Docker container image already registered in docker.io like so:

$ docker pull mosuka/cete:latest

See https://hub.docker.com/r/mosuka/cete/tags/

Pulling Cete Docker container image from docker.io

You can also use the Docker container image already registered in docker.io like so:

$ docker pull mosuka/cete:latest

Running Cete node on Docker

Running a Cete data node on Docker. Start Cete node like so:

$ docker run --rm --name cete-node1 \
    -p 7000:7000 \
    -p 8000:8000 \
    -p 9000:9000 \
    mosuka/cete:latest cete start \
      --id=node1 \
      --raft-address=:7000 \
      --grpc-address=:9000 \
      --http-address=:8000 \
      --data-directory=/tmp/cete/node1

You can execute the command in docker container as follows:

$ docker exec -it cete-node1 cete node --grpc-address=:9000

Securing Cete

Cete supports HTTPS access, ensuring that all communication between clients and a cluster is encrypted.

Generating a certificate and private key

One way to generate the necessary resources is via openssl. For example:

$ openssl req -x509 -nodes -newkey rsa:4096 -keyout ./etc/cete-key.pem -out ./etc/cete-cert.pem -days 365 -subj '/CN=localhost'
Generating a 4096 bit RSA private key
............................++
........++
writing new private key to 'key.pem'

Secure cluster example

Starting a node with HTTPS enabled, node-to-node encryption, and with the above configuration file. It is assumed the HTTPS X.509 certificate and key are at the paths server.crt and key.pem respectively.

$ ./bin/cete start --id=node1 --raft-address=:7000 --grpc-address=:9000 --http-address=:8000 --data-directory=/tmp/cete/node1 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost
$ ./bin/cete start --id=node2 --raft-address=:7001 --grpc-address=:9001 --http-address=:8001 --data-directory=/tmp/cete/node2 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost
$ ./bin/cete start --id=node3 --raft-address=:7002 --grpc-address=:9002 --http-address=:8002 --data-directory=/tmp/cete/node3 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost

You can access the cluster by adding a flag, such as the following command:

$ ./bin/cete cluster --grpc-address=:9000 --certificate-file=./cert.pem --common-name=localhost | jq .

or

$ curl -X GET https://localhost:8000/v1/cluster --cacert ./cert.pem | jq .