Bunshin (分身) is Japanese term that means "clone" or "replication". It was a word popularised by Naruto (ナルト) , one of our favourite Anime Series. (If you haven't watched it, you really should)
In the series, the main character, Naruto has a signature technique called Kage Bunshin No Jutsu ((影分身の術), which translates to Shadow Replication Technique. By using this technique widely, Naruto was often able to devise interesting battle strategies and survive enemy attacks, triumphing in almost every battle he gets into. In other words, it is his ability to create many clones at once that allows Naruto to be Highly Available and Fault Tolerant.
Additionally, the replication technique in Naruto is not strongly consistent. Turns out, Masashi Kishimoto, (岸本 斉史) the author of Naruto, has taken extra effort to imbue realistic assumptions into this replication technique. In Chapter 315 of the manga series, Kakashi, Naruto's sensei, showed Naruto how knowledge transfer works amongst clones. Basically, knowledge amongst the clones of Naruto are synchronised only when a clone disappears -he knowledge of a clone is transferred to all other clones when it disappears. In other words, Naruto's clones are Eventually Consistent. By using this technique, Naruto was able to speed up his training and achieve mastery over extremely difficult techniques in record time by having all his clones do the training together.
Hence, BunshinDB is a fitting name for the system, given how we design our system to be Highly Available, Fault Tolerant and Eventually Consistent, using techniques described in Amazon's Dynamo.
System Components:
- Stethoscope Server -> Checks liveness of Ring Server and Node Servers
- Ring Server -> Assign Node Servers different positions in the ring and ensures each node is aware of the hash ring
- Node Servers -> Coordinates read and write requests based on their position on the ring and the hash of the key
- Client -> Serves as the backend for the shopping website relays get and put requests from the users
- Shopping Site -> Simple e-commerce web prototype
- Dashboard GUI -> Visual overlay on the ring structure and the nodes' liveness status
- High availability of Reads and Writes
- Even load distribution with Consistent Hashing
- Fast and consistent performance
- Sloppy Quorum allows reads and writes operations to be completed without intermittent failures or congestion affecting the operations
- Fault tolerance
- Nodes have a set of replicas based on their positions on the ring
- In the case of temporary failures, data is stored temporarily by the replica and the replicas can serve the requests for the primary
- In the case of permanent failures, the failed node is removed from the ring and at most K/N keys are reshuffled amongst the remaining N nodes.
- Horizontal scalability
- Distributed Key Value database across different computers
- Eventually consistent
- Data versioning with vector clocks ensures that items across different nodes will be up to date eventually
Glog -> github.com/golang/glog
Badger -> github.com/dgraph-io/badger/
Hugo -> https://gohugo.io/
go get github.com/golang/glog
go get github.com/dgraph-io/badger/
brew install hugo
For better Sanity check, we created a Frontend Monitoring Tool using React, which allows us to know the updated state of the Ring and status of all the Nodes, updated at the resolution of 1 second.
#please ensure you have npm installed
cd ConHashRing
npm install
npm run start
Please see below for a gif demo of how the monitoring tool works.
Static shopping site without Bunshin set up
https://chongyicheng.github.io/BunshinDB/
Execution flow -> Stethoscope -> RingServer -> NodeServer(s) -> Client(s) -> FrontEnd
#To run a stethoscope
go run cmd/stethoServer.go
#To run a ringserver
go run cmd/ringServer.go
#To run a nodeServer
go run cmd/nodeServer.go <portNumber> <pathToDbFiles> <nodeId> <shouldRegister true|false>
Note: The last argument, `shouldRegister` is either `"true"` or `"false"`.
Set it to false if we want to simulate a revival of the node.
#To run a client
go run cmd/client.go <portNumber>
#To run frontend
cd ShoppingSite
hugo server
After setting up hugo server, visit localhost:1313 on a web browser.
Shopping Cart data is stored in BunshinDB, easily save and update your shopping cart from any device.
To run a nodeServer
go run cmd/nodeServer.go 8080 /tmp/badger8080 1
help
To run a client
go run cmd/client.go 9000
help
Fault Detection
go run cmd/ringServer.go
go run cmd/stethoServer.go
go run cmd/nodeServer 8081 /tmp/badger8081 1 true