-
Notifications
You must be signed in to change notification settings - Fork 207
Getting Started with BFT SMaRt
In this page we describe how to create a simple Map application using BFT-SMaRt. The complete code for the application described here can be found in the src/bftsmart/demo/map/ directory of the BFT-SMaRt distribution folder.
First, download the latest stable version of BFT-SMaRt from our site or from the repository.
BFT-SMaRt's code must be installed in each replica and client that will use it. To do so, first, extract the downloaded archive. After that, copy the following files and folders to each of the replicas and clients:
- bin/BFT-SMaRt.jar
- config/
- lib/
After copying the files to the replicas, the first step is to get the IP addresses from each replica and define a port for each one to receive the messages from other replicas. After that, edit the file config/hosts.config in each replica to set the IP address and port for each one. The information must be the same across all replicas. Let's use as example this configuration:
0 127.0.0.1 10001
1 127.0.0.2 10001
2 127.0.0.3 10001
3 127.0.0.4 10001
For each line, the first parameter is the replica ID. The second parameter is the IP address and the third is the port. This information must be the same across all replicas.
If you wish to install all BFT-SMaRt replicas in the same host, do not forget to assign different ports to each replica, like this:
0 127.0.0.1 10000
1 127.0.0.1 11000
2 127.0.0.1 12000
3 127.0.0.1 13000
By now you should be able to start replicas and run demo examples included in BFT-SMaRt binary. Instructions to run these demos can be found at our Demos page.
Below, we describe how to use BFT-SMaRt to build your own application. We will take as an example a implementation of the java.util.Map interface. In this example, multiple clients will be able to perform operations in a TreeMap, whose data will be replicated among several replicas.
The implementation uses generic types to allow key-value pairs to be of any type (in this example, key-value pairs are Strings).
Firstly, all operations supported by the application are specified in the following enum, shared by both clients and servers:
package bftsmart.demo.map;
public enum MapRequestType {
PUT,
GET,
SIZE,
REMOVE,
KEYSET;
}
We start by creating the client implementation of the Map interface:
package bftsmart.demo.map;
public class MapClient<K, V> implements Map<K, V> {
The interface java.util.Map defines several methods to be implemented. Since the goal of this example is to show how to extend BFT-SMaRt to do your own application, we will only implement the methods put(K, V), get(Object), remove(Object), size() and keySet().
The map is implemented as a BFT-SMaRt client that interact with replicas through the class bftsmart.tom.ServiceProxy, which can interact with the replicas in a transparent way. We will also define a constructor to pass the client ID as a parameter, so the requests submitted by this client can be uniquely identified:
ServiceProxy serviceProxy;
public MapClient(int clientId) {
serviceProxy = new ServiceProxy(clientId);
}
To implement the put method, we first use ByteArrayOutputStream and ObjectOutputStream to encode <key, value> pairs into a byte array. Since this is a write operation, we use the ServiceProxy.invokeOrdered method to guarantee that all replicas observe and process the same sequence of operations.
@Override
public V put(K key, V value) {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
objOut.writeObject(MapRequestType.PUT);
objOut.writeObject(key);
objOut.writeObject(value);
objOut.flush();
byteOut.flush();
byte[] reply = serviceProxy.invokeOrdered(byteOut.toByteArray());
if (reply.length == 0)
return null;
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(reply);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
return (V)objIn.readObject();
}
} catch (IOException | ClassNotFoundException e) {
System.out.println("Exception putting value into map: " + e.getMessage());
}
return null;
}
The get method invokes ServiceProxy.invokeUnordered. This operation does not need to be ordered since it is a read-only operation, i.e., it does not modify the application's state.
@Override
public V get(Object key) {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
objOut.writeObject(MapRequestType.GET);
objOut.writeObject(key);
objOut.flush();
byteOut.flush();
byte[] reply = serviceProxy.invokeUnordered(byteOut.toByteArray());
if (reply.length == 0)
return null;
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(reply);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
return (V)objIn.readObject();
}
} catch (IOException | ClassNotFoundException e) {
System.out.println("Exception getting value from map: " + e.getMessage());
}
return null;
}
The remove method is similar to the put method, since removing a key is also a write operation.
@Override
public V remove(Object key) {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
objOut.writeObject(MapRequestType.REMOVE);
objOut.writeObject(key);
objOut.flush();
byteOut.flush();
byte[] reply = serviceProxy.invokeOrdered(byteOut.toByteArray());
if (reply.length == 0)
return null;
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(reply);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
return (V)objIn.readObject();
}
} catch (IOException | ClassNotFoundException e) {
System.out.println("Exception removing value from map: " + e.getMessage());
}
return null;
}
The size and keyset methods are similar to get, with the difference being that they do not require any parameter.
@Override
public int size() {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
objOut.writeObject(MapRequestType.SIZE);
objOut.flush();
byteOut.flush();
byte[] reply = serviceProxy.invokeUnordered(byteOut.toByteArray());
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(reply);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
return objIn.readInt();
}
} catch (IOException e) {
System.out.println("Exception reading size of map: " + e.getMessage());
}
return -1;
}
@Override
public Set<K> keySet() {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
objOut.writeObject(MapRequestType.KEYSET);
objOut.flush();
byteOut.flush();
byte[] reply = serviceProxy.invokeUnordered(byteOut.toByteArray());
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(reply);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
int size = objIn.readInt();
Set<K> result = new HashSet<>();
while (size-- > 0) {
result.add((K)objIn.readObject());
}
return result;
}
} catch (IOException | ClassNotFoundException e) {
System.out.println("Exception getting keyset from map: " + e.getMessage());
}
return null;
}
As already mentioned, we do not aim to support all methods provided by the Map interface, so we need to implement the remaining methods using the following code:
throw new UnsupportedOperationException("Not supported yet.");
BFT-SMaRt requires the replicas to implement the Executable and
Recoverable interfaces. The Executable interface defines methods to process ordered and unordered requests. The application must implement it's business logic within these methods.
The Recoverable interface defines methods to manage the application's state. It is used to store the state during system execution and fetch it back if a replica fails or is added into the system on-the-fly.
In our example, we extend the class DefaultSingleRecoverable, which provides a basic state management strategy. It extends the SingleExecutable interface, that receives one request at a time from BFT-SMaRt.
package bftsmart.demo.map;
public class MapServer<K, V> extends DefaultSingleRecoverable {
private Map<K, V> replicaMap;
private Logger logger;
public MapServer(int id) {
replicaMap = new TreeMap<>();
logger = Logger.getLogger(MapServer.class.getName());
new ServiceReplica(id, this, this);
}
public static void main(String[] args) {
if (args.length < 1) {
System.out.println("Usage: demo.map.MapServer <server id>");
System.exit(-1);
}
new MapServer<String, String>(Integer.parseInt(args[0]));
}
replicaMap is the actual object in which the data will be stored in the server. The ServiceReplica object is responsible for delivering requests and return replies. We also need to implement the main method, since MapServer is also responsible for launching the processes associated with the application and the BFT-SMaRt replication protocol.
Because BFT-SMaRt provides execution of ordered and unordered requests, we need to implement the abstract methods appExecuteOrdered and appExecuteUnordered. Operations that are of type writes (put, remove) and reads (get, size, keyset) are implemented in appExecuteOrdered. Read operations are implemented both in appExecuteOrdered and appExecuteUnordered.
The implementation of these two methods are similar. Each one start by identifying the operation type, followed by reading the respective arguments (if any). The operations are then applied to replicaMap and the result is returned by the method, to eventually be delivered to the client that issued the operation.
@Override
public byte[] appExecuteOrdered(byte[] command, MessageContext msgCtx) {
byte[] reply = null;
K key = null;
V value = null;
boolean hasReply = false;
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(command);
ObjectInput objIn = new ObjectInputStream(byteIn);
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
MapRequestType reqType = (MapRequestType)objIn.readObject();
switch (reqType) {
case PUT:
key = (K)objIn.readObject();
value = (V)objIn.readObject();
V oldValue = replicaMap.put(key, value);
if (oldValue != null) {
objOut.writeObject(oldValue);
hasReply = true;
}
break;
case GET:
key = (K)objIn.readObject();
value = replicaMap.get(key);
if (value != null) {
objOut.writeObject(value);
hasReply = true;
}
break;
case REMOVE:
key = (K)objIn.readObject();
value = replicaMap.remove(key);
if (value != null) {
objOut.writeObject(value);
hasReply = true;
}
break;
case SIZE:
int size = replicaMap.size();
objOut.writeInt(size);
hasReply = true;
break;
case KEYSET:
keySet(objOut);
hasReply = true;
break;
}
if (hasReply) {
objOut.flush();
byteOut.flush();
reply = byteOut.toByteArray();
} else {
reply = new byte[0];
}
} catch (IOException | ClassNotFoundException e) {
logger.log(Level.SEVERE, "Ocurred during map operation execution", e);
}
return reply;
}
@SuppressWarnings("unchecked")
@Override
public byte[] appExecuteUnordered(byte[] command, MessageContext msgCtx) {
byte[] reply = null;
K key = null;
V value = null;
boolean hasReply = false;
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(command);
ObjectInput objIn = new ObjectInputStream(byteIn);
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut);) {
MapRequestType reqType = (MapRequestType)objIn.readObject();
switch (reqType) {
case GET:
key = (K)objIn.readObject();
value = replicaMap.get(key);
if (value != null) {
objOut.writeObject(value);
hasReply = true;
}
break;
case SIZE:
int size = replicaMap.size();
objOut.writeInt(size);
hasReply = true;
break;
case KEYSET:
keySet(objOut);
hasReply = true;
break;
default:
logger.log(Level.WARNING, "in appExecuteUnordered only read operations are supported");
}
if (hasReply) {
objOut.flush();
byteOut.flush();
reply = byteOut.toByteArray();
} else {
reply = new byte[0];
}
} catch (IOException | ClassNotFoundException e) {
logger.log(Level.SEVERE, "Ocurred during map operation execution", e);
}
return reply;
}
private void keySet(ObjectOutput out) throws IOException, ClassNotFoundException {
Set<K> keySet = replicaMap.keySet();
int size = replicaMap.size();
out.writeInt(size);
for (K key : keySet)
out.writeObject(key);
}
State management is needed to support recovery of replicas. DefaultSingleRecoverable provides developers with a basic state management strategy that uses periodic checkpoints. These checkpoints are created by taking a snapshot of the application state and storing the following requests into a log.Once that log reaches a predetermined height, a new snapshot is taken and the log discarded. When a replica asks for a state transfer, DefaultSingleRecoverable sends the latest snapshot and correspondent log to that replica.
It is the onus of the developer to define how the state is encoded (when taking a periodic snapshot) and decoded (when installing a snapshot on the recovered replica). This must be implemented, respectively, in the abstract methods getSnapshot and installSnapshot. Whereas getSnapshot is invoked periodically, installSnapshot might be invoked when (1) replicas that are to late to process requests jump to a more up-to-date state, or (2) replicas that were once failed/crashed re-start their execution from scratch.
Below there are simple implementations of these methods, using a methodology similar to the one used to send and receive client requests.
@Override
public byte[] getSnapshot() {
try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
ObjectOutput objOut = new ObjectOutputStream(byteOut)) {
objOut.writeObject(replicaMap);
return byteOut.toByteArray();
} catch (IOException e) {
logger.log(Level.SEVERE, "Error while taking snapshot", e);
}
return new byte[0];
}
@SuppressWarnings("unchecked")
@Override
public void installSnapshot(byte[] state) {
try (ByteArrayInputStream byteIn = new ByteArrayInputStream(state);
ObjectInput objIn = new ObjectInputStream(byteIn)) {
replicaMap = (Map<K, V>)objIn.readObject();
} catch (IOException | ClassNotFoundException e) {
logger.log(Level.SEVERE, "Error while installing snapshot", e);
}
}
To test this replicated Map implementation, you can use the following console application to perform operations in the classes we described.
package bftsmart.demo.map;
import java.io.Console;
import java.util.Set;
public class MapInteractiveClient {
public static void main(String[] args) {
if(args.length < 1) {
System.out.println("Usage: demo.map.MapInteractiveClient <client id>");
}
int clientId = Integer.parseInt(args[0]);
MapClient<String, String> map = new MapClient<>(clientId);
Console console = System.console();
boolean exit = false;
String key, value, result;
while(!exit) {
System.out.println("Select an option:");
System.out.println("0 - Terminate this client");
System.out.println("1 - Insert value into the map");
System.out.println("2 - Retrieve value from the map");
System.out.println("3 - Removes value from the map");
System.out.println("4 - Retrieve the size of the map");
System.out.println("5 - List all keys available in the table");
int cmd = Integer.parseInt(console.readLine("Option:"));
switch (cmd) {
case 0:
map.close();
exit = true;
break;
case 1:
System.out.println("Putting value in the map");
key = console.readLine("Enter the key:");
value = console.readLine("Enter the value:");
result = map.put(key, value);
System.out.println("Previous value: " + result);
break;
case 2:
System.out.println("Reading value from the map");
key = console.readLine("Enter the key:");
result = map.get(key);
System.out.println("Value read: " + result);
break;
case 3:
System.out.println("Removing value in the map");
key = console.readLine("Enter the key:");
result = map.remove(key);
System.out.println("Value removed: " + result);
break;
case 4:
System.out.println("Getting the map size");
int size = map.size();
System.out.println("Map size: " + size);
break;
case 5:
System.out.println("Getting all keys");
Set<String> keys = map.keySet();
System.out.println("Total number of keys found: " + keys.size());
for (String k : keys)
System.out.println("---> " + k);
break;
default:
break;
}
}
}
}
To execute the test, start by launching each server from the one with the lowest ID to the one with the highest. After all servers are ready, you can launch the console client.