Ziften Kafka Example

This app is an example on how to consume Ziften Protocolbuffers straight from Ziften's Kafka topic.

Running the example

A prebuilt Java 8 jar is in /build

Run from command line:

$ java -jar build/ziften-kafka-example.jar ec2-54-16-12-14.compute.amazonaws.com:9092 ZIFTEN.DATACOLLECTION_

• First parameter is the Kafka server string: remote_host:9092
• Second paramter is the topic name: ZIFTEN.DATACOLLECTION_

In some installations the topic name could be different. Contact Ziften to find out the exact name.

Building and using the protocol buffers library and Ziften's helper functions

The Ziften DATACOLLECTION Topic has messages in this format:

<pb type="MessageType" hmac="">BASE64==</pb>

Each type is associated with a protobuf message. The text of the pb element is a base64 encoded protobuf. You can use the type to determine what protobuf message you need to use to decode the message. Some helper utilties were provided.

 String xmlMSg=" <pb type="MessageType" hmac="">BASE64aaabbbccccc==</pb>"
 AbstractMessage abstractMessage = new ProtobufMessageUtil().decode(xmlMsg);
 System.out.printf("\n\nFound protobuf message: %s", abstractMessage.getClass().getSimpleName());
 System.out.println("\n\t" + ProtobufToJSON.toJSONObjectModel(abstractMessage));  

• Please note that ProtobufTOJSON now actually returns a Map<String,Object>.

A jar containing just the protobuffer messages is at:

• /protocol-0.0.2-SNAPSHOT.jar

An example of consuming off a Kafka topic is also provided. Ziften uses Kafka 0.10. The consumer was refactored in Kafka 0.9, so please consult the 0.10 documentaiton if you are not familiar: https://kafka.apache.org/documentation

To build the whole project: mvn installl

This should result in a new shaded jar at target/kafka-example-jar At this point you run it the same way as the provided example jar:

    $ java -jar build/ziften-kafka-example.jar ec2-54-161-12-134.compute.amazonaws.com:9092 ZIFTEN.DATACOLLECTION_

Ziften server changes

The following are currently required changes to be made for remote Kafka connection

vim /opt/ziften/kafka/config/server.properties

Replace the line

advertised.listeners=PLAINTEXT://localhost:9092

with

advertised.listeners=PLAINTEXT://NEW_IP:9092

Where NEW_IP is an ip (or hostname) that is accessible from both the server and the location the new consumer will run. This is required because Ziften only listens on localhost by default.

Make sure the port is open:

iptables -A INPUT -p tcp --dport 9092 -j ACCEPT

Restart the Kafka service:

service ziften restart kafka

Verify Kafka is back up and running. This should list all topics:

/opt/ziften/kafka/binkafka-topics.sh --list --zookeeper localhost:2181