DISCONTINUATION OF PROJECT.
This project will no longer be maintained by Intel.
This project has been identified as having known security escapes.
Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
Intel no longer accepts patches to this project.
This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.
Allows publishing of data to Apache Kafka
It's used in the snap framework.
- Uses sarama golang client for Kafka by Shopify
You can get the pre-built binaries for your OS and architecture at plugin's GitHub Releases page.
Fork https://github.com/intelsdi-x/snap-plugin-publisher-kafka
Clone repo into $GOPATH/src/github.com/intelsdi-x/:
$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-kafka.git
Build the plugin by running make within the cloned repo:
$ make
This builds the plugin in ./build
- Set up the snap framework
In task manifest, in config section of Kafka publisher the following settings can be declared:
| Key | Type | Default value | Description |
|---|---|---|---|
| topic | string | "snap" | The topic to send messages |
| brokers | string | "localhost:9092" | Semicolon delimited list of "server:port" brokers Kafka's standard broker communication port is 9092 |
Notice To publish to remote Kafka endpoint, please ensure that all listed brokers are available from the host where snap-plugin-publisher-kafka is running. You can simply check it by running telnet providing address and port where Kafka advertised broker is running, for example telnet 123.45.67.89 9092.
This is a minimal-configuration needed to run the Kafka broker service on Docker
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.
$ docker run -d --name zookeeper jplock/zookeeper:3.4.6
Check if ZooKeeper docker is running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b0ddbdd75cd jplock/zookeeper:3.4.6 "/opt/zookeeper/bin/ 38 seconds ago Up 38 seconds 2181/tcp, 2888/tcp, 3888/tcp zookeeper
docker run -d --name kafka --link zookeeper:zookeeper ches/kafka
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dfb3cdfb3f87 ches/kafka:latest "/start.sh" 7 seconds ago Up 6 seconds 7203/tcp, 9092/tcp kafka
9b0ddbdd75cd jplock/zookeeper:3.4.6 "/opt/zookeeper/bin/ 3 minutes ago Up 3 minutes 2181/tcp, 2888/tcp, 3888/tcp zookeeper
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka
172.17.0.14
Read more about Kafka on http://kafka.apache.org
The plugin publishes all collected metrics serialized as JSON to Kafka. An example of published data is below:
[
{
"timestamp": "2016-07-25T11:27:59.795548989+02:00",
"namespace": "/intel/mock/bar",
"data": 82,
"unit": "",
"tags": {
"plugin_running_on": "my-machine"
},
"version": 0,
"last_advertised_time": "2016-07-25T11:27:21.852064032+02:00"
},
{
"timestamp": "2016-07-25T11:27:59.795549268+02:00",
"namespace": "/intel/mock/foo",
"data": 72,
"unit": "",
"tags": {
"plugin_running_on": "my-machine"
},
"version": 0,
"last_advertised_time": "2016-07-25T11:27:21.852063228+02:00"
}
]Example of running psutil collector plugin and publishing data to Kafka.
Set up the Snap framework
Ensure Snap daemon is running:
- initd:
service snap-telemetry start - systemd:
systemctl start snap-telemetry - command line:
sudo snapteld -l 1 -t 0 &
Download and load Snap plugins (paths to binary files for Linux/amd64):
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-kafka/latest/linux/x86_64/snap-plugin-publisher-kafka
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-psutil/latest/linux/x86_64/snap-plugin-collector-psutil
$ snaptel plugin load snap-plugin-publisher-kafka
$ snaptel plugin load snap-plugin-collector-psutil
Create a task manifest (see exemplary tasks),
for example psutil-kafka.json with following content:
{
"version": 1,
"schedule": {
"type": "simple",
"interval": "1s"
},
"workflow": {
"collect": {
"metrics": {
"/intel/psutil/load/load1": {},
"/intel/psutil/load/load15": {}
},
"publish": [
{
"plugin_name": "kafka",
"config": {
"topic": "test",
"brokers": "172.17.0.14:9092"
}
}
]
}
}
}Create a task:
$ snaptel task create -t psutil-kafka.json
Watch created task:
$ snaptel task watch <task_id>
To stop previously created task:
$ snaptel task stop <task_id>
There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.
If you have a feature request, please add it as an issue and/or submit a pull request.
This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework
We love contributions!
There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.
Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.
- Author: Nicholas Weaver
And thank you! Your contribution, through code and participation, is incredibly important to us.