Kafka producer and consumer tool in protobuf format.
- Consume and produce messages using Protobuf protocol
- Trace messages with Jaeger
- Create custom templates for one or multiple messages and produce them to Kafka
go install github.com/kuper-tech/protokaf@latest
Configuration file is optional, so you can skip this section.
In order for Protokaf to work, it needs to know how to reach your Kafka broker. First option is to provide --broker
each time you invoke Protokaf. Another option is to use a configuration file. You can provide Protokaf with a configuration file with option -F ..
on the command line. Or by default Protokaf will search its config files in .protokaf.yaml
and $HOME/.protokaf.yaml
respectively.
Example of .protokaf.yaml
debug: true
broker: "<addr>:<port>"
kafka-auth-dsn: "SCRAM-SHA-512:<namespace>:<passwd>"
proto: "<dir>/<protofile>"
$ protokaf help
$ protokaf list [-t <topic>(,<topic>...)]
1 brokers:
broker 1 "127.0.0.1:9093"
2 topics:
topic "test-topic", partitions: 1
partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]
topic "test", partitions: 1
partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]
$ protokaf produce -h
This proto file will be used in the examples below.
api/example.proto
syntax = "proto3";
package example;
message HelloRequest {
string name = 1;
int32 age = 2;
}
A simple produce message
$ protokaf produce HelloRequest \
--broker kafka:9092 \
--proto internal/proto/testdata/example.proto \
--topic test \
--data '{"name": "Alice", "age": 11}'
Produce message with headers
$ protokaf produce HelloRequest \
--broker kafka:9092 \
--proto internal/proto/testdata/example.proto \
--topic test \
--header "priority=high" \
--header "application=protokaf" \
--data '{"name": "Alice", "age": 11}'
Produce message with template
$ protokaf produce HelloRequest \
--broker kafka:9092 \
--proto internal/proto/testdata/example.proto \
--topic test \
--data '{"name": {{randomFemaleName | quote}}, "age": {{randomNumber 10 20}}}' \
--count 10 \
--seed 42
Produce message with Kafka auth
$ protokaf produce HelloRequest \
--broker kafka:9093 \
--kafka-auth-dsn "SCRAM-SHA-512:login:passwd" \
--proto internal/proto/testdata/example.proto \
--topic test \
--data '{"name": "Alice", "age": 11}'
Read data from stdin or flag
Read message HelloRequest
from stdin
, produce to test
topic
$ echo '{"name": "Alice", "age": 11}' | protokaf produce HelloRequest -t test
Read message HelloRequest
from -d
value, produce to test
topic
$ protokaf produce HelloRequest -t test -d '{"name": "Alice", "age": 11}'
Template options
--seed <int>
You can set number greater then zero to produce the same pseudo-random sequence of messages--count <int>
Useful for generating messages with random data--concurrency <int>
Number of message senders to run concurrently for const concurrency producing
Show all template functions
$ protokaf produce --template-functions-print
This can be useful for creating body for produce command
$ protokaf build HelloRequest --proto internal/proto/testdata/example.proto
For proto file
syntax = "proto3";
package example;
message HelloRequest {
enum Status {
PENDING = 0;
COMPLETED = 1;
}
string name = 1;
int32 age = 2;
optional float amount = 4;
Status status = 5;
repeated string keys = 6;
}
command will print
{
"name": "",
"age": 0,
"amount": 0,
"status": "PENDING",
"keys": [
""
]
}
$ protokaf help consume
$ protokaf consume HelloRequest \
--broker kafka:9092 \
--proto internal/proto/testdata/example.proto \
--group mygroup \
--topic test
Read messages from Kafka test
topic, use group mygroup
, print to stdout
$ protokaf consume HelloRequest -G mygroup -t test
Read the last 10
messages from test
topic, then exit
$ protokaf consume HelloRequest -G mygroup -t test -c 10
Read from offset 5
messages from test
topic
$ protokaf consume HelloRequest -G mygroup -t test -o 5
make docker-dev-up
make kafka-users
make test
make install # optional (you can use 'go run . <args> <flags>')