Cannot connect to Azure EventHub using inputs.kafka_consumer

[OnurbHartmann]

The Microsoft Azure EventHub exposes a Kafka 1.0 compatible protocol. With that we tried to connect using the inputs.kafka_consumer and SASL Authentication. However it won't work.
As described in the sarama issue below even though Microsoft implemented 1.0 protocol the SASL authentication works only with V0. This cannot be configured via telegraf.conf yet.

Relevant telegraf.conf:

[[inputs.kafka_consumer]]
#   ## kafka servers
   brokers = ["<namespace>.servicebus.windows.net:9092"]
#   ## topic(s) to consume
   topics = ["<Event Hub>"]
#   ## Add topic as tag if topic_tag is not empty
#   # topic_tag = ""
#
#   ## Optional Client id
    client_id = "AzEvHub"
#
#   ## Set the minimal supported Kafka version.  Setting this enables the use of new
#   ## Kafka features and APIs.  Of particular interest, lz4 compression
#   ## requires at least version 0.10.0.0.
#   ##   ex: version = "1.1.0"
    version = "1.1.0"
#
#   ## Optional TLS Config
#   # tls_ca = "/etc/telegraf/ca.pem"
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#   ## Use TLS but skip chain & host verification
#   # insecure_skip_verify = false
#
#   ## Optional SASL Config
    sasl_username = "<username>"
    sasl_password = "<password>"
#
#   ## the name of the consumer group
   consumer_group = "telegraf"
#   ## Offset (must be either "oldest" or "newest")
   offset = "oldest"
#   ## Maximum length of a message to consume, in bytes (default 0/unlimited);
#   ## larger messages are dropped
   max_message_len = 1000000
#
#   ## Maximum messages to read from the broker that have not been written by an
#   ## output.  For best throughput set based on the number of metrics within
#   ## each message and the size of the output's metric_batch_size.
#   ##
#   ## For example, if each message from the queue contains 10 metrics and the
#   ## output metric_batch_size is 1000, setting this to 100 will ensure that a
#   ## full batch is collected and the write is triggered immediately without
#   ## waiting until the next flush_interval.
#   # max_undelivered_messages = 1000
#
#   ## Data format to consume.
#   ## Each data format has its own unique set of configuration options, read
#   ## more about them here:
#   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
   data_format = "influx"

System info:

• Latest Azure EventHub with activated Kafka protocol
• Telegraf 1.12.0-rc1,

Steps to reproduce:

1. create a azure EventHub with dedicated consumer group, poicies etc...
2. send data to EvHub using e.g. AzureFunctions
3. setup telegraf with the upper config

Expected behavior:

Read events from eventhub

Actual behavior:

Get a timeout from sarama client
2019-09-02T07:38:19Z D! [sarama] client/metadata fetching metadata for all topics from broker *.servicebus.windows.net:9092 2019-09-02T07:38:49Z D! [sarama] Failed to connect to broker *.servicebus.windows.net:9092: dial tcp 13.69.64.0:9092: i/o timeout 2019-09-02T07:38:49Z D! [sarama] client/metadata got error from broker -1 while fetching metadata: dial tcp 13.69.64.0:9092: i/o timeout

Additional info:

https://community.influxdata.com/t/telegraf-to-read-azure-event-hub-via-inputs-kafka-consumer/11092/3
IBM/sarama#1408

[danielnelson]

Here are some testing build that add a new option sasl_version to the kafka_consumer input. It can be set to either 0 or 1:

sasl_version = 0

I suspect this won't be enough to get it working, as version 0 should already be the default.

• telegraf-1.13.0~366a2e14-0.x86_64.rpm
• telegraf_1.13.0~366a2e14-0_amd64.deb
• telegraf-1.13.0~366a2e14_windows_amd64.zip

[OnurbHartmann]

Your are right ... that did not help.
Maybe I need to setup sarama on my vm and check directly.
Or any other thoughts?
Thx

[R290]

I don't know if Kafka is a strict requirement, but I'm working on an plugin based on the default AMQP back-end: https://github.com/R290/telegraf/tree/eventhub

The pull request is closed at the moment due to telegraf lacking go mod support.

[danielnelson]

I've merged the sasl_version option in for Telegraf 1.14, and also identified an issue setting up TLS that had kept this from working previously. It should be possible to use the next nightly build to connect to Azure Event Hub with both the kafka input and output plugins, although you may still run into this error:

response did not contain all the expected topic/partition blocks

This is an issue on the Event Hub side but there is a fix in place that is currently being rolled out. If you hit this, you may need to hold tight for a bit longer but it should resolved soon.