After 1.13.1 upgrade, writes to kafka fail if timestamps are invalid

[sjafferali]

I use telegraf to relay metrics to kafka. Was previously running 1.11.2. After upgrading to 1.13.1, I noticed the metric buffers filling up on my telegraf instances. It appears to be due to invalid timestamps that are received to telegraf.

Relevant telegraf.conf:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 100000
  metric_buffer_limit = 1000000
  collection_jitter = "5s"
  flush_interval = "10s"
  flush_jitter = "5s"
  precision = "1ns"
  debug = false
  quiet = false
  logfile = "/var/log/telegraf/telegraf.log"
  hostname = ""
  omit_hostname = true

[[inputs.influxdb_listener]]
  service_address = ":8094"
  read_timeout = "60s"
  write_timeout = "60s"
  max_body_size = 0
  max_line_size = 0

[[outputs.kafka]]
  brokers = ["host1:6667","host2:6667","host3:6667","host4:6667"]
  topic = "topicname"
  version = "0.10.2.1"
  routing_key = "random"

System info:

Ubuntu 16
Telegraf Version 1.13.1

Steps to reproduce:

Upgrade to 1.13.1
Send a metric similar to the below line.

telegraf_metric,component=metrics row_count=157796669i -7240509273709551616

Expected behavior:

The metric should get dropped or we should have the option to write metrics into kafka excluding the timestamp, such as previous versions, which wrote the message to the topic regardless of the timestamp.

Actual behavior:

The below error will get logged.

[agent] Error writing to outputs.kafka: kafka: Failed to produce message to topic topicname: kafka: error encoding packet: invalid timestamp (1740-07-22 09:32:28.290448384 -0752 LMT)

The write to kafka fails, but the metric itself does not get dropped causing the metric buffer to fill up.

Additional info:

This functionality was implemented in #6746, however, it causes telegraf to handle messages with invalid timestamps much differently. Previously, metrics would get written regardless of the timestamp value, which is no longer the case.

As this is a limitation with kafka, adding an option to exclude the timestamp from being written into kafka would make this function like previous versions.

[danielnelson]

Thanks for opening the issue, we will make sure to have a fix for this in 1.13.2. However, I just wanted to double check that you are expecting to have metrics from 1740 and that there isn't another issue around the creation of these old timestamps.

[sjafferali]

Thanks for opening the issue, we will make sure to have a fix for this in 1.13.2. However, I just wanted to double check that you are expecting to have metrics from 1740 and that there isn't another issue around the creation of these old timestamps.

Yes, we have some apps that emit invalid timestamps due to a bug in the apps themselves. Although, it generally doesn't cause any issues, as we filter them out further down in our pipeline.

Thanks!