Inverted order of flushing to Influx when using buffering from kafka_consumer input

[DerMika]

We have a Telegraf setup which consumes from Kafka topics using the kafka_consumer input plugin and flushes those messages to Influx using the Influxdb output.

We buffer up to 400 messages to speed up processing, and we have found out that the order in which Telegraf pushes the data to influx is inverted for the buffered measurements

For example, Kafka topic contains:

Partition	Offset	Key	Message
1	0	1	PLANT,id=a active_power=4300 1575992352000000000
1	1	1	PLANT,id=a active_power=5600 1575992352000000000

Which Telegraf will POST to the Influx REST endpoint in 1 HTTP message (due to buffering)

We noticed that the value saved in Influx for the active_power measurement above will be 4300, instead of the expected 5600.

We also sniffed the traffic between telegraf and the Influx REST interface, and noticed that Telegraf's POST contains the above measurements in inverted order.

Relevant telegraf.conf:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 400
  metric_buffer_limit = 2000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  logfile = ""
  hostname = ""

[[inputs.kafka_consumer]]
  topics = ["input.raw"]
  version = "2.3.1"
  brokers = ["broker:9092"]
  consumer_group = "influxsink"
  offset = "oldest"
  max_undelivered_messages = 400
  data_format = "influx"
 [inputs.kafka_consumer.tags]
   retention = "raw"

[[outputs.influxdb]]
  urls = ["$INFLUX_URL"]
  database = "$INFLUX_DATABASE"
  precision = "s"
  timeout = "5s"
  retention_policy = "autogen"
  username = "$INFLUX_USERNAME"
  password = "$INFLUX_PASSWORD"

System info:

Telegraf 1.12.6
Influx 1.7.9

Both running the official Influx docker images influxdb:1.7.9-alpine and telegraf:1.12.6-alpine

Steps to reproduce:

I've created a docker-compose reproduction scenario with Kafka, Influx and Telegraf, please follow the instructions to reproduce here: https://bitbucket.org/mind4energy/reproduce-telegraf/src/master/README.md

Expected behavior:

I expect the value for active_power to be saved to influx to be the last value (5600) when querying the database

Actual behavior:

The value for active_power saved to influx is the oldest (obsolete) value - 4300.

Additional info:

[danielnelson]

Let's merge this issue with #5633.