InfluxDB Line Protocol parser should support partial writes

[piotrp]

Feature Request

InfluxDB Line Protocol parser (plugins/parsers/influx) should be able to reject invalid measurements while still allowing to write properly parsed data - making it treat errors the same way that InfluxDB does. That would allow to use influxdb_listener as a proxy for filtering data.

Proposal:

Make parser capable of rejecting only malformed measurements. Log errors in a way that can be filtered out from being put into Telegraf log (so that known offenders don't clobber log file) or keep current logging behavior and silently drop them.

Current behavior:

Malformed measurement causes parser to reject entire payload.

Desired behavior:

Parity with InfluxDB: rejecting only malformed data. influxdb_listener could respond the same way that InfluxDB does: with {"error":"partial write: unable to parse '...': ..."} response returned as HTTP 400 Bad Request.

Example:

[[inputs.influxdb_listener]]
  service_address = ":8186"

echo -e "test_metric1,host=server01 value=∞ 1434055562000000000\ntest_metric2,host=server01 value=0.64 1434055562000000000" | curl -i -XPOST 'http://localhost:8186/write' --data-binary @-

should result in test_metric2 being accepted.

Use case:

Preprocessing of metrics from applications supporting InfluxDB Line Protocol before they get inserted into InfluxDB.

In my case the problem is Apache Flink, which reports a lot tags with UUIDs that I want to drop. Currently it's not possible with Telegraf because it reports verbatim metrics from Kafka, which uses infinity/-infinity (send via line protocol by InfluxDB-Java library as ∞/-∞) to denote uninitialized metrics.

[danielnelson]

Thanks, we have an existing issue for this, #6124, so I'm going to close this one.

Would be interested to know more about the ∞/-∞ behavior in InfluxDB-Java, can you show me an example of line protocol output with this? Is it considered a bug in InfluxDB-Java or an extension?

[piotrp]

@danielnelson

Discussion on these infinity values can be found Apache Flink bug FLINK-12147 (and its PR apache/flink#8513).

InfluxDB-Java writes points using NumberFormat, which outputs point values that are equal to eg. Double.POSITIVE_INFINITY as ∞ (Point#concatenatedFields).