Support the existing JSON output format as an input format for relevant service inputs.

[mircopolo]

Feature Request

Support the existing JSON output protocol as an input format for service inputs such as UDP, TCP and Kafka..

Proposal:

It would be a very useful feature for to be able to use the existing JSON Output format as an input to the service plugins.

Current behavior:

As per existing input format documentation.

Desired behavior:

Allow the current Telegraf Output JSON format to be used as an input format for all service inputs that currently support the input JSON format, this would allow for a simple means for users to specify dynamic measurement names, and a clear definition of what is a tags and what is a field.

Use case:

Currently using logstash and elasticsearch for metric/transport and storage. The use case for me is that I would like to slowly extend this to include influxdb and kapacitor (with a view to ultimately having all metrics in influxdb instead). In the interim I would like to start storing some existing metrics in both ES and InfluxDB.

The main problem for me is that the current InfluxDb logstash output plugin is not well maintained, and although I know the option exist, I'd rather not have to use the graphite plugin instead as I want to avoid its limitations / complexity of tempting when used as an influxdb input format. For almost all of my existing metrics, it would be a trivial task to transform these into the JSON output format, and use telegraf as a proxy for InfluxDb / Kapacitor via one of the Service inputs.

I am aware of #1363 (so please close this if you feel it belongs there) but I think that if possible this proposed feature would address a small subset of the use cases, and I would imagine would be a relatively trivial change by comparison.

[sparrc]

sure, we could name it influx-json and have it as it's own data format

[rconn01]

I agree this would be an excellent feature to have. To add an additional use case for this request.

It would great to have the kafka input and output plugins talk the same json format. This way all telegraf agents could write their data into kafka and additional agents could be listening to that that topic and processing it accordingly to other sources. With the current JSON formats there doesn't appear to be a good way to chain telegraf instances together with kafka. The only way to achieve this is with the influx format

[sparrc]

that is correct, though even once this feature is implemented I'd always recommend using the influx format, as parsing it uses less cpu and memory than JSON.

[dgalichet]

Hi, is there any status update on this ?

[glinton]

@dgalichet what part of this are you looking for an update on? Input plugins generally allow the input format to be configured via data_format="..."

[JulienChampseix]

@glinton @dgalichet is talking about a "real" json support as input format IMHO as initialy requested in this issue (for a more flexible json parser #4912)

[danielnelson]

I think he is asking about roundtripping the JSON format, both of these options are still wanted however no one is working on them right now.

[dgalichet]

yes I mean round-tripping json format. I would like to be able to use kafka as a media between local collectors and influxdb, using the json format because I want other systems to read logs (json seems a good format for that point).
So I need to be able to read json outputed by telegraf from a telegraf input

[danielnelson]

Sorry, that linked pr is not for this issue.

It occurs to me that one major issue with round-tripping the JSON output format is that we will lose the information about if a field is an integer or a float, which will result in it not being possible to convert back into line protocol in a way that is compatible with InfluxDB.

[sjwang90]

Has anyone here tried round-tripping with the new json_v2 parser? @mircopolo @dgalichet

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!