Support JSON Path-like json parsing #1363

sparrc · 2016-06-10T16:30:59Z

Feature Request

Proposal:

Support something similar to JSONPath for customizing which fields/paths of json get picked up as tags/fields/types/etc.

Doesn't need to be JSONPath specifically, since TBH the official json path spec is fairly ugly. Something more simple like https://github.com/jmoiron/jsonq would work as well.

Current behavior:

flattens json into numerical fields only, grabs tags only from the top level.

Desired behavior:

more customization to deal better with diverse range of different json formats.

Use case: [Why is this important (helps with prioritizing requests)]

Telegraf frequently gets issues opened for users who want specific changes & modifications to the json parser for their particular use-case. These cannot all be satisfied without a truly generic solution.

sparrc · 2016-09-05T12:55:01Z

this should also support parsing timestamps in the same way

fromanator · 2016-09-16T16:10:44Z

@sparrc response to your question in #1554 :
Q: "can you talk more about the drawbacks of using line-protocol? This protocol is ideal for working with influxdb and is able to be encoded & decoded much faster than JSON."

A: This is great for use cases where there's not a telegraf instance loading the data to a kafka topic for ingestion, hence it's not trivial to load the data with the influx line protocol. For my specific use case we are getting a stream of syslog lines from and external server. The syslog line is grokked into JSON and loaded into a Kafka topic. Then we have Telegraf read that kafka topic to load the metrics into a metric storage backend (influx, graphite, etc).

kevinjos · 2016-11-25T18:45:20Z

Hello, I opened #2066 and am taking a look at the more generic solution proposed here. What are you thinking in terms of configuration @sparrc ? Currently, I see that httpjson conf can..

set a global servers=$VALUE tag
specify a tag_keys array
parse top level arrays of objects to unique points in line protocol

What should a user of the generic JSON parser be able to configure in addition to the above? Should the new JSONParser be backwards compatible with the existing parser?

sparrc · 2016-11-26T11:18:07Z

@kevinjos this issue is specific to JSON parsing, not the httpjson plugin. For that reason, there will not be a change to the first bullet-point.

Ideally it will be backwards-compatible, tag_keys will continue to be supported, and if the JSON parsing rules fail then it will fall back to the current behavior, which is to parse the top-level arrays.

I haven't thought too much about what this will look like as I haven't had time to work on it, but the idea would be to create a generic way for users to specify paths to nested objects and arrays, and have custom patterns for parsing different json object types.

kevinjos · 2016-11-28T22:28:14Z

I opened a PR that introduces jsonq to the json parser to support parsing nested objects and arrays of objects into unique metrics with default, top-level and object-specific tags:

#2101

Feedback is much appreciated.

olsky · 2017-01-27T11:46:40Z

I'd like to add one more use-case here: Mqtt-consumer with Json payload, where Timestamp overwrite would be very useful.

sparrc · 2017-01-27T18:45:20Z

@olsky what do you mean by "timestamp overwrite"?

olsky · 2017-01-30T08:31:57Z

by "timestamp overwrite" I mean: use a custom timestamp (also in the past), not the instance when the message is received.

asatriani · 2017-06-16T22:40:31Z

Hi all,
I think this feature is very important because current input telegraf plugin behavior with json format is too poor. In addition to a configuration field to set the metric/event timestamp, another field to ignore numeric data in json message would be very useful

danielnelson · 2017-06-16T22:54:20Z

Anyone have opinions on JSONPath vs jsonq vs json-ptr? I think the parser could work something like what we sketched up for a XML parser on #1758.

freeseacher · 2017-09-15T13:04:58Z

any news ?

fabianbaier · 2017-11-08T10:08:15Z

has there been progress? This feature looks particular interesting and I agree with desired behavior mentioned by @dgarros in #1332

danielnelson · 2018-08-23T02:35:45Z

I merged a bunch of new features into the existing JSON parser, I think it may meet the needs for many watching this issue. Check the JSON data format for details.

There is still a place for a new parser, so I will keep this issue open.

conet · 2020-04-27T17:11:13Z

It's a shame there is still no way to parse something like this:

[
    {
        "timestamp": "2020-01-01T00:00:00.000Z", 
        "events": [
            {
                "metric_type": "metric1",
                "value": 20
            },
            {
                "metric_type": "metric2",
                "value": 10,
            },
            {
                "metric_type": "metric3",
                "value": 5
            }
        ]
    }
]

jq syntax or transformation would be wonderfull.

piyushsharma · 2020-06-10T07:52:54Z

After closing issue #4260, any tags specified as tag_keys will be extracted from json nested fields. Current functionality will set any int or float value as a field, unless otherwise specified as a tag in tag_keys. To add more customization of metrics, I propose adding field_keys to the JSON parser config. If users specify field_keys on config, only values named in field_keys will be reported as fields in the returned metric. In this manner, there would be full customization of which json values are reported as tags or fields. If field_keys is not specified, the parser will have current functionality.

Would this suit the above use cases?

From a nested json structure, if I want to parse only specific numeric fields, this solution of "field_keys" would have helped a lot. The current implementation only works if all the fields I want are in one specific chunk of the json (by leveraging json_query). Would be great to have a support a list of json_query filters OR have a way to specify which fields should be measured.

danielnelson · 2020-06-10T19:43:45Z

You may be able to do with GJSON's multipaths syntax. If you show an example document I'd have a better idea if this will work for you.

piyushsharma · 2020-06-17T00:16:11Z

@danielnelson : tried using the multipath syntax but cannot get it to work.

{
  "message": "abc",
  "fields": {
    "status": 200,
    "key": 1,
    "json": [
      {
        "duration": 100,
        "code": 1,
        "label": 2,
        "line": 3
      }
    ],
    "duration": 2
  }
}

Say I have a nested JSON like described above with many more numerical fields, and I am only interested in the following: "fields.duration", "fields.status", "fields.json#duration".

danielnelson · 2020-06-17T01:18:42Z

When you say many more numerical fields, do you mean the like:

{
  "message": "abc",
  "fields": {
    "status": 200,
    "key": 1,
    "json": [
      {
        "duration": 100,
        "code": 1,
        "label": 2,
        "line": 3,
        "many": 4,
        "more": 5,
        "numerical": 6,
        "fields": 7
      }
    ],
    "duration": 2
  }
}

Or is it more like this?

{
  "message": "abc",
  "fields": {
    "status": 200,
    "key": 1,
    "json": [
      {
        "duration": 100,
        "code": 1,
        "label": 2,
        "line": 3,
      },
      {
        "many": 4,
        "more": 5,
        "numerical": 6,
        "fields": 7
      }
    ],
    "duration": 2
  }
}

piyushsharma · 2020-06-17T01:42:03Z

Like:

{
  "message": "abc",
  "fields": {
    "status": 200,
    "key": 1,
    "json": [
      {
        "duration": 100,
        "code": 1,
        "label": 2,
        "line": 3,
        "many": 4,
        "more": 5,
        "numerical": 6,
        "fields": 7,
        "nest": {
                "label": 2,
                "line": 3,
                "many": 4,
                "more": 5,
                "numerical": 6,
                "fields": 7
         }
      }
    ],
    "duration": 2
  }
}

For what I need, we can assume "fields.json" is an array of one item. I only care about some properties of "fields" and some properties of "fields.json", I don't need anything from the nested structure "fields.json.nest".

danielnelson · 2020-06-17T01:46:26Z

How about something like this? It's a pretty basic multiselect but we need to provide a custom name for duration to avoid the naming conflict.

json_query = '{fields.duration,fields.status,"json_0_duration":fields.json.0.duration}'

piyushsharma · 2020-06-17T02:02:42Z

Sample Data:

{
  "@message": "Finished Request",
  "@timestamp": "2020-06-09T12:49:18.747Z",
  "@fields": {
    "event": "request",
    "method": "GET",
    "path": "xyz",
    "responseStatus": 200,
    "duration": 10.3845,
    "requestId": "ddd",
    "referer": "abc",
    "level": "info",
    "label": 100
  }
}

Telegraf Config:

tag_keys = ["@message"]
json_query = '{@timestamp,@message,@fields.duration,@fields.responseStatus}'
json_time_key = "@timestamp"
json_time_format = "2006-01-02T15:04:05Z07:00"

I get the following error:

panic: runtime error: index out of range [0] with length 0

goroutine 28 [running]:
github.com/tidwall/gjson.Get(0xc0005000f0, 0xe6, 0xc000052ec1, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/pkg/mod/github.com/tidwall/gjson@v1.3.0/gjson.go:1722 +0xe24
github.com/tidwall/gjson.Get(0xc0005000f0, 0xe6, 0xc000052ec0, 0x3d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/pkg/mod/github.com/tidwall/gjson@v1.3.0/gjson.go:1734 +0x4ec
github.com/tidwall/gjson.getBytes(0xc0005000f0, 0xe6, 0xf0, 0xc000052ec0, 0x3d, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/pkg/mod/github.com/tidwall/gjson@v1.3.0/gjson_ngae.go:18 +0x10d
github.com/tidwall/gjson.GetBytes(...)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/pkg/mod/github.com/tidwall/gjson@v1.3.0/gjson.go:1814
github.com/influxdata/telegraf/plugins/parsers/json.(*Parser).Parse(0xc000244320, 0xc0005000f0, 0xe6, 0xf0, 0xe6, 0xf0, 0xc0001f81e0, 0xe6, 0xc00048c080)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/src/github.com/influxdata/telegraf/plugins/parsers/json/parser.go:189 +0x41b
github.com/influxdata/telegraf/plugins/inputs/tail.parseLine(0x68a8a80, 0xc000244320, 0xc0001f81e0, 0xe6, 0xc0001f8101, 0xe6, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/src/github.com/influxdata/telegraf/plugins/inputs/tail/tail.go:206 +0x280
github.com/influxdata/telegraf/plugins/inputs/tail.(*Tail).receiver(0xc000120200, 0x68a8a80, 0xc000244320, 0xc0002e2bb0)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/src/github.com/influxdata/telegraf/plugins/inputs/tail/tail.go:222 +0x25d
github.com/influxdata/telegraf/plugins/inputs/tail.(*Tail).tailNewFiles.func1(0xc000120200, 0x68a8a80, 0xc000244320, 0xc0002e2bb0)
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/src/github.com/influxdata/telegraf/plugins/inputs/tail/tail.go:177 +0x76
created by github.com/influxdata/telegraf/plugins/inputs/tail.(*Tail).tailNewFiles
	/private/tmp/telegraf-20200429-52913-1ietmxf/telegraf-1.14.2/src/github.com/influxdata/telegraf/plugins/inputs/tail/tail.go:175 +0x530

danielnelson · 2020-06-17T03:56:46Z

Can you open a new bug issue for this and we can address it in 1.14.5. Probably related to the @ symbol since this somewhat special in GJSON.

piyushsharma · 2020-06-17T06:46:37Z

Thanks, will do. It does not work even if I remove the @ prefix from the properties. Tried many combinations -- seeing a lot of JSON time key could not be found at times even though my json_query has the property with timestamp in it.

piyushsharma · 2020-06-17T07:10:35Z

filed #7699

sinistram · 2020-07-16T13:44:15Z

+1 to the issue. It will be great to have an option parse tags like this:

{
    "tags": {
        "tag1" : "value1",
        "tag2" :  "value2"
     },
    "field1" : 2
}

tag_keys = [ "tags_*"]

sjwang90 · 2021-06-10T22:02:21Z

The new JSON_v2 parser (json_v2) has just been released in Telegraf 1.19 rc-1. We built it to address the use cases discussed in this thread so please download and test it out with your JSON files and open any bugs or feature gaps you may encounter.

This was referenced Sep 5, 2016

add timestamp selector to JSON parser #1557

Closed

[Feature Request] JSON Input support a timestamp field #1554

Closed

sparrc mentioned this issue Oct 11, 2016

[Feature request] Ability to parse array in httpjson responce #1882

Closed

sparrc mentioned this issue Nov 3, 2016

Feature request: support graphite templates for plugin httpjson #1986

Closed

sparrc mentioned this issue Nov 22, 2016

Add support to parse nested JSON array #2066

Closed

This was referenced Dec 14, 2016

HTTPJson GET fails if the response returns array of objects #2162

Closed

Renaming field key using the tag value #2173

Closed

mircopolo mentioned this issue Jan 11, 2017

Support the existing JSON output format as an input format for relevant service inputs. #2252

Closed

sparrc mentioned this issue Feb 20, 2017

Configuration flexibility in httpjson input plugin #2434

Closed

danielnelson mentioned this issue May 23, 2017

Ability to retrieve string value via httpjson #2835

Closed

danielnelson mentioned this issue Jul 10, 2017

Parsing Array of json using Telegraf Input.exec pluging #2996

Closed

danielnelson mentioned this issue Jan 3, 2018

httpjson throwing parse errors on valid json #3599

Closed

danielnelson mentioned this issue Feb 20, 2018

inputs.httpjson plugin tag_keys is not working correctly #3805

Closed

danielnelson mentioned this issue Nov 29, 2018

Parse multiple json queries to form output? #5058

Closed

danielnelson mentioned this issue Jan 4, 2019

JSON parsing on different levels with telemetry as nested JSON #5228

Closed

danielnelson mentioned this issue Apr 3, 2019

parser json return only first element of hashes #5671

Closed

danielnelson mentioned this issue Jul 7, 2019

Json array it seems doesn't work for http_listener. #6082

Closed

danielnelson mentioned this issue Sep 24, 2019

JSON Nested Key Tags #6437

Closed

danielnelson added the area/json json and json_v2 parser/serialiser related label Jan 6, 2020

hackery mentioned this issue May 29, 2020

Json Parser wildcard tagkey support #7531

Closed

danielnelson mentioned this issue Jun 17, 2020

Link to GJSON playground in json parser documentation #7698

Merged

3 tasks

piyushsharma mentioned this issue Jun 17, 2020

error using json_query multiselect to reduce telegraf JSON data processing #7699

Closed

sjwang90 mentioned this issue Jul 30, 2020

Json Parser User Number as json_name_key #7212

Closed

sjwang90 mentioned this issue Sep 28, 2020

JSON Parser for Boolean requires "json_string_fields" #7097

Closed

sjwang90 added the influx label Oct 20, 2020

sjwang90 removed the team/influx label May 27, 2021

sspaink mentioned this issue Jun 2, 2021

New JSON Parser #9246

Merged

2 tasks

reimda linked a pull request Jun 10, 2021 that will close this issue

New JSON Parser #9246

Merged

2 tasks

reimda closed this as completed Jun 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support JSON Path-like json parsing #1363

Support JSON Path-like json parsing #1363

sparrc commented Jun 10, 2016

sparrc commented Sep 5, 2016

fromanator commented Sep 16, 2016

kevinjos commented Nov 25, 2016

sparrc commented Nov 26, 2016

kevinjos commented Nov 28, 2016

olsky commented Jan 27, 2017

sparrc commented Jan 27, 2017

olsky commented Jan 30, 2017

asatriani commented Jun 16, 2017

danielnelson commented Jun 16, 2017

freeseacher commented Sep 15, 2017

fabianbaier commented Nov 8, 2017

danielnelson commented Aug 23, 2018

conet commented Apr 27, 2020

piyushsharma commented Jun 10, 2020

danielnelson commented Jun 10, 2020

piyushsharma commented Jun 17, 2020 •

edited

Loading

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020 •

edited

Loading

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

sinistram commented Jul 16, 2020

sjwang90 commented Jun 10, 2021

Support JSON Path-like json parsing #1363

Support JSON Path-like json parsing #1363

Comments

sparrc commented Jun 10, 2016

Feature Request

Proposal:

Current behavior:

Desired behavior:

Use case: [Why is this important (helps with prioritizing requests)]

sparrc commented Sep 5, 2016

fromanator commented Sep 16, 2016

kevinjos commented Nov 25, 2016

sparrc commented Nov 26, 2016

kevinjos commented Nov 28, 2016

olsky commented Jan 27, 2017

sparrc commented Jan 27, 2017

olsky commented Jan 30, 2017

asatriani commented Jun 16, 2017

danielnelson commented Jun 16, 2017

freeseacher commented Sep 15, 2017

fabianbaier commented Nov 8, 2017

danielnelson commented Aug 23, 2018

conet commented Apr 27, 2020

piyushsharma commented Jun 10, 2020

danielnelson commented Jun 10, 2020

piyushsharma commented Jun 17, 2020 • edited Loading

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020 • edited Loading

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

danielnelson commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

piyushsharma commented Jun 17, 2020

sinistram commented Jul 16, 2020

sjwang90 commented Jun 10, 2021

piyushsharma commented Jun 17, 2020 •

edited

Loading

piyushsharma commented Jun 17, 2020 •

edited

Loading