JSON parser: adding of string fields not explicitly specified

[omgold]

Feature Request

The JSON parser should be more flexible concerning what to do with passed bool or string fields.

Proposal:

I see three options:

• allow globs or regexps in json_string_fields
• add a flag making all fields to be included.
• let the JSON input itself override the configuration parameters tag_keys, json_string_fields and json_key_name

Current behavior:

All string or bool fields not explicitly specified in json_string_fields are discarded.

Desired behavior:

A method to include such fields in bulk

Use case:

• Many fields, making it tedious to list all in the config.
• There are monitored systems where the fields are not known up front, like importing data from another database.
• API changes of input collectors require to update telegraf config. Why are fields discarded in the first place? Shouldn't the input source know best what to submit?

[danielnelson]

We should add glob support to string_fields.

let the JSON input itself override the configuration parameters tag_keys, json_string_fields and json_key_name

This sounds interesting, can you explain further how this could work?

[omgold]

What I was thinking about here, was adding a special key to the JSON, containing a map with the these parameters to override. That key should probably be something not valid as a field name, like beginning with a '.'. Let's call it '.telegraf_params' in the example below (I don't really care about the name).

So to override these fields in a JSON sent, it could look like this:

[
  {
   ".telegraf_params": {
    "key_name": "measurement_name",
    "string_fields": ["my_string_field1","my_string_field2"],
    "tag_keys": ["my_tag1","my_tag2"]
   },
    "measurement_name": "my_measurement",
    "my_tag1": "a",
    "my_tag2": "b",
    "my_string_field1": "x",
    "my_string_field2": "y",
    "my_number_field1": 1,
    "my_number_field2": 2
  }
]

[dgalichet]

Hi, I'm adding my 2 cents on this issue.

Actually, json output format output data in a document that looks like:

{
  "name": "...",
  "timestamp": ...,
  "tags": [ ... ],
  "fields": [ ... ]
}

I could use this output format to send metrics or logs through kafka for example and collect them on the other hand with an json input format that could be configured like:

     data_format = "json"
     json_query = ""
     tag_keys = ["tags.*"]
     json_string_fields = ["fields.*"]
     json_name_key = "name"
     json_time_key = "timestamp"
     json_time_format = "unix_ms"

This would provide a generic way to collect metrics/logs from different sources. The filtering would then be done in chronograf (tags_my_tag = expected or even better my_tag = expected)

EDIT: there's a question regarding json_string_fields because all fields in fields should not be processed as string.

EDIT2: found #2252 that is exactly what I would expect

[taylor840326]

I replace "v==k" by regexp.Match

187         //remove any additional string/bool values from fields
188         for k := range fields {
189                 //check if field is in StringFields
190                 sField := false
191                 for _, v := range p.StringFields {
192                         if ok,_ := regexp.Match(v,[]byte(k));ok{
193                                 sField = true
194                         }
195                 }
196                 if sField {
197                         continue
198                 }

is only a temporary measure,though.