Timestamp not parsing seconds correctly

[tgirotto]

Coming from https://community.influxdata.com/t/timestamp-not-being-recorded-json-v2/22818/6

Relevent telegraf.conf

[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will cache metric_buffer_limit metrics for each output, and will
  ## flush this buffer on a successful write.
  metric_buffer_limit = 10000
  ## Flush the buffer whenever full, regardless of flush_interval.
  flush_buffer_when_full = true

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. You shouldn't set this below
  ## interval. Maximum flush_interval will be flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## Run telegraf in debug mode
  debug = true
  ## Run telegraf in quiet mode
  quiet = false
  ## Override default hostname, if empty use os.Hostname()
  hostname = ""

# Read formatted metrics from one or more HTTP endpoints
[[inputs.http_listener_v2]]
  #name_override = "Steamaco"

  ## Address and port to host HTTP listener on
  service_address = ":8080"

  ## Path to listen to.
  path = "/telegraf1"

  methods = ["POST", "PUT"]

  basic_username = "test"
  basic_password = "test"

  data_format = "json_v2"

  [[inputs.http_listener_v2.json_v2]]
    measurement_name = "steamaco_v2"
    [[inputs.http_listener_v2.json_v2.object]]
      path = "data"
      timestamp_key = "created_at"
      timestamp_format = "2006-01-02T15:04:05Z"
      included_keys = [
        "kwh",
        "created_at"
      ]

[[outputs.influxdb_v2]]
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
urls = ["${INFLUX_HOST}"]
token = "${INFLUX_TOKEN}"
bucket = "***"

System info

Linux 4.15.0-163-generic Ubuntu; Telegraf 1.20.4 (git: HEAD 34ad5aa)

Docker

No response

Steps to reproduce

1. Send with postman the following

{
    "data" : [
        {
            "created_at" : "2021-12-07T14:56:02Z",
            "meter_id" : 14,
            "meter_external_reference": "1002848",
            "meter_external_id" : 12,
            "kwh" : 497,
            "kwh_delta" : 13,
            "core_external_reference" : "1231231",
            "core_external_id" : 0,
            "grid_id" : 52,
            "grid_name" : "Ogheye",
            "meter_type_id" : 2,
            "meter_type_name" : "Essential Service"
        }
    ]
}

### Expected behavior

The time of the datapoint being recorded should be 2021-12-07T14:56:02Z

### Actual behavior

The time of the datpoint is 2021-12-07T14:56:10.000Z

### Additional info

_No response_

[powersj]

Can you please provide the actual output that shows it off by 2 seconds?

I am not seeing the same thing:

> steamaco_v2 kwh=497 1638888962000000000

Then convert the timestamp to a real date:

❯ TZ=UTC0 printf '%(%c)T\n' 1638888962
Tue 07 Dec 2021 02:56:02 PM UTC

edit: if you could run this with --test to get just a single output in Line Protocol format that would be helpful to see what is getting parsed by the input. Then you can determine if your output is messing with the date.

[tgirotto]

@powersj I am new to influx db, so I am not sure what commands I need to use to get the results in the form you expect them, but here are some screenshots (we are using Influx Cloud):

Also, others seem to have had a similar problem, (#4928 (comment)) maybe related?

[powersj]

That top screenshot's timestamp format is not what you showed in your original comment.

In the post on the community forum and the initial post you showed:

"created_at" : "2021-12-07T14:56:02Z",

But that 2nd screenshot shows:

"created_at" : "2021-12-07T14:56:02.123Z",

So yes, you will need to set the timestamp format correctly.

[tgirotto]

@powersj my bad, I was testing all possibilities. The result is the same, independently of the input.

The result is the same, in particular the UI in Influx cloud seems to always set seconds to 10 and ms to 000 (you can see that also all the previous times have sec == 10 and ms == 000)

In other words, the problem occurs both with 2021-12-07T14:56:02Z and 2021-12-07T14:56:02.123Z formats.

[powersj]

ok so something is happening between when you read the data and when you see it in InfluxDB. Can you remove InfluxDB from the equation, and run telegraf independently by running with the [[outputs.file]] This will output to stdout and I would like to see the output.

[tgirotto]

@powersj I think it's a bug in the influx cloud UI (I mean in the Influx explorer). In fact, I have read the data via influx cli, and the timestamps seem to be picked up correctly, with seconds and ms showing just fine.

[powersj]

Awesome, thanks for confirming! If you could file a bug with InfluxDB UI here I would appreciate it.

I'm going to go ahead and close this.