promtail: timestamp in --dry-run mode is missing timezone information

[kbudde]

Describe the bug
Given a nginx log with date & time with missing timezone information. I would like to interpret the time as local timezone.
I tried timestamp stage with location field but it looks like that this field does nothing.

Maybe I do something terrible wrong but how can I read a nginx log without timezone information and interpret it as local timezone?

To Reproduce

1. Login to a machine with timezone != UTC (for example Europe/Berlin)
2. Start promtail with dry-run and config-file below (binary, not container)
3. check output. Timestamp field is not affected by location parameter

Expected behavior
The timestamp should be affected by location field in some way, or not?
Even for timestamps with timezone information does not affect timestamp at all.
Maybe it's just a dry-run bug not showing the timezone?

Environment:

• Ubuntu 20.04
• promtail, version 2.2.1 (branch: HEAD, revision: babea82)

Screenshots, Promtail config, or terminal output

file without timezone: notz.log

2021/04/30:00:57:19 [error]

promtail.yaml

scrape_configs:
  - job_name: test
    static_configs:
    - labels:
        __path__: "notz.log"
    pipeline_stages:
    - match:
        selector: '{filename=~".*notz.log"}'
        stages:
        - regex:
            expression: '^(?P<time_local>[^ ]+) \[(?P<level>\w+)\]'
        - timestamp:
            source: time_local
            location: "Europe/Berlin"
            format: "2006/01/02:15:04:05"
        - labels:
            level:
            time_local:

location: "America/New_York"

2021-04-30T00:57:19 {filename="/home/kbudde/example/notz.log", level="error", time_local="2021/04/30:00:57:19"} 2021/04/30:00:57:19 [error]

location: "Europe/Berlin"

2021-04-30T00:57:19 {filename="/home/kbudde/example/notz.log", level="error", time_local="2021/04/30:00:57:19"} 2021/04/30:00:57:19 [error]

location: "Local"

2021-04-30T00:57:19 {filename="/home/kbudde/example/notz.log", level="error", time_local="2021/04/30:00:57:19"} 2021/04/30:00:57:19 [error]

location: "Foo"

... invalid timestamp stage config: invalid location specified: unknown time zone Foo

removed timestamp stage
2021-05-04T07:19:05 {filename="/home/kbudde/example/notz.log", level="error", time_local="2021/04/30:00:57:19"} 2021/04/30:00:57:19 [error]

file with timezone tz.log

2021/04/30:00:57:19+0300 [error]

promtail.yaml

scrape_configs:
  - job_name: test
    static_configs:
    - labels:
        __path__: "tz.log"
    pipeline_stages:
    - match:
        selector: '{filename=~".*tz.log"}'
        stages:
        - regex:
            expression: '^(?P<time_local>[^ ]+) \[(?P<level>\w+)\]'
        - timestamp:
            source: time_local
            location: "America/New_York"
            format: "2006/01/02:15:04:05-0700"
        - labels:
            level:
            time_local:

location: "America/New_York"

2021-04-30T00:57:19 {filename="tz.log", level="error", time_local="2021/04/30:00:57:19+0300"} 2021/04/30:00:57:19+0300 [error]

location: "Europe/Berlin"

2021-04-30T00:57:19 {filename="tz.log", level="error", time_local="2021/04/30:00:57:19+0300"} 2021/04/30:00:57:19+0300 [error]

location: "Local"

2021-04-30T00:57:19 {filename="tz.log", level="error", time_local="2021/04/30:00:57:19+0300"} 2021/04/30:00:57:19+0300 [error]

location: "Foo"

.... invalid timestamp stage config: invalid location specified: unknown time zone Foo"

removed timestamp stage

2021-05-04T07:03:08 {filename="tz.log", level="error", time_local="2021/04/30:00:57:19+0300"} 2021/04/30:00:57:19+0300 [error]

[cyriltovena]

It's possible that promtail always output in the host time. This is where the code is https://github.com/grafana/loki/blob/main/clients/pkg/promtail/client/logger.go#L77

It's also possible that the timestamp is not parsed correctly have you tried debug logs ?

[kbudde]

You are right, it is just the logger.

I would propose to change the logger to either use local time or include timezone in output

Promtail logger (current)
Berlin:   2021-05-01T10:11:12
New York: 2021-05-01T10:11:12

using local timezone for output
Berlin:   2021-05-01T10:11:12
New York: 2021-05-01T16:11:12

time with TZ
Berlin:   2021-05-01T10:11:12+0200
New York: 2021-05-01T10:11:12-0400

See https://play.golang.org/p/1S8iNtX2RZ3

I can create PR, if it's clear what's needed.