Promtail stops shipping logs when an error is encountered in journald

[highsineburgh]

Describe the bug
We have promtail configured to forward logs from journald running as a SystemD unit. When promtail encounters an error streaming from journald the service stops forwarding logs until the unit is restarted.

To Reproduce

1. Run promtail as a SystemD unit streaming and forwarding journald
2. Encounter error streaming journald (we are working on trying to figure out what might be causing this error here, bad message seems to indicate the stream encountered a non text/string character)
3. New log entries will not appear in Loki until the promtail SystemD unit is restarted (we typically detect this via loki-canary).

Expected behavior
Promtail continues to stream and forward logs or exits so it can be restarted by SystemD.

Environment:

• Infrastructure: SystemD unit on ec2 instances

Screenshots, Promtail config, or terminal output
Log from promtail:

level=error ts=2020-10-26T14:32:39.993094479Z caller=journaltarget.go:174 msg="received error during sdjournal follow" err="failed to read message field: bad message"

Promtail config:

server:
      http_listen_port: 9080
      grpc_listen_port: 0
    scrape_configs:
    - job_name: journal
      journal:
        max_age: 12h
        labels:
          jobs: systemd-journal
      relabel_configs:
        - source_labels: ['__journal__systemd_unit']
          target_label: 'unit'
        - source_labels: ['__journal__systemd_unit']
          regex: 'loki-canary.service'
          replacement: 'stdout'
          target_label: 'stream'