Debuggin and tracing write Loki setup and connection

[aslom]

Is there a flag or some other way to see DEBUG/ or TRACE log with connection to Loki and what is sent there?

Right now I have simple pipeline with

  - name: write_loki
    write:
      type: loki
      loki:
        url: http://localhost:3100
        staticLabels:
          job: flowlogs-pipeline

and it runs without errors (and no output) even if there is nothing on localhost:3100 ( checked with curl -v localhost:3100/ready fails)

@ronensc
@eranra

[ronensc]

When setting the log level to trace, FLP emits a log message for every flowlog passing through the WriteLoki stage.

flowlogs-pipeline/pkg/pipeline/write/write_loki.go

Line 209 in 25d78d7

log.Tracef("writing entry: %#v", entry)

FLP uses loki-client-go to send the flowlog to Loki. loki-client-go is async. It waits for a full batch or for some timeout to expire before actually sending the flowlogs to loki. I couldn't find any trace log messages emitted when the actual sending is happening. A PR against loki-client-go should be submitted to add it.

You probably don't see anything on localhost:3100 because FLP exits before loki-client-go has sent the flowlogs to Loki.
As a quick and dirty hack, You may increase the waiting time to 10 seconds.

flowlogs-pipeline/cmd/flowlogs-pipeline/main.go

Line 228 in 665de17

time.Sleep(time.Second)

A more proper solution would be to make FLP wait for all in-processing stuff to be finished before exiting.