Promtail drops static_configs target although file __path__ is differnt

[Mario-Hofstaetter]

Describe the bug

One application may have several different logfiles (in different locations).
Using two entries in static_configs of a job with DIFFERENT __path__ values, but otherwise the same set of labels, causes promtail to drop one of the configurations (Dropped: ignoring target, already exists) and not tailing the logfile.

To Reproduce

1. Run promtail.exe Version 2.2.1 on Windows 10 bare metal
2. Using this in a job in your promtail-config.yaml:

    static_configs:
      - targets:
          - "localhost"
        labels:
          __path__: "D:/MyApp/Log/MainFile.log.json"
          app: "MyApp"
      - targets:
          - "localhost"
        labels:
          __path__: "D:/MyApp/Log/XXXXXX.log.json"
          app: "MyApp"

(job label removed for brevity)

3. Go to http://127.0.0.1:9080/service-discovery , I am getting this:

Dropped: ignoring target, already exists

There is only one target active in http://127.0.0.1:9080/targets

Expected behavior

IMHO both targets should be used as this is what a user would expect?
The __path__ label is after all stored in loki as filename, so I guess promtail can use the path label for target uniqueness?

Changing one of the "localhost" values of targets: is not helping, because __address__ is also ignored (See also #1831)

To fix the issue, one has to add an actual dummy label (❓):

    static_configs:
      - targets:
          - "localhost"
        labels:
          __path__: "D:/MyApp/Log/MainFile.log.json"
          app: "MyApp"
      - targets:
          - "localhost"
        labels:
          __path__: "D:/MyApp/Log/XXXXXX.log.json"
          foobar: baz    # <<< :-(
          app: "MyApp"

Which is unfortunate because labels decrease loki performance (chunks, best practices docs, ...)

Am I doing something wrong?

Issue #1754 sounding similar, but OP there already used actual labels with different labels.

Environment:

• Infrastructure: Windows 10 20H2, promtail.exe Version 1.2.1on bare metal.

[3Xpl0it3r]

all xx label will be removed
the label value of path will passed to tail, then the __path__label will be removed

			for k := range labels {
				if strings.HasPrefix(string(k), "__") {
					delete(labels, k)
				}
			}
....

			level.Info(s.log).Log("msg", "Adding target", "key", key)
			t, err := s.newTarget(string(path), labels, discoveredLabels)

as the value of path passed to go-tail library, I am not sure the tail is support regex, you can look into this, if not support you may need add an extra label for differ.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[Mario-Hofstaetter]

I believe this behavior to be unintuitive for users, the need to add "junk" labels just so multiple log files are not ignored by promtail.

Is this something loki devs would reconsider? Can't PR anything myself unfortunately.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[Mario-Hofstaetter]

This issue seems to be resolved by #4599 ✅

I just compiled main, commit 07ea1ed99c4ef10cbafc4212f589500afa85c59d on windows and ran a quick test with one job containing (compare with OP above):

    static_configs:
      - labels:
          __path__: "C:/temp/log1.log.json"
          app: "myapp"
      - labels:
          __path__: "C:/temp/log2.log.json"
          app: "myapp"

resulting in this behavior:

I have not yet tested if logs from both files are correctly tailed and pushed to loki but it looks promising.

[jeyrce]

I met the same problem and must use the file_sd_config rather than static_sd_config, so how to build a valid file_sd_config to solve this problem?