Docker Logging Driver: Unable to use labels with dot in name

[hamiltont]

Describe the bug

I do not think it is possible to use the labels log-opt with any labels that have a dot in the name. Unfortunately this limitation happens before the relabeling, so it is impossible to workaround. Also unfortunately, Docker's label guidance recommends the use of reverse DNS notation, which has plenty of dots, making it quite probable users are going to run into this bug.

To Reproduce

1. Run any container as follows:

• enabe the loki driver
• Pass a docker label with a dot (e.g. git.sha)
• Request loki remember that label

$ docker run --log-driver="loki" -l git.sha=15f123d2 --log-opt "labels=git.sha" --log-opt loki-url=<snip> nginx
docker: Error response from daemon: failed to initialize logging driver: error creating logger: error creating loki logger: loki: invalid label name from attribute: git.sha.

2. Note that labelmap is unable to fix the issue:

$ read -d '' relabel << EOF
- regex: (git.sha)
  replacement: git_sha
  action: labelmap
EOF
$ docker run --log-driver="loki" -l git.sha=15f123d2 --log-opt "labels=git.sha" --log-opt loki-relabel-config="$relabel" --log-opt loki-url=<snip> nginx
docker: Error response from daemon: failed to initialize logging driver: error creating logger: error creating loki logger: loki: invalid label name from attribute: git.sha.

3. Note that label replace is unable to fix the issue:

$ read -d '' relabel << EOF
- source_labels: [git.sha]
  target_label: git_sha
EOF
$ docker run --log-driver="loki" -l git.sha=15f123d2 --log-opt "labels=git.sha" --log-opt loki-relabel-config="$relabel" --log-opt loki-url=<snip> nginx
docker: Error response from daemon: failed to initialize logging driver: error creating logger: error creating loki logger: loki: invalid label name from attribute: git.sha.

Expected behavior

The problem appears to be that the label names (passed via log-opt key 'labels') are being checked for prometheus validity before being remapped. This makes little sense at this early in the flow - these labels are coming in from docker, and may very well be invalid according to prometheus. (In fact, docker recommends reverse notation, so it's quite likely they will be invalid!)

Instead, labels should be checked for prometheus validity only after all the remapping is done

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[hamiltont]

Ok, small rant here -- Perhaps set the stale bot to be less restrictive if there is not enough bandwidth for a response within 30 days

It's frustrating to write up a solid issue report with the intention to help the tool be better on an edge case that will likely bite multiple folks and was painful to debug, only to have the report auto-closed because no one had time to reply

For clarity - an ultra slow reply is fine, this is not a paid support forum and I understand that completely. I'm only asking you consider not auto-closing issues before someone on the team has had a chance to respond. Perhaps set it to what you actually need - 3 months, 6 months, 1 year, etc, not an arbitrary deadline. (Side note - I really wish stale bot had a feature to inform maintainers of their setting e.g. "12% of issues closed as stale had no maintainer response when closed")

[cyriltovena]

I think this has been fixed recently ? By #2459 if not let me know, basically . is replace by _

[hamiltont]

Awesome!