Nomad should pass job, alloc and task ids as docker labels

[samart]

It woudl be very useful if nomad passed jobId, alloc Id, and taskid as docker labels on container.create().

This will allow us to correlate - via monitoring tools, (Sysdig/Prometheus/etc) and security tooling - running containers with nomad Runtime metadata. Its very common for these tools to pull in docker labels as means for identification and other categorization purposes.

@rcgenova

[jippi]

You can just pass them in

job "demo" {
  group "demo" {
    task "demo" {
      config {
        labels {
          key = value
        }
      }
    }
  }  
}

[samart]

Where would the values come from at runtime? Are there variables i can use to set them? (Let me look..)

Edit - i looked and I only see ENV vars that have these details (aside from jobId), which i’m not sure we can pass in as labels in the job file. Or can we?

Note - mesos does this automatically. (It sorta feels like it shoudl be an automatic thing)

[jippi]

you can reference any environment key using interpolation

[drarnold]

Hey @jippi ,

I was going to be using Docker Labels for our containers to autodiscover haproxy and would be putting it into the Nomad Definition. I am curious how to handle each of the keys:
https://docs.datadoghq.com/integrations/haproxy/#containerized
(Especially the Instance Config parameter - <INSTANCE_CONFIG>)

What would the syntax look like because it seems like single quotes are not recognized by the Nomad definition?

Thanks

[kshpytsya]

Absence of this feature prevents me from effectively using cAdvisor (with Prometheus), as we have same task names in different nomad jobs. I am even considering implementing an augmenting proxy that would rewrite Prometheus metrics obtained from cAdvisor adding Nomad job and group name based on the allocation id, which is included as a suffix in the name label (querying local Nomad agent to get allocation info). But of course, it would be much better to just have it properly implemented in Nomad itself. Modifying job descriptions as suggested in #4781 (comment) is highly impractical in our case.

[m1keil]

@kshpytsya you can expose these via cadvisor's --docker_env_metadata_whitelist

[sofixa]

Any news on this? It would be a very useful feature for log shipping agents ( like promtail) which can only read Docker labels.

[sofixa]

FYI i've created a PR adding those labels #9885

[tgross]

Forgot to close this one out when #9885 was merged. Marked for 1.1.0

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.