[feature] - change the filename of logfiles generated for tasks in Nomad

[henrikjohansen]

Finding a decent solution for centralized, multi-tenant logging with Nomad is ... difficult ... at the moment, especially when the desire is to keep nomad logs working alongside a centralized log shipping solution.

Our current approach is to inject a Filebeat based sidecar per task that adds several additional fields that are absolutely necessary in multi-tenant setups :

filebeat.inputs:
- paths:
    - /alloc/logs/*.stdout.0
    - /alloc/logs/*.stderr.0
  encoding: utf-8
  fields:
    nomad_job: ${NOMAD_JOB_NAME}
    nomad_group: ${NOMAD_GROUP_NAME}
    nomad_datacenter: ${NOMAD_DC}
    nomad_region: ${NOMAD_REGION}
    ...

... but with several hundred sidecars running this is not really feasible any longer :(

But if the names of the actual log files generated by Nomad could be changed to include more than just the task name the above problem could be solved by running a single log shipper per host.

Inspired by the metrics format that Nomad uses an example could be :

<Namespace>.<Job>.<TaskGroup>.<Task>.stdout.0

Based on the path and filename we could then route the correct logs to the correct teams and enable much easier correlation between the logs and metrics that Nomad offers.

[henrikjohansen]

Perhaps it would even be feasible to include this as a configurable option in the log stanza and use the standard variables for Interpolation?

logs {
        max_files       = 10
        max_file_size   = 10
        filename_prefix = "${NOMAD_NAMESPACE.NOMAD_JOB_NAME.NOMAD_GROUP_NAME.NOMAD_TASK_NAME"}
}

The namespace would be available now that #6192 has been merged.