Publishing metrics for job summary

[diptanu]

Fixes #3465

Publishing the job metrics in the leader node since they need to be published only from one of the servers.

Going to add docs, and related things once the implementation looks good to folks.

@dadgar @chelseakomlo

[chelseakomlo]

Thanks for the quick PR! Looks good, left a couple comments.

[chelseakomlo]

As of Nomad 0.7, metrics are published in a tagged format. See:

• https://www.nomadproject.io/api/metrics.html
• https://www.nomadproject.io/docs/agent/configuration/telemetry.html#backwards_compatible_metrics

[diptanu]

Yeah I saw that, but I wasn't sure what the tags are going to be in this case.

[diptanu]

Any suggestions?

[dadgar]

I think the job should be a tag.

[chelseakomlo]

Is it necessary to reset the timer in each case statement?

[diptanu]

@chelseakomlo Since we are resetting the timer at the end of the loop, we need to reset it before doing a continue in the error branches. If we don't reset the timer, we won't publish anymore metrics.

[chelseakomlo]

If the timing doesn't need to be exact, could you reset the timer at the beginning of the loop for all cases? Otherwise then maybe add comments to explain why timers are set only for some branches.

[diptanu]

@chelseakomlo I was just thinking since the time taken by the work is not O(1) but O(n) we might cascade a few ticks if the iteration takes longer for some reason like GC pauses or underlying OS issues. I can add comments to explain this.

[dadgar]

I think it is worth the simpler code to just reset once at the top of the case

[diptanu]

@chelseakomlo I added support for tagged metrics, currently making job id and taskgroup name as tags. Please let me know if that addresses your comment regarding that?

[diptanu]

From the metrics endpoint, with this patch -

{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.complete",
Value: 0
},
{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.failed",
Value: 0
},
{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.lost",
Value: 0
},
{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.queued",
Value: 0
},
{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.running",
Value: 1
},
{
Labels: {
job: "example",
task_group: "cache"
},
Name: "nomad.nomad.job_summary.starting",
Value: 0
},

[jippi]

i like the irony of a new telemetry feature have a BackwardsCompatibleMetrics check :P

[preetapan]

@diptanu can you update the Changelog for this

[jesusvazquez]

Hi, I can't find this metrics in here https://www.nomadproject.io/docs/agent/telemetry.html. Can we update the documentation accordingly? This metrics are definitely useful.

[schmichael]

@jesusvazquez Good call. See #3467

[jesusvazquez]

Thanks @schmichael I was about to implement my own solution for this and to my surprise it was already there!.

Keep up the good work. 💪

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.