Metrics for job-controller

[alculquicondor]

What would you like to be added:

Metrics for the job controller operations, including:

• latency of a job sync job_sync_duration_seconds
• sync count job_sync_total (with a result label for success/error)
• TBD job finished count job_finished_total (with a result label for success/error)

All metrics should include labels that allow to narrow delays or errors to features or code paths:

• completion_mode: NonIndexed, Indexed.
• TBD pattern: single, workQueue, fixedCompletion
• TBD type of event that triggered sync, and whether it was immediate.

Why is this needed:

This is essential to diagnose common usage patterns or problems in the controller.

Note:

A KEP will be written for this feature, but we want to gather feedback on other useful metrics or labels. The lists are currently open to additions or removals.

/sig apps
/area workload-api/job

[alculquicondor]

cc @soltysh @ahg-g @erictune

[alculquicondor]

cc @yangkev

[prydonius]

/triage accepted

[alculquicondor]

I've updated the job description to include the metric names and labels I propose.

The ones that are preceded with TBD I'm not convinced we need.

I'm wondering if a KEP is an overkill.

[ahg-g]

I feel the metrics are straightforward enough. Perhaps we add each metrics in a separate PR in which we discuss the exact semantics.

[alculquicondor]

cc @AliceZhang2016

[AliceZhang2016]

/assign

[alculquicondor]

I'm documenting this in the KEP for beta graduation for indexed job kubernetes/enhancements#2616