Metrics exposure for volcano scheduling start time and end time for each job

[dontan001]

What would you like to be added:

Currently we find there are no metrics to indicate the scheduling start time and end time for each job, and only scheduling duration is available, which is e2e_job_scheduling_duration.

We would like these two timestamps could be exposed in Volcano, there after could be visualized via Grafana annotation in dashboard, and it is a great help when we are doing the Spark+Volcano performance analysis

Why is this needed:

It is for spark+volcano performance analysis

[dontan001]

Please help take a look @william-wang @hwdef @Thor-wl , I can help raise a PR for this if there is chance

[hwdef]

I think this is useful.
And we already added start and end time in job condition.

[william-wang]

@dontan001 That's reasonable to have start time and end time of the job. Go ahead to raise a PR :)

[hwdef]

ref: #1764

[dontan001]

Thank you @hwdef for sharing the PR
I did a quick check and seems the conditions are updated for CR vcjobs in the above PR, while in our case we are using native spark-submit directly

Let me prepare a PR then please comment further, thanks very much for the update @hwdef @william-wang

[Thor-wl]

@dontan001 That's reasonable to have start time and end time of the job. Go ahead to raise a PR :)

+1. It makes sense.

[dontan001]

• As for the end time of job scheduling last time, please help review PR To record the start and end time of job scheduling #2318
• As for the start time of job scheduling, right now it is the creation timestamp of PodGroup, and we could leverage the custom resource metrics from kube-state-metrics
  • The custom resource metrics capability is recently introduced in PR feat: enable customresource metrics by configuration kubernetes/kube-state-metrics#1710
  • While one minor enhancement by me to support timestamp To support read the timestamp fields of custom resources kubernetes/kube-state-metrics#1766

Please review and let me know if any suggestion, thanks a lot

[dontan001]

Considering the kube-state-metrics may not be deployed in user environment, it would be brilliant to expose the start time in Volcano as well.

I have refined the PR to include both, please review again when get chance @hwdef @william-wang @Thor-wl

[Thor-wl]

/close

[volcano-sh-bot]

@Thor-wl: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.