Need metric to return job failed reason

[tatchiuleung]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:
Missing a metric like kube_job_status_failed_reason
What you expected to happen:
Add kube_job_status_failed_reason to display the failed reason, such as Evicted
How to reproduce it (as minimally and precisely as possible):
Create a cronjob but limit the resource

Anything else we need to know?:
K8S returns the following in the Evicted job

status:
  message: 'The node was low on resource: ephemeral-storage. Container backups was
    using 3405160Ki, which exceeds its request of 0. '
  phase: Failed
  reason: Evicted
  startTime: "2019-10-11T00:00:02Z"

We would like to filter out the Evicted jobs, even if it's failed.
Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T23:49:07Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.24", GitCommit:"2ce02ef1754a457ba464ab87dba9090d90cf0468", GitTreeState:"clean", BuildDate:"2019-08-12T22:05:28Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

• Kube-state-metrics image version

quay.io/coreos/kube-state-metrics:v1.7.2

[AdityaMisra]

@brancz @lilic I want to work on this issue.

[brancz]

If there is a bound number of reasons I think this would be ok to have.

[lilic]

@AdityaMisra do you mind looking up the number of reasons first for failed jobs, so we can make sure does not have a high unbound cardinality. Otherwise go ahead. :)

/assign @AdityaMisra

[juliantaylor]

This is also needed for normal pods, not only jobs.
maybe via a label with kube_pod_status_phase

[lilic]

@juliantaylor Feel free to open a separate issue for the pod, as its a different resource makes sense to discuss that separately.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[AdityaMisra]

/remove-lifecycle stale

[RajatVaryani]

@AdityaMisra Are you still working on this?

[AdityaMisra]

@RajatVaryani I'm working on it. Will be generating PR soon.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lilic]

/remove-lifecycle stale

There is a PR for this.