Add reason tag to kubernetes_state.job.failed

[keisku]

What does this PR do?

Add reason:backofflimitexceeded,deadlineexceeded to the kubernetes_state.job.failed.

Motivation

Currently, we cannot monitor why a job failed.

Additional Notes

• Job termination and cleanup
• BackoffLimitExceeded and DeadlineExceeded are GA. Other reasons are still beta or alpha.
  • https://github.com/kubernetes/kubernetes/blob/f6fde3ba7fb0755eb0d982b6ed2617258f1f4f8e/staging/src/k8s.io/api/batch/v1/types.go#L638-L642
• kube_job_status_failed in KSM(code) is the only metric that contains reason in labels. kube_job_failed doesn't have reason label, see kube_job_failed should have reason label kubernetes/kube-state-metrics#2382
• kube_pod_container_status_terminated_reason in KSM (code) does not have reason:backofflimitexceeded,deadlineexceeded even it can have reason:ContainerStatusUnknown, etc.

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Collect KSM metrics from this job.

apiVersion: batch/v1
kind: Job
metadata:
  name: exit-fail
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["exit", "1"]
      restartPolicy: OnFailure

[pr-commenter]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=33332919 --os-family=ubuntu

[hestonhoffman]

Docs 👍

[keisku]

Unless we update kube-state-metrics/v2 to at least v2.9.0 that contains the bug fix, kubernetes/kube-state-metrics#2046, we cannot add reason:deadlineexceeded to kubernetes_state.job.failed.

kube-state-metrics/v2 update should be in a different PR as the context below.

Why do currently we use k8s.io/kube-state-metrics/v2 v2.8.2?
This is because we have to align with interface change before bumping up the kube-state-metrics/v2.

Current implementations in
https://github.com/DataDog/datadog-agent/tree/2feb83da045935df7986e56504bd297922a32ebb/pkg/collector/corechecks/cluster/ksm/customresources don't follow type RegistryFactory interface updated by kubernetes/kube-state-metrics#1851.

[keisku]

This logic is now in validateJob().

[keisku]

/merge

[dd-devflow]

🚂 MergeQueue

Pull request added to the queue.

This build is going to start soon! (estimated merge in less than 28m)

Use /merge -c to cancel this operation!