Delete jobs that have failed for at least the last 60 days in a row

[fejta]

http://velodrome.k8s.io/dashboard/db/bigquery-metrics

Delete any job which:

• Ran this week
• Failed every run for the last 60 days.

Example jobs:

 "ci-kubernetes-e2e-gci-gce-examples": {
    "failing_days": 172
  },
  "ci-kubernetes-e2e-gce-examples": {
    "failing_days": 172
  },
  "ci-kubernetes-e2e-gce-latest-upgrade-cluster": {
    "failing_days": 165
  },
  "ci-kubernetes-e2e-gci-gke-pre-release": {
    "failing_days": 163
  },
  "ci-kubernetes-e2e-gke-pre-release": {
    "failing_days": 159
  },
  "ci-kubernetes-e2e-kops-aws-slow": {
    "failing_days": 146
  },
  "ci-kubernetes-e2e-kops-aws-serial": {
    "failing_days": 146
  },
  "ci-kubernetes-e2e-gke-stackdriver": {
    "failing_days": 94
  },
  "ci-kubernetes-e2e-ubuntu-gke-serial": {
    "failing_days": 80
  },
  "ci-kubernetes-e2e-ubuntu-gke-1-6-serial": {
    "failing_days": 80
  },
  "ci-kubernetes-e2e-ubuntu-gke-1-6-flaky": {
    "failing_days": 78
  },
  "ci-kubernetes-node-docker-benchmark": {
    "failing_days": 74
  },
  "ci-kubernetes-node-docker": {
    "failing_days": 74
  },
  "ci-kubernetes-node-kubelet-flaky": {
    "failing_days": 68
  },
  "ci-kubernetes-pull-gce-federation-deploy-canary": {
    "failing_days": 66
  },
  "ci-kubernetes-e2e-gce-gci-qa-serial-master": {
    "failing_days": 62
  },
  "pr:pull-kubernetes-e2e-kubeadm-gce": {
    "failing_days": 62
  },
  "ci-kubernetes-soak-gke-gci-test": {
    "failing_days": 61
  },
  "ci-kubernetes-e2e-gce-etcd3-release-1-5": {
    "failing_days": 60
  },
  "ci-kubernetes-soak-gke-test": {
    "failing_days": 60
  },
  "ci-kubernetes-e2e-kops-aws-canary": {
    "failing_days": 60
  },

Previous cleanup work: #2453

Current status: http://storage.googleapis.com/k8s-metrics/failures-latest.json

[rmmh]

Some tests are ONLY run on flaky suites. Are we going to just stop running them?

I guess that's a general problem. We should graph the test matrix-- show all the tests we have defined and which jobs they have run on in the last week. That way we can find tests that never run and flag them for revival or deletion.

[fejta]

There are a bunch of flaky suites.

[fejta]

I want to do the following:

• Run the minimum set of testing necessary to give us confidence in our releases
• Run the maximum set of testing we have the ability to maintain.

Right now it seems like we are running more tests than we have the ability to maintain. Therefore I am deleting the tests that seem to be providing the least amount of marginal value (based on the fact that they never pass).

[fejta]

Will delete these tests in a couple weeks unless someone signs up to fix them: https://github.com/kubernetes/test-infra/blob/master/experiment/bigquery/failures-latest.json

[pipejakob]

Sign me up for pr:pull-kubernetes-e2e-kubeadm-gce. I have a WIP PR (#2509) to fix it.

[fejta]

Cool! And that one is a month away from the 60d mark anyway :)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[BenTheElder]

Heh, @fejta I think maybe we don't want the stale job issue to go stale :-)

[BenTheElder]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[BenTheElder]

/remove-lifecycle stale

/cc @mithrav @spiffxp @AishSundar
we should codify something like this and enact it to clean up jobs that have been failing for ridiculously long

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close