Migrate k8s-gubernator:builds dataset to wg-k8s-infra

[spiffxp]

Part of migrating away from gcp-project k8s-gubernator: #1308

This dataset: https://console.cloud.google.com/bigquery?p=k8s-gubernator

It's used by:

• kettle, which writes to it (migration ref: Migrate kettle to k8s-infra #787)
• go.k8s.io/triage, which uses results from queries against it (migration ref: Migrate triage dashboard to wg-k8s-infra #1305)
• metrics jobs, which run queries against it and store results in gcs (migration ref: Migrate test-infra metrics to wg-k8s-infra #1306)

It's publicly readable (though compute time for queries will be charged to their own GCP account), but not publicly writable

It's not quite clear to me what strategies are available to us:

• stop-the-world: stop writing data, copy to new dataset, point everything at new dataset
• sync-the-world: replicate data from old dataset to new dataset, point reporting tools to replicated dataset, then point kettle at new dataset
• backfill-the-world: cut everything over to the new dataset, then backfill (day, then week, then all)

Given that this is a public artifact I'm inclined to suggest we home this under the kubernetes-public gcp project. We should examine billing briefly to understand if there are associated network/compute costs we would want billed elsewhere.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

/milestone v1.21
/assign @spiffxp @MushuEE
to investigate possible approaches

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[spiffxp]

/remove lifecycle-stale
/milestone v1.23

[ameukam]

@spiffxp With kettle currently down, we can go with

stop-the-world: stop writing data, copy to new dataset, point everything at new dataset

I can add a BQ dataset in kubernetes-public and data transfer job.

[spiffxp]

/remove-priority important-longterm
/priority important-soon

[spiffxp]

Pretty sure I've seen activity on this in PRs related to kettle:

• kubernetes-public: Add BigQuery dataset for kettle #2737
• kubernetes-public: BigQuery Data transfer to k8s-infra-kettle #2747
• kubernetes-public: Enable BigQuery Data Transfer #2749

[ameukam]

Pretty sure I've seen activity on this in PRs related to kettle:

* [kubernetes-public: Add BigQuery dataset for kettle #2737](https://github.com/kubernetes/k8s.io/pull/2737)

* [kubernetes-public: BigQuery Data transfer to k8s-infra-kettle #2747](https://github.com/kubernetes/k8s.io/pull/2747)

* [kubernetes-public: Enable BigQuery Data Transfer #2749](https://github.com/kubernetes/k8s.io/pull/2749)

Next steps are :

• enable monitoring for gs://kubernetes-jenkins: https://github.com/kubernetes/test-infra/tree/master/kettle#pubsub
• deploy a staging instance on aaa

[ameukam]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/milestone clear
/priority backlog
/lifecycle frozen

[ameukam]

We can close this. From: #789.

All the datasets are gone.

cc @BenTheElder

/close

[k8s-ci-robot]

@ameukam: Closing this issue.

In response to this:

We can close this. From: #789.

All the datasets are gone.

cc @BenTheElder

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

Yeah, this should be in k8s-infra already. Thanks for closing.