Migrate away from google.com gcp project kubernetes-jenkins

[spiffxp]

Part of umbrella issue to migrate away from google.com gcp projects: #1469

Umbrella issue for migrating or removing dependence on all of the kubernetes project infra that lives under kubernetes-jenkins:

GCS buckets:
gsutil ls -p kubernetes-jenkins | grep -v kubernetes-staging- | sed -e 's/.*/- [ ] `&`/'

• gs://artifacts-upload-test/: TODO: ???
• gs://artifacts.kubernetes-jenkins.appspot.com/: TODO: sincerely doubt we need to migrate GCR buckets
• gs://gcf-sources-661044217466-us-central1/: TODO: going to assume we can ignore this
• gs://k8s-bazel-cache/: May be used by kubernetes/kubernetes bazel, may be defunct? Last updated files 2019-03-11
• gs://k8s-kops-gce/: TODO: ???
• gs://kubernetes-e2e-soak-configs/: TODO: ???
• gs://kubernetes-federation-release/: TODO: ??? for this and the other federation buckets, surely these aren't used
• gs://kubernetes-federation-release-1-4/
• gs://kubernetes-federation-release-1-5/
• gs://kubernetes-federation-release-1-6/
• gs://kubernetes-federation-release-1-7/
• gs://kubernetes-federation-release-1-8/
• gs://kubernetes-federation-release-jenkins/: TODO: ???
• gs://kubernetes-federation-release-serial/: TODO: ???
• gs://kubernetes-jenkins/: hosts all logs/artifacts from prow.k8s.io, will need a plan/proposal
• gs://kubernetes-jenkins-gcslogs/: definitely do not migrate
• gs://kubernetes-jenkins-old/: TODO: : we probably don't care if this is migrated
• gs://kubernetes-release-dev-jenkins/: TODO: this probably shouldn't be migrated
• gs://kubernetes-test-history/: we can ignore this
• gs://sig-scalability-logs/: should happen as part of Migrate away from google.com gcp project kubernetes-scale #2241
• gs://us.artifacts.kubernetes-jenkins.appspot.com/: TODO: sincerely doubt we need to migrate GCR buckets

Service Accounts:
I'm not sure of everything I am (not) allowed to list here. So this isn't an exhaustive list. But we should make sure none of the service accounts in this project are used in any IAM bindings in kubernetes.io. Googlers will need to help identify this.

• kubekins@kubernetes-jenkins.iam.gserviceaccount.com - This is the big one, identify when/where this has access to services/resources that should be migrated or have equivalents stood up in kubernetes.io
• test-owners@kubernetes-jenkins.iam.gserviceaccount.com
• queue-health@kubernetes-jenkins.iam.gserviceaccount.com

[spiffxp]

At the moment it's not clear to me whether we can update prow.k8s.io to support different jobs writing to different buckets, or if we're going to have to make the change in lockstep. This is why migrating gs://kubernetes-jenkins needs a proposal

[spiffxp]

/lifecycle frozen

[tosi3k]

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (Add GCS_ARTIFACTS_DIR env var handling in pod-utils test-infra#17215)
2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)

[ameukam]

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (Add GCS_ARTIFACTS_DIR env var handling in pod-utils test-infra#17215)
2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)

@tosi3k Do you want the contents of this bucket to be preserved? if yes, what is the estimated size of sig-scalability-logs ?

[tosi3k]

@tosi3k Do you want the contents of this bucket to be preserved?

We just need the last 90 days of the contents therein.

I wonder - would applying some retention policy to gs://sig-scalability-logs bucket make sense? I'm not a big GCS / test-infra expert but I think that some kind of mechanism like this is already done for gs://kubernetes-jenkins bucket where we store the logs for jobs that have finished in the last 90 days IIUC.

After we migrate the scalability job configs to use gs://k8s-infra-scalability-tests-logs instead and 90 days pass, we could remove the old bucket (gs://sig-scalability-logs) completely as there would be no need for it anymore. Would that make sense here?

if yes, what is the estimated size of sig-scalability-logs?

I don't know but if we were to introduce a 90d retention policy mechanism we would probably comfortably fit in 5 TBs. This is the size of the bucket after 97 days of existence:

[spiffxp]

/milestone v1.23
I don't know that we can get all of this done within v1.23 but I'd like to see us start pruning the random buckets away at the very least

[spiffxp]

/milestone v1.24

[ameukam]

/milestone clear

[BenTheElder]

I'm working on eliminating old unused buckets from this project.

kubernetes-jenkins and sig-scalability-logs still need a plan.

There's also the mysterious kubernetes-jenkins-gcslogs which has content written to it recently judging by the file names but we can't tell where from yet ...

[BenTheElder]

Circled back:

$ gsutil logging get gs://kubernetes-jenkins
{"logBucket": "kubernetes-jenkins-gcslogs", "logObjectPrefix": "kubernetes-jenkins"}

Cleaning that up now.

[BenTheElder]

gs://kubernetes-jenkins/ and gs://sig-scalability-logs/ remain, I cleaned up the rest, including the many many gs://kubernetes-staging-*

[tosi3k]

Opened kubernetes/test-infra#33248 for getting rid of gs://sig-scalability-logs from the scalability jobs configs.

[BenTheElder]

Thank you! The only remaining reference is

k8s.io/apps/gcsweb/deployment.yaml

Line 36 in a554228

- -b=sig-scalability-logs

which we should leave for a bit.

Later we should drop write permissions to this bucket, and then when we still don't see issues then we should drop it from gcsweb (no rush ...)

[BenTheElder]

so mostly we need to migrate off of gs://kubernetes-jenkins, I think with everything else going on there's a high chance we'll defer this to after the migration, as far as I know aside from the prow control plane there's that one and gs://kubernetes-release and then we're pretty much out of google.com into kubernetes.io 🤞

[ameukam]

/remove-lifecycle frozen
/milestone v1.32