use gs://k8s-release-dev for kubernetes CI builds

[spiffxp]

Part of umbrella issue to migrate the project away from the google.com-owned gs://kubernetes-release-dev GCS bucket: kubernetes/k8s.io#2318

I broke this up into multiple commits for ease of revert. Each commit message includes a list of jobs potentially impacted

Since one of them involves a kubekins-e2e change, a followup bump PR will be required for all changes to take effect.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [spiffxp]
• images/kubekins-e2e/OWNERS [justaugustus,spiffxp]
• kubetest/OWNERS [spiffxp]
• scenarios/OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@spiffxp: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key kops-periodics-gce.yaml using file config/jobs/kubernetes/kops/kops-periodics-gce.yaml
• key kops-periodics-misc2.yaml using file config/jobs/kubernetes/kops/kops-periodics-misc2.yaml
• key kops-periodics-versions.yaml using file config/jobs/kubernetes/kops/kops-periodics-versions.yaml
• key kops-presubmits-e2e.yaml using file config/jobs/kubernetes/kops/kops-presubmits-e2e.yaml
• key kops-presubmits.yaml using file config/jobs/kubernetes/kops/kops-presubmits.yaml
• key kubernetes-builds.yaml using file config/jobs/kubernetes/sig-release/kubernetes-builds.yaml
• key 1.19.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.19.yaml
• key 1.20.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.20.yaml
• key 1.21.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.21.yaml
• key coverage.yaml using file config/jobs/kubernetes/sig-testing/coverage.yaml

In response to this:

Part of umbrella issue to migrate the project away from the google.com-owned gs://kubernetes-release-dev GCS bucket: kubernetes/k8s.io#2318

I broke this up into multiple commits for ease of revert. Each commit message includes a list of jobs potentially impacted

Since one of them involves a kubekins-e2e change, a followup bump PR will be required for all changes to take effect.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

This kicked off two image build jobs:

• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-test-infra-push-bootstrap/1413599075063828480
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-test-infra-push-kubekins-e2e/1413599075097382912

Before being so bold as to bump all the jobs to pickup the new kubekins, I'd like to ensure it's been built using the bootstrap image that was just pushed. So, a bump PR just for bootstrap image references: #22849

[spiffxp]

Now, a bump PR for kubekins-e2e: #22852

[spiffxp]

Timeline:

• 2021-07-09 20:40:00 UTC: use gs://k8s-release-dev for kubernetes CI builds #22840 merges - jobs using flags will be affected
• 2021-07-09 21:54:00 UTC: bump bootstrap image references #22849 merges - jobs using bootstrap will be affected
• 2021-07-09 23:51:00 UTC: jobs: bump kubekins-e2e #22852 merges - jobs using kubekins-e2e to run either bootstrap or kubetest will be affected

[justaugustus]

FYI @spiffxp @kubernetes/release-engineering -- the build-deprecated-** jobs have been failing

[CecileRobertMichon]

FYI all the Cluster API Azure/AWS/Openstack (possibly others too) conformance jobs on k8s main started failing over the weekend, I believe due to this PR.

kubernetes-sigs/cluster-api-provider-azure#1510

[justaugustus]

Slack chat: https://kubernetes.slack.com/archives/C2C40FMNF/p1625862678497300?thread_ts=1625862678.497300&cid=C2C40FMNF

[spiffxp]

@justaugustus

FYI @spiffxp @kubernetes/release-engineering -- the build-deprecated-** jobs have been failing

I opened kubernetes/kubernetes#103647 to track, and this PR should address the issue: #22864

[spiffxp]

@CecileRobertMichon from what I can gather this was due to references hardcoded in your repo? or was this due to the deprecated build jobs falling stale?

Looking at the linked issue it seems like y'all addressed with kubernetes-sigs/cluster-api-provider-azure#1512. Let me know if there's anything more to do

[CecileRobertMichon]

@CecileRobertMichon from what I can gather this was due to references hardcoded in your repo? or was this due to the deprecated build jobs falling stale?

The former, we have custom scripts that fetch artifacts to test Kubernetes CI versions which started failing because they were referencing gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images. We should be good now and have a handle on the fixes needed from our side, just wanted to give a heads up in case you were tracking the blast radius of the change.

[spiffxp]

@CecileRobertMichon

We should be good now and have a handle on the fixes needed from our side, just wanted to give a heads up in case you were tracking the blast radius of the change.

kubernetes/k8s.io#2318 is the tracking issue, if "our side" includes the other cluster-api projects I will hold off on PR'ing changes to those, I was going to proceed with kops next

[spiffxp]

FWIW I think the change that actually broke you was 10 days ago when we flipped dl.k8s.io to point to k8s-release-dev kubernetes/k8s.io#1857 (comment), I see there are bits of code that reference dl.k8s.io in your repo, e.g. https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/99748d4eadcbb98d7df50e9afa043048e5b1c7b0/scripts/ci-entrypoint.sh#L75

We just never noticed at that time because we happened to be lucky that the deprecated build job was keeping parity with the regular build job (which was my intent, seems like there's a ~90% parity between the two these days). I think kubernetes/kubernetes#103647 is what allowed you to notice that buckets had changed.

Apologies for the bump there.

[spiffxp]

Surveying for kops:

• I see a lot of red at https://testgrid.k8s.io/kops
• Spot checking though I still see recent passes, and things that are failing have always been failing (e.g. https://testgrid.k8s.io/kops-distros#kops-aws-distro-imagecentos7)
• Some of the red tabs are refusing to load (too many cells?) https://testgrid.k8s.io/kops-versions#kops-aws-k8s-latest

So it's not really easy to get a sense of "did everything break compared to before". I can maybe try throwing a query at the k8s-gubernator:builds dataset and see what comes back.

Having said that, I'm pretty sure I should expect to see a /kops/ subdir here in the new bucket and I don't

$ gsutil ls -l gs://kubernetes-release-dev/kops/ci/*/linux/amd64/kops | sort -k2
# ...
 163099887  2021-07-07T03:28:33Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+442e5eacbd/linux/amd64/kops
 163115795  2021-07-07T11:48:18Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+1a23c7dfef/linux/amd64/kops
 163115795  2021-07-07T16:10:30Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+b58caf6130/linux/amd64/kops
 163115795  2021-07-07T23:56:31Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+8aefbb3e29/linux/amd64/kops
 163139492  2021-07-08T12:29:14Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+53c7849d97/linux/amd64/kops
 163139492  2021-07-08T16:19:41Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+098a4a91ee/linux/amd64/kops
 163135412  2021-07-08T21:42:34Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+b358037896/linux/amd64/kops
 163135412  2021-07-09T12:12:50Z  gs://kubernetes-release-dev/kops/ci/1.22.0-alpha.2+e6ce40c8e4/linux/amd64/kops
$ gsutil ls -l gs://k8s-release-dev
                                 gs://k8s-release-dev/ci/

Searching around it seems like this job started failing: https://testgrid.k8s.io/kops-presubmits#kops-postsubmit

From the first failure:

>ResumableUploadAbortException: 403 pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com does not have storage.objects.create access to k8s-release-dev/kops/ci/1.22.0-alpha.2+4fe31321db/darwin/amd64/kops.

Yeah... that makes sense. This job is not running on the community-owned prow build cluster, and so far only the community cluster has privileges to write to community-iwned GCS buckets. I really wish I could get a list of everything else that's trying and failing to write to this bucket.

The quick fix would be to give pr-kubekins write access. But pr-kubekins is used to run more than just kubernetes community jobs (ref: #12863), so we could see things land in there that might need to be purged.

The longer fix would be to migrate this job, and any job trying and failing to write to this bucket, to the community-owned cluster. This then bumps into the can of worms that right now, merge-blocking and release-blocking jobs don't actually have their own dedicated node-pools. They've been getting by on the fact that nothing else substantial is on the community-owned cluster.

I'm gonna go ahead and say the "quick fix" is the one to take here, and we can gameday or schedule access revocation to see what truly needs it.

[spiffxp]

Opened kubernetes/k8s.io#2333 for the quick fix approach

[spiffxp]

New problem for kops

Building synchronization state...
Starting synchronization...
Copying file:///home/prow/go/src/k8s.io/kops/.bazelbuild/upload/kops/1.22.0-alpha.2+963149791f/darwin/amd64/kops [Content-Type=application/octet-stream]...
ResumableUploadAbortException: 400 Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access

Which, yeah, I'm holding the line on that. It needs to be changed to work with a UBLA bucket.

[spiffxp]

kubernetes/kops#11994 should fix the kops job