Kops - Migrate remaining upgrade jobs to k8s infra prow clusters

[rifelpet]

This migrates the remaining kops upgrade jobs to the k8s infra prow clusters. These jobs will fail because they use the k8s 1.24 / 1.25 / 1.26 e2e.test binaries which are built with an aws-sdk-go version too old to authenticate on the EKS prow clusters. Once we see the failures we can assess whether we can ignore specific test cases or whether the whole job wont work in which case we'll just delete these jobs.

Refs: kubernetes/kops#16637 kubernetes/k8s.io#5127 kubernetes/k8s.io#2625

/cc @upodroid @hakman

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/kops/OWNERS [hakman,rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@rifelpet: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key kops-periodics-conformance.yaml using file config/jobs/kubernetes/kops/kops-periodics-conformance.yaml
• key kops-periodics-distros.yaml using file config/jobs/kubernetes/kops/kops-periodics-distros.yaml
• key kops-periodics-grid.yaml using file config/jobs/kubernetes/kops/kops-periodics-grid.yaml
• key kops-periodics-misc2.yaml using file config/jobs/kubernetes/kops/kops-periodics-misc2.yaml
• key kops-periodics-network-plugins.yaml using file config/jobs/kubernetes/kops/kops-periodics-network-plugins.yaml
• key kops-periodics-pipeline.yaml using file config/jobs/kubernetes/kops/kops-periodics-pipeline.yaml
• key kops-periodics-upgrades.yaml using file config/jobs/kubernetes/kops/kops-periodics-upgrades.yaml
• key kops-periodics-versions.yaml using file config/jobs/kubernetes/kops/kops-periodics-versions.yaml
• key kops-presubmits-distros.yaml using file config/jobs/kubernetes/kops/kops-presubmits-distros.yaml
• key kops-presubmits-e2e.yaml using file config/jobs/kubernetes/kops/kops-presubmits-e2e.yaml
• key kops-presubmits-network-plugins.yaml using file config/jobs/kubernetes/kops/kops-presubmits-network-plugins.yaml

In response to this:

This migrates the remaining kops upgrade jobs to the k8s infra prow clusters. These jobs will fail because they use the k8s 1.24 / 1.25 / 1.26 e2e.test binaries which are built with an aws-sdk-go version too old to authenticate on the EKS prow clusters. Once we see the failures we can assess whether we can ignore specific test cases or whether the whole job wont work in which case we'll just delete these jobs.

Refs: kubernetes/kops#16637 kubernetes/k8s.io#5127

/cc @upodroid @hakman

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[upodroid]

btw, we need to eliminate the dependency on the kops-ci bucket as that is no longer being updated after 1st of Aug.

k8s-staging-kops is the replacement bucket and we do have some jobs fetching kops from there

[upodroid]

I'll open a PR updating the marker locations

[rifelpet]

@upodroid which job publishes latest-ci.txt version marker to k8s-staging-kops? I see the marker files exist but I can't find the job that publishes them. I'd like to use it as a reference for migrating the release branch version markers to k8s-staging-kops.

[upodroid]

This job https://testgrid.k8s.io/sig-cluster-lifecycle-kops#kops-postsubmit-push-to-staging

[rifelpet]

Ah I see. This job doesn't create a kops cluster so the only access it needs is to the k8s-staging-kops bucket.

Any thoughts on how a job can both create AWS kops clusters and have write access to GCS? This is what blocks the presubmits and updown and pipeline jobs from being migrated to community-owned infra.

If that won't be possible, then we'd need the kops clusters to run in the same cloud provider as the artifact bucket, either move the jobs to create GCP clusters or create an S3 artifacts bucket. Moving the artifacts to S3 shouldn't be a problem because any consumers of the bucket will access it publicly and anonymously.