Develop a prow migration plan

[spiffxp]

Break out into issues when we've gotten consensus

Current sketch is:

• decide whether we want a build cluster, and/or how many we we want
• allow prow.k8s.io to schedule jobs on our build cluster(s)
• setup a prow instance (in aaa?)
• figure out how we want to migrate/cut-over github traffic

Some concerns include:

• jobs that run in same cluster as prow.k8s.io to take advantage of the ghproxy instance that lives there
• using a new github bot user/token so as not to conflict with k8s-ci-robot while prow.k8s.io continues to use it
• setting up boskos in our build cluster(s)
• setting up greenhouse in our build cluster(s)
• setting up projects for boskos to manage
• migration of google-owned buckets like kubernetes-jenkins
• on-call

[spiffxp]

/wg k8s-infra
/sig testing
/area cluster-infra
/assign

[spiffxp]

I prototyped a single build cluster here #806

I have it hooked up to my personal prow instance here: https://prow.bashfire.dev

[spiffxp]

Trying to work through what the plan looks like, it's unlikely we'll get to discussing the actual "migrate prow.k8s.io" part in much detail this week.

https://docs.google.com/document/d/1g7RBDhbTAH1VBMl2kJZR71OfzM6cJF277zbysd2AoCQ/edit#

[spiffxp]

Discussed during today's meeting, came away with:

• We'll do two build clusters, each in their own project (trusted, untrusted)
• Untrusted build cluster will use same machine type as used by prow.k8s.io's existing build cluster (n1-highmem-8)
• Will use regional clusters
• Will start with e2e project pool that is not sig/job-specific
  • Adjust when costs get to unknown/unexplained size
  • Consider per-sig project pools
• Naming Is Hard:
  • k8s-infra-prow-build - untrusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-prow-build-trusted - trusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-e2e-{suffix} - e2e projects intended to be pinned to a job
  • k8s-infra-e2e-boskos-{nnn} - e2e projects intended to be managed by boskos
• What I'm aiming for in two weeks:
  • clusters hooked up to prow.k8s.io
  • confirmed migration of:
    • ci-kubernetes-node-kubelet
    • ci-kubernetes-e2e-gci-gce
    • (an image pushing job?)
  • dev/test cycle for migrating jobs

[spiffxp]

#830 implements all of the above except dev/test cycle

[spiffxp]

Opened up some more issues to represent followup work:

• - Migrate release-master-blocking jobs to k8s-infra-prow-build #841 - migrate release-master-blocking jobs to prow-build cluster
• - Setup greenhouse in k8s-infra-prow-build cluster #842 - setup greenhouse
• - Setup ghproxy in k8s-infra-prow-build-trusted #843 - setup ghproxy
• - Develop better ACLs for prow build clusters and e2e projects #844 - better ACLs to empower more people
• - Setup postsubmit jobs to deploy prow build cluster resources #845 - postsubmit jobs to deploy cluster resources
• - Migrate jobs away from gs://kubernetes-release-dev #846 - migrate jobs away from gs://kubernetes-release-dev

[MHBauer]

Learning about sig-node ci, can I help with or shadow along changes to ci-kubernetes-node-kubelet?

[spiffxp]

@MHBauer add yourself to the k8s-infra-prow-viewers@kubernetes.io group in groups/groups.yaml and let me know if you can see https://console.cloud.google.com/monitoring/dashboards?project=k8s-infra-prow-build

[helenfeng737]

I can help with the migration work too if needed. Already added myself in k8s-infra-prow-viewers

[spiffxp]

About ready to enumerate:

• setting up a staging prow instance to experiment with running in community infra
• how to shift traffic from prow.k8s.io

[spiffxp]

/assign @ameukam
since also working on staging prow (ref: #1394)

[spiffxp]

/priority important-longterm
I don't know how much time I'll have to work on completing a plan by end of v1.21

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

/lifecycle frozen

[spiffxp]

/milestone v1.23

[spiffxp]

/remove-priority important-longterm
/priority important-soon

[spiffxp]

The proposal linked in this issue likely needs to be refreshed, but dropping a few thoughts before I forget.

Blockers to migrating all jobs over to a community-run build cluster include:

• setting resource requests/limits on all jobs to properly trigger autoscaling
• using community-owned projects or granting privileges to the community-owned build cluster and boskos to manage google.com projects
  • manually pinned projects (e.g. Remove custom projects from SIG Node test manifests test-infra#23822, the list from Umbrella issue: migrate away from google.com gcp projects #1469)
  • gke projects (Decommission gke-project-pool test-infra#23778)
• more budget than we have capacity for based on our current spend w/ scalability jobs migrated over

Blockers to shutting down prow.k8s.io and then standing it all up over in kubernetes-public include:

• all of the existing kubernetes/test-infra tooling / manifests assume deployment to its own cluster vs. a shared cluster
• allowing a community-owned prow to trigger builds in the google.com-owned "default" build cluster (if we fail to migrate all jobs over
• allowing a community-owned prow to write/update gs://k8s-testgrid
• allowing a community-owned prow to write/update gs://kubernetes-jenkins
• allowing a community-owned prow to write to the google.com-owned k8s-prow project
• prow.k8s.io currently runs jobs for more than the kubernetes community (Migrate non-Kubernetes repos off of prow.k8s.io test-infra#12863)
• (not sure what else I'm missing here...)

[ameukam]

/milestone v1.24

[ameukam]

/milestone v1.30
/lifecycle clear

Rebooted this effort. See: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit?usp=sharing

[ameukam]

/milestone v1.31