release-blocking jobs must run in dedicated cluster: ci-kubernetes-build

[spiffxp]

What should be cleaned up or changed:

This is part of #18549

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

NOTE: I am not tagging this as "help wanted" because it is blocked on kubernetes/k8s.io#846. I would also recommend doing ci-kubernetes-build-fast first. Here is my guess at how we could do this:

• create a duplicate job that pushes to the new bucket writable by k8s-infra-prow-build
• ensure it's building and pushing appropriately
• update a release-blocking job to pull from the new bucket
• if no problems, roll out changes progressively
  • a few more jobs in release-blocking
  • all jobs in release-blocking that use this job's results
  • a job that still runs in the "default" cluster
  • all jobs that use this job's results
• rename jobs / get rid of the job that runs on the "default" cluster
• do the same for release-branch variants, can probably do a faster rollout

It will be helpful to note the date/time that PR's merge. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=ci-kubernetes-build
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#ci-kubernetes-build&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=ci-kubernetes-build
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/ci-kubernetes-build
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for ci-kubernetes-build for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for ci-kubernetes-build for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs

[spiffxp]

/sig release
FYI @kubernetes/release-engineering

[justaugustus]

Arnaud has PRs in flight for this.
/assign @ameukam

and Carlos has been working on the build-fast stuff.
/assign @cpanato

[spiffxp]

Current obstacle is deciding how to allow the ci-kubernetes-build-canary job to push to gcr.io/k8s-staging-ci-images.

Currently:

• ci-kubernetes-build runs on default build cluster aka google.com-owned k8s-prow-builds
• all jobs in this cluster use the default pr-kubekins@kubernetes-jenkins gcp service account
• pr-kubekins has role roles/editor on the kubernetes-ci-images project
• gs://artifacts.kubernetes-ci-images.appspot.com/ (the bucket that corresponds to gcr.io/kubernetes-ci-images) gives roles/storage.legacyBucketOwner to project editors

What this means:

• any job that runs in default can push to kubernetes-ci-images, including presubmits for unrelated repos

What could we do for pushing to gcr.io/k8s-staging-ci-images?

• follow the same approach, allow the default prow-build@k8s-infra-prow-build service account on k8s-infra-prow-build to push to gcr.io/k8s-staging-ci-images
  • do this via adding the service account e-mail address to the k8s-infra-staging-ci-images@kubernetes.io group?
  • do this via direct iam bindings on the projects or buckets in question?
• create a service account that is granted push access to gcr.io/k8s-staging-ci-images, restrict its use to well known jobs via static tests
  • could do this for each staging project, or on demand
  • could hardcode job names in tests, or try enforcing some kind of path-based convention
• suggest that any project that needs to push images to staging repos should run on the "trusted" cluster
  • this would mean no presubmits could push images to staging repos

[ameukam]

create a service account that is granted push access to gcr.io/k8s-staging-ci-images, restrict its use to well known jobs via static tests

+1 for doing this for each staging project.
Does this involve a k8s service account for each staging project?

[ameukam]

do this via adding the service account e-mail address to the k8s-infra-staging-ci-images@kubernetes.io group?

ci-kubernetes-build-canary still fails even after the service account is added (see kubernetes/k8s.io#1393) to k8s-infra-staging-ci-images@kubernetes.io : https://testgrid.k8s.io/sig-testing-canaries#build-master-canary

prow-build service account inherits of the permissions of the role roles/cloudbuild.builds.editor as member of k8s-infra-staging-ci-images@kubernetes.io :

https://github.com/kubernetes/k8s.io/blob/74bfdc5741bdde3b8f489bdd8327474101b3b5e4/infra/gcp/lib.sh#L209-L231

which is not enough to make the job successful.

[spiffxp]

My read of GCS permissions is that members of the group have write and admin privileges to the bucket

$ gsutil iam get gs://artifacts.k8s-staging-ci-images.appspot.com/
{
  "bindings": [
    {
      "members": [
        "group:k8s-infra-artifact-admins@kubernetes.io",
        "projectEditor:k8s-staging-ci-images",
        "projectOwner:k8s-staging-ci-images"
      ],
      "role": "roles/storage.legacyBucketOwner"
    },
    {
      "members": [
        "projectViewer:k8s-staging-ci-images"
      ],
      "role": "roles/storage.legacyBucketReader"
    },
    {
      "members": [
        "group:k8s-infra-staging-ci-images@kubernetes.io" # here
      ],
      "role": "roles/storage.legacyBucketWriter"
    },
    {
      "members": [
        "group:k8s-infra-artifact-admins@kubernetes.io",
        "group:k8s-infra-staging-ci-images@kubernetes.io" # here
      ],
      "role": "roles/storage.objectAdmin"
    },
    {
      "members": [
        "allUsers"
      ],
      "role": "roles/storage.objectViewer"
    }
  ],
  "etag": "CAg="
}

[spiffxp]

Per https://cloud.google.com/iam/docs/understanding-roles

roles/storage.legacyBucketOwner

storage.buckets.get
storage.buckets.getIamPolicy
storage.buckets.setIamPolicy
storage.buckets.update
storage.objects.create
storage.objects.delete
storage.objects.list

roles/storage.legacyBucketWriter

storage.buckets.get
storage.objects.create
storage.objects.delete
storage.objects.list

roles/storage.objectAdmin

resourcemanager.projects.get
resourcemanager.projects.list
storage.objects.*

roles/storage.admin

firebase.projects.get
resourcemanager.projects.get
resourcemanager.projects.list
storage.buckets.*
storage.objects.*

[spiffxp]

Perhaps the service account isn't getting picked up as a group member

spiffxp@cloudshell:~ (k8s-infra-prow-build)$ gcloud auth activate-service-account prow-build@k8s-infra-prow-build.iam.gserviceaccount.com --key-file=prow-build.service-account.json
spiffxp@cloudshell:~ (k8s-infra-prow-build)$ docker push gcr.io/k8s-staging-ci-images/conformance-amd64:v1.20.0-beta.1.209_bf5382a53e1546
The push refers to repository [gcr.io/k8s-staging-ci-images/conformance-amd64]
3836a827d4dc: Preparing
75b964e845dc: Preparing
50aa71a59909: Preparing
a4ffc6e6326f: Preparing
fb8aa2688f2b: Preparing
a2bc020e977c: Waiting
b0300c32a489: Waiting
denied: Token exchange failed for project 'k8s-staging-ci-images'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

[spiffxp]

Opened kubernetes/k8s.io#1401 to do what I did manually

$ gsutil iam ch serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com:roles/storage.objectAdmin gs://artifacts.k8s-staging-ci-images.appspot.com
$ gsutil iam ch serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com:roles/storage.legacyBucketWriter gs://artifacts.k8s-staging-ci-images.appspot.com

Which I verified as working via a push which had failed before:

$ docker push gcr.io/k8s-staging-ci-images/conformance-amd64:v1.20.0-beta.1.209_bf5382a53e1546
The push refers to repository [gcr.io/k8s-staging-ci-images/conformance-amd64]
3836a827d4dc: Pushed
75b964e845dc: Pushed
50aa71a59909: Pushed
a4ffc6e6326f: Pushed
fb8aa2688f2b: Pushed
a2bc020e977c: Pushed
b0300c32a489: Pushed
v1.20.0-beta.1.209_bf5382a53e1546: digest: sha256:a97c22c59446a50398f42ebcd3af4f9999ec82adb008ce7dbed2f016536d3ef2 size: 1793

[LappleApple]

@cpanato @ameukam Hello both: Wondering what your suggested next step might be from a Release Engineering group perspective?

[cpanato]

I need to see what is the status of this item and maybe we can plan the next steps (@LappleApple)

@ameukam do you have any updates that you can provide to check where we are with this issue?

[ameukam]

@LappleApple @cpanato The only update I'm aware is the switch to gcr.io/k8s-staging-ci-images : kubernetes/kubernetes#97087.
As suggested (#19483 (comment)) by spiffxp, the next step would be a KEP describing how we can migrate away from kubernetes-release-dev bucket.

[spiffxp]

IMO next steps are:

• migrate job configs to use community-owned artifacts: this would close out Migrate jobs away from gs://kubernetes-release-dev k8s.io#846 (comment) along the way, I would approach this by doing release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment) and release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment)
• deprecate all further use of kubernetess-release-dev (could be a KEP): release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment)
• implement deprecation window (keep both jobs? setuping syncing?)

[spiffxp]

/milestone v1.21

[spiffxp]

/priority important-longterm

[spiffxp]

/remove-priority important-longterm
/priority important-soon

[spiffxp]

• Log or enforce kubernetes jobs pull from k8s-infra buckets #20885 - enforced that all release-blocking jobs pull from gs://k8s-release-dev
• Deprecate google-hosted kubernetes-build jobs #20964 - proposes renaming build jobs to build-deprecated and moving to release-informing, renaming build-canary jobs to build and moving to release-blocking

[spiffxp]

/close
Work related to fully deprecating the google-hosted artifacts from these jobs (#19483 (comment)) will be tracked under kubernetes/k8s.io#1571

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Work related to fully deprecating the google-hosted artifacts from these jobs (#19483 (comment)) will be tracked under kubernetes/k8s.io#1571

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.