release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

spiffxp · 2020-10-07T21:10:58Z

What should be cleaned up or changed:

This is part of #18549

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

NOTE: I am not tagging this as "help wanted" because it is blocked on kubernetes/k8s.io#846. I would also recommend doing ci-kubernetes-build-fast first. Here is my guess at how we could do this:

create a duplicate job that pushes to the new bucket writable by k8s-infra-prow-build
ensure it's building and pushing appropriately
update a release-blocking job to pull from the new bucket
if no problems, roll out changes progressively
- a few more jobs in release-blocking
- all jobs in release-blocking that use this job's results
- a job that still runs in the "default" cluster
- all jobs that use this job's results
rename jobs / get rid of the job that runs on the "default" cluster
do the same for release-branch variants, can probably do a faster rollout

It will be helpful to note the date/time that PR's merge. This will allow you to compare before/after behavior.

Things to watch for the job

https://prow.k8s.io/?job=ci-kubernetes-build
- does the job start failing more often?
- does the job start going into error state?
https://testgrid.k8s.io/presubmits-kubernetes-blocking#ci-kubernetes-build&graph-metrics=test-duration-minutes
- does the job duration look worse than before? spikier than before?
https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=ci-kubernetes-build
- do more failures show up than before?
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/ci-kubernetes-build
- (can be used to answer some of the same questions as above)
metrics explorer: CPU limit utilization for ci-kubernetes-build for 6h
- is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
- (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
metrics explorer: Memory limit utilization for ci-kubernetes-build for 6h
- is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
- (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

prow-build dashboard 1w
- is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
- (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
prowjobs-experiment 1w
- (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs

The text was updated successfully, but these errors were encountered:

spiffxp · 2020-10-07T21:28:18Z

/sig release
FYI @kubernetes/release-engineering

justaugustus · 2020-11-04T14:26:07Z

Arnaud has PRs in flight for this.
/assign @ameukam

and Carlos has been working on the build-fast stuff.
/assign @cpanato

spiffxp · 2020-11-05T00:48:59Z

Current obstacle is deciding how to allow the ci-kubernetes-build-canary job to push to gcr.io/k8s-staging-ci-images.

Currently:

ci-kubernetes-build runs on default build cluster aka google.com-owned k8s-prow-builds
all jobs in this cluster use the default pr-kubekins@kubernetes-jenkins gcp service account
pr-kubekins has role roles/editor on the kubernetes-ci-images project
gs://artifacts.kubernetes-ci-images.appspot.com/ (the bucket that corresponds to gcr.io/kubernetes-ci-images) gives roles/storage.legacyBucketOwner to project editors

What this means:

any job that runs in default can push to kubernetes-ci-images, including presubmits for unrelated repos

What could we do for pushing to gcr.io/k8s-staging-ci-images?

follow the same approach, allow the default prow-build@k8s-infra-prow-build service account on k8s-infra-prow-build to push to gcr.io/k8s-staging-ci-images
- do this via adding the service account e-mail address to the k8s-infra-staging-ci-images@kubernetes.io group?
- do this via direct iam bindings on the projects or buckets in question?
create a service account that is granted push access to gcr.io/k8s-staging-ci-images, restrict its use to well known jobs via static tests
- could do this for each staging project, or on demand
- could hardcode job names in tests, or try enforcing some kind of path-based convention
suggest that any project that needs to push images to staging repos should run on the "trusted" cluster
- this would mean no presubmits could push images to staging repos

ameukam · 2020-11-05T01:17:03Z

create a service account that is granted push access to gcr.io/k8s-staging-ci-images, restrict its use to well known jobs via static tests

+1 for doing this for each staging project.
Does this involve a k8s service account for each staging project?

ameukam · 2020-11-06T14:47:31Z

do this via adding the service account e-mail address to the k8s-infra-staging-ci-images@kubernetes.io group?

ci-kubernetes-build-canary still fails even after the service account is added (see kubernetes/k8s.io#1393) to k8s-infra-staging-ci-images@kubernetes.io : https://testgrid.k8s.io/sig-testing-canaries#build-master-canary

prow-build service account inherits of the permissions of the role roles/cloudbuild.builds.editor as member of k8s-infra-staging-ci-images@kubernetes.io :

https://github.com/kubernetes/k8s.io/blob/74bfdc5741bdde3b8f489bdd8327474101b3b5e4/infra/gcp/lib.sh#L209-L231

which is not enough to make the job successful.

spiffxp · 2020-11-06T18:06:17Z

My read of GCS permissions is that members of the group have write and admin privileges to the bucket

$ gsutil iam get gs://artifacts.k8s-staging-ci-images.appspot.com/
{
  "bindings": [
    {
      "members": [
        "group:k8s-infra-artifact-admins@kubernetes.io",
        "projectEditor:k8s-staging-ci-images",
        "projectOwner:k8s-staging-ci-images"
      ],
      "role": "roles/storage.legacyBucketOwner"
    },
    {
      "members": [
        "projectViewer:k8s-staging-ci-images"
      ],
      "role": "roles/storage.legacyBucketReader"
    },
    {
      "members": [
        "group:k8s-infra-staging-ci-images@kubernetes.io" # here
      ],
      "role": "roles/storage.legacyBucketWriter"
    },
    {
      "members": [
        "group:k8s-infra-artifact-admins@kubernetes.io",
        "group:k8s-infra-staging-ci-images@kubernetes.io" # here
      ],
      "role": "roles/storage.objectAdmin"
    },
    {
      "members": [
        "allUsers"
      ],
      "role": "roles/storage.objectViewer"
    }
  ],
  "etag": "CAg="
}

spiffxp · 2020-11-06T19:41:34Z

Per https://cloud.google.com/iam/docs/understanding-roles

roles/storage.legacyBucketOwner

storage.buckets.get
storage.buckets.getIamPolicy
storage.buckets.setIamPolicy
storage.buckets.update
storage.objects.create
storage.objects.delete
storage.objects.list

roles/storage.legacyBucketWriter

storage.buckets.get
storage.objects.create
storage.objects.delete
storage.objects.list

roles/storage.objectAdmin

resourcemanager.projects.get
resourcemanager.projects.list
storage.objects.*

roles/storage.admin

firebase.projects.get
resourcemanager.projects.get
resourcemanager.projects.list
storage.buckets.*
storage.objects.*

spiffxp · 2020-11-06T19:42:51Z

Perhaps the service account isn't getting picked up as a group member

spiffxp@cloudshell:~ (k8s-infra-prow-build)$ gcloud auth activate-service-account prow-build@k8s-infra-prow-build.iam.gserviceaccount.com --key-file=prow-build.service-account.json
spiffxp@cloudshell:~ (k8s-infra-prow-build)$ docker push gcr.io/k8s-staging-ci-images/conformance-amd64:v1.20.0-beta.1.209_bf5382a53e1546
The push refers to repository [gcr.io/k8s-staging-ci-images/conformance-amd64]
3836a827d4dc: Preparing
75b964e845dc: Preparing
50aa71a59909: Preparing
a4ffc6e6326f: Preparing
fb8aa2688f2b: Preparing
a2bc020e977c: Waiting
b0300c32a489: Waiting
denied: Token exchange failed for project 'k8s-staging-ci-images'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

spiffxp · 2020-11-06T20:23:52Z

Opened kubernetes/k8s.io#1401 to do what I did manually

$ gsutil iam ch serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com:roles/storage.objectAdmin gs://artifacts.k8s-staging-ci-images.appspot.com
$ gsutil iam ch serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com:roles/storage.legacyBucketWriter gs://artifacts.k8s-staging-ci-images.appspot.com

Which I verified as working via a push which had failed before:

$ docker push gcr.io/k8s-staging-ci-images/conformance-amd64:v1.20.0-beta.1.209_bf5382a53e1546
The push refers to repository [gcr.io/k8s-staging-ci-images/conformance-amd64]
3836a827d4dc: Pushed
75b964e845dc: Pushed
50aa71a59909: Pushed
a4ffc6e6326f: Pushed
fb8aa2688f2b: Pushed
a2bc020e977c: Pushed
b0300c32a489: Pushed
v1.20.0-beta.1.209_bf5382a53e1546: digest: sha256:a97c22c59446a50398f42ebcd3af4f9999ec82adb008ce7dbed2f016536d3ef2 size: 1793

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

Add canary jobs for ci-kubernetes-build job variants.

lasomethingsomething · 2021-01-19T16:01:06Z

@cpanato @ameukam Hello both: Wondering what your suggested next step might be from a Release Engineering group perspective?

cpanato · 2021-01-19T16:08:03Z

I need to see what is the status of this item and maybe we can plan the next steps (@LappleApple)

@ameukam do you have any updates that you can provide to check where we are with this issue?

ameukam · 2021-01-20T19:41:21Z

@LappleApple @cpanato The only update I'm aware is the switch to gcr.io/k8s-staging-ci-images : kubernetes/kubernetes#97087.
As suggested (#19483 (comment)) by spiffxp, the next step would be a KEP describing how we can migrate away from kubernetes-release-dev bucket.

spiffxp · 2021-01-25T23:07:37Z

IMO next steps are:

migrate job configs to use community-owned artifacts: this would close out Migrate jobs away from gs://kubernetes-release-dev k8s.io#846 (comment) along the way, I would approach this by doing release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment) and release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment)
deprecate all further use of kubernetess-release-dev (could be a KEP): release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483 (comment)
implement deprecation window (keep both jobs? setuping syncing?)

spiffxp · 2021-02-02T00:38:20Z

/milestone v1.21

spiffxp · 2021-02-08T17:37:39Z

/priority important-longterm

spiffxp · 2021-02-09T19:59:20Z

/remove-priority important-longterm
/priority important-soon

Ensure periodic-kubernetes-bazel-build can run k8s-infra-prow-build cluster and push artifacts on k8s-release-dev bucket. Related to : kubernetes#19483 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

spiffxp · 2021-02-23T08:12:44Z

Log or enforce kubernetes jobs pull from k8s-infra buckets #20885 - enforced that all release-blocking jobs pull from gs://k8s-release-dev
Deprecate google-hosted kubernetes-build jobs #20964 - proposes renaming build jobs to build-deprecated and moving to release-informing, renaming build-canary jobs to build and moving to release-blocking

spiffxp · 2021-02-24T17:46:14Z

/close
Work related to fully deprecating the google-hosted artifacts from these jobs (#19483 (comment)) will be tracked under kubernetes/k8s.io#1571

k8s-ci-robot · 2021-02-24T17:46:23Z

@spiffxp: Closing this issue.

In response to this:

/close
Work related to fully deprecating the google-hosted artifacts from these jobs (#19483 (comment)) will be tracked under kubernetes/k8s.io#1571

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

spiffxp added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Oct 7, 2020

k8s-ci-robot added wg/k8s-infra sig/testing Categorizes an issue or PR as relevant to SIG Testing. area/jobs labels Oct 7, 2020

spiffxp mentioned this issue Oct 7, 2020

Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

Closed

27 tasks

k8s-ci-robot added the sig/release Categorizes an issue or PR as relevant to SIG Release. label Oct 7, 2020

spiffxp mentioned this issue Oct 8, 2020

Migrate jobs away from gs://kubernetes-release-dev kubernetes/k8s.io#846

Closed

ameukam mentioned this issue Oct 14, 2020

Create ci-kubernetes-build-canary job #19574

Merged

justaugustus mentioned this issue Nov 4, 2020

releng: Teach jobs using latest-fast marker to extract from K8s Infra #19841

Merged

k8s-ci-robot assigned ameukam and cpanato Nov 4, 2020

justaugustus mentioned this issue Nov 6, 2020

push-build.sh container image pushes should precede staging GCS artifacts and writing version markers kubernetes/release#1693

Closed

thejoycekung mentioned this issue Nov 6, 2020

[Failing Test] build-canary fails on push-build.sh (exit status 2) due to lack of permissions kubernetes/kubernetes#95936

Closed

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

a769ca3

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam mentioned this issue Nov 10, 2020

Add canary jobs for ci-kubernetes-build job variants. #19904

Merged

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

c13249d

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

3d2c36e

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

9f0d9c8

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

k8s-ci-robot added a commit that referenced this issue Nov 10, 2020

Merge pull request #19904 from ameukam/GH-19483

75af433

Add canary jobs for ci-kubernetes-build job variants.

justaugustus mentioned this issue Nov 10, 2020

ci-kubernetes-build-* jobs should build without requiring bootstrap.py scenarios kubernetes/release#1711

Closed

23 tasks

This was referenced Nov 30, 2020

kinder: migrate away from gs://kubernetes-release-dev kubernetes/kubeadm#2355

Closed

kubeadm: change the default imageRepository value for CI images kubernetes/kubeadm#2356

Closed

This was referenced Dec 3, 2020

Add staging GCR repo for Kubernetes CI images kubernetes/k8s.io#1355

Merged

Migrate away from google.com gcp project kubernetes-ci-images kubernetes/k8s.io#1460

Closed

SataQiu mentioned this issue Jan 25, 2021

add ci-kubernetes-build-1-20-canary job #20586

Merged

k8s-ci-robot added this to the v1.21 milestone Feb 2, 2021

k8s-ci-robot added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Feb 8, 2021

ameukam mentioned this issue Feb 22, 2021

Add canary jobs for periodic-kubernetes-bazel-build variants #20948

Closed

spiffxp mentioned this issue Feb 23, 2021

Deprecate google-hosted kubernetes-build jobs #20964

Merged

spiffxp mentioned this issue Feb 24, 2021

Migrate away from google.com GCP project google-containers kubernetes/k8s.io#1571

Open

6 tasks

k8s-ci-robot closed this as completed Feb 24, 2021

spiffxp mentioned this issue Jul 29, 2021

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images kubernetes/k8s.io#2318

Closed

42 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

spiffxp commented Oct 7, 2020 •

edited

Loading

spiffxp commented Oct 7, 2020

justaugustus commented Nov 4, 2020

spiffxp commented Nov 5, 2020

ameukam commented Nov 5, 2020 •

edited

Loading

ameukam commented Nov 6, 2020 •

edited

Loading

spiffxp commented Nov 6, 2020 •

edited

Loading

spiffxp commented Nov 6, 2020

spiffxp commented Nov 6, 2020

spiffxp commented Nov 6, 2020

lasomethingsomething commented Jan 19, 2021

cpanato commented Jan 19, 2021

ameukam commented Jan 20, 2021

spiffxp commented Jan 25, 2021

spiffxp commented Feb 2, 2021

spiffxp commented Feb 8, 2021

spiffxp commented Feb 9, 2021

spiffxp commented Feb 23, 2021

spiffxp commented Feb 24, 2021

k8s-ci-robot commented Feb 24, 2021

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

Comments

spiffxp commented Oct 7, 2020 • edited Loading

spiffxp commented Oct 7, 2020

justaugustus commented Nov 4, 2020

spiffxp commented Nov 5, 2020

ameukam commented Nov 5, 2020 • edited Loading

ameukam commented Nov 6, 2020 • edited Loading

spiffxp commented Nov 6, 2020 • edited Loading

spiffxp commented Nov 6, 2020

spiffxp commented Nov 6, 2020

spiffxp commented Nov 6, 2020

lasomethingsomething commented Jan 19, 2021

cpanato commented Jan 19, 2021

ameukam commented Jan 20, 2021

spiffxp commented Jan 25, 2021

spiffxp commented Feb 2, 2021

spiffxp commented Feb 8, 2021

spiffxp commented Feb 9, 2021

spiffxp commented Feb 23, 2021

spiffxp commented Feb 24, 2021

k8s-ci-robot commented Feb 24, 2021

spiffxp commented Oct 7, 2020 •

edited

Loading

ameukam commented Nov 5, 2020 •

edited

Loading

ameukam commented Nov 6, 2020 •

edited

Loading

spiffxp commented Nov 6, 2020 •

edited

Loading