Release 1.28.6

[justinsb]

Release 1.28.6

[k8s-triage-robot]

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

• The PR does have any do-not-merge/* labels
• The PR does not have the needs-ok-to-test label
• The PR is mergeable (does not have a needs-rebase label)
• The PR is approved (has cncf-cla: yes, lgtm, approved labels)
• The PR is failing tests required for merge

You can:

• Review the full test history for this PR
• Prevent this bot from retesting with /lgtm cancel or /hold
• Help make our tests less flaky by following our Flaky Tests Guide

/retest

[rifelpet]

/hold

@justinsb looks like some permissions need adjusting given the prow cluster migration

[justinsb]

Thanks @rifelpet

I think it's specific to the release-1.28 branch, because the 1.28 pull request CI jobs are trying to upload to kops-ci, which broke in the move to the prow community cluster. Newer branches have a different default for pull requests (because of #15981), so I proposed a cherry-pick to 1.28 of that PR to 1.28: #16678

There may well be other problems, we do seem to be using a mix of kops-ci markers and k8s-staging-kops markers, but I figure we can start here!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

/hold

actually can we get one more cherrypick in? #16673

[hakman]

/unhold
/test all

[hakman]

/retest

[hakman]

/hold

[hakman]

@ameukam Seems we need delete permission on the CI bucket:

AccessDeniedException: 403 prow-build@k8s-infra-prow-build.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.
make: *** [Makefile:270: gcs-publish-ci] Error 1

[ameukam]

@ameukam Seems we need delete permission on the CI bucket:

AccessDeniedException: 403 prow-build@k8s-infra-prow-build.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.
make: *** [Makefile:270: gcs-publish-ci] Error 1

I thought we fixed this. IIRC @upodroid updated the permissions for this bucket.

[hakman]

/hold

[hakman]

I thought we fixed this. IIRC @upodroid updated the permissions for this bucket.

Not sure if delete was also part of the permissions...

[upodroid]

It was just object creator, let me switch it to object admin

[justinsb]

Thanks for looking at it...

gsutil -h "Cache-Control:private, max-age=0, no-transform" cp /home/prow/go/src/k8s.io/kops/.build/upload/latest-ci.txt gs://k8s-staging-kops/pulls/pull-kops-e2e-k8s-aws-calico-1-28/pull-fdc9e9ca4416e65161a6716cd9125d46480178c3/
Copying file:///home/prow/go/src/k8s.io/kops/.build/upload/latest-ci.txt [Content-Type=text/plain]...
/ [0 files][    0.0 B/  165.0 B]                                                
AccessDeniedException: 403 prow-build@k8s-infra-prow-build.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

We are uploading to a per-job, per-sha directory. So I'm guessing this only needs delete when we are re-running the job. If delete is a problem, I could also rebase or tweak the commit comment to try to force a different SHA. Though I agree that we probably need delete permission... although it would be nice to catch instances where we try to upload the same GCS file twice, I think in practice there might be too many places where we do that legitimately.

[upodroid]

/retest

it's fixed now

[hakman]

/unhold