Migrate 100 node scalability release-blocking job to k8s-infra-prow-build

[spiffxp]

Demonstrate use of the scalability-project pool added to k8s-infra's boskos instance added via kubernetes/k8s.io#898

[wojtek-t]

/assign @mm4tt @jprzychodzen

@mborsz - FYI as current oncall

[jprzychodzen]

It seems that this change will affect access to projects, affecting possibility to debug for sig-scalability members.

Could you explain what is an expected IAM policy for projects in this pool?

[spiffxp]

@jprzychodzen what IAM policy do you need? I can create an @kubernetes.io google group for whomever scalability needs, and assign it the project viewer role for these projects, WDYT?

[jprzychodzen]

Right now we have an Owner on those projects, so it's hard to provide complete list of permissions that we are requiring.

Viewer seems like a good starting point for most of the use cases, however it would be great to have elevated privileges for some small subset of people - like Leadership and sig-scalability oncall. This way we will have quick way to react during emergencies (eg. broken tests).

[spiffxp]

I've opened kubernetes/k8s.io#919 which uses owner for leads and oncall

[spiffxp]

kubernetes/k8s.io#919 has merged, I'd like to see this in during working hours today so I can rollback if there are any issues, and leave it to soak over the weekend if not

[BenTheElder]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-scalability/OWNERS [BenTheElder,spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@spiffxp: Updated the job-config configmap in namespace default at cluster default using the following files:

• key sig-scalability-release-blocking-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml

In response to this:

Demonstrate use of the scalability-project pool added to k8s-infra's boskos instance added via kubernetes/k8s.io#898

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jprzychodzen]

Thanks! It looks good, it seems that there are appropriate permissions in place.

[spiffxp]

Keeping an eye on runs

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1266440343058911233 - timed out

Earlier runs hover around ~50min overall

So a 40min git fetch is probably part of the problem, but doesn't explain how it got over 120min

I0529 18:49:00.764] Call:  git fetch --quiet --tags https://github.com/kubernetes/kubernetes master
I0529 19:28:40.363] process 66 exited with code 0 after 39.7m

[jprzychodzen]

I do not know the details, but sig-scalability test jobs are not migrated to pod-utilities and use service account to directly access bucket and store logs. Just from a quick glance this can be related.

Ofc it's not realated to long git fetch, but also should be checked.

[jprzychodzen]

Last run passed, logexporter provided artifact into expected location.

[spiffxp]

FWIW this cluster uses a different service account, but it's also allowed write access into gs://kubernetes-jenkins

Looks like the timeout was a blip. Will let this run over the weekend to see how it fares before switching over the release-branch variants of this job

[spiffxp]

Checking back in: https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-scalability-100&width=5&graph-metrics=test-duration-minutes

There were some sporadic failures or timeouts over the weekend which I'm attributing to the build cluster being overloaded.

• scale prow-build pool 3x k8s.io#930 - raised the size of the cluster around 2020-05-31 3pm PT
• Upgrade prow-build to latest kubernetes 1.14 k8s.io#931 - upgraded all of the nodes around 2020-05-31 7:30pm PT onward

All seems calm since then.

[spiffxp]

ref: kubernetes/k8s.io#841