Revert "Revert "Add boskos pool for scalability jobs. Move scalabilit…

[amwat]

…y jobs from us-central1-f to us-east1-b.""

This reverts commit 1f93fa8.
/cc @krzyzacy @shyamjvs

[krzyzacy]

/lgtm
/hold
🤞

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amwat, krzyzacy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• boskos/OWNERS [amwat,krzyzacy]
• config/OWNERS [amwat,krzyzacy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[amwat]

/hold cancel
I'll keep an eye on the scalability jobs.

[k8s-ci-robot]

@amwat: Updated the job-config configmap using the following files:

• key sig-scalability-periodic-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml
• key kubetest-canaries.yaml using file config/jobs/kubernetes/sig-testing/kubetest-canaries.yaml

In response to this:

…y jobs from us-central1-f to us-east1-b.""

This reverts commit 1f93fa8.
/cc @krzyzacy @shyamjvs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[amwat]

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/18163/build-log.txt has cluster creation using the new boskos project_type working.

[shyamjvs]

I'm still seeing a bunch of issues with Cluster up in some scalability jobs, e.g:

• https://k8s-testgrid.appspot.com/sig-scalability-gce#gce-1.9
• https://k8s-testgrid.appspot.com/sig-scalability-gce#gce-1.10

@amwat - Could you take a look?

[amwat]

Looks like those cluster up failures are before this fix went in on 09/25. After that all the latter jobs are either passing or flaking on kubetest timeout.

[shyamjvs]

We're still seeing failures, even from yesterday (e.g - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/18215)
And I think those kubetest timeouts are related to the boskos change. For e.g the run I linked failed due to timing out while trying to create all nodes, which usually happens when there isn't enough quota in the project. And indeed the in-use IP addresses quota seems to be exhausted. Also, I think the limit of 23 explains why 78 nodes were pending to be created (from build-log).

[shyamjvs]

There seems to be same problem even for this project 'k8s-jenkins-gci-scalability'

[krzyzacy]

@amwat didn't you ran quota-fixer and unified all quotas?

[amwat]

Hmm, the quotas are fixed but looks like some quota requests are under provisioned, I'll add a fix for that.