Refactor and better manage scale-testing infra #9223

shyamjvs · 2018-08-31T18:38:52Z

Following today's infra-team meeting, I'm creating this umbrella for things we'd need to do for sig-scalability (that I can think of atm):

Move smaller scale projects to boskos wherever possible (short-term)
Move all scalability jobs to boskos (long-term, and needs more work)
Isolate GKE-testing from OSS ones project-wise
Delete deprecated projects or move to boskos. Few I could find:
- k8s-jkns-pr-kubemark
- k8s-jenkins-gci-kubemark (after moving kubemark-canary to boskos)
- k8s-jenkins-scalability (after moving 100-node scalability canary to boskos)
Host perf-dash.k8s.io on CNCF project and probably change it's DNS entry
Below are projects (roughly sorted from largest to smallest) in use by scalability jobs currently. We need to get their billing right.
- kubernetes-scale (still being used for some GKE scale tests)
- k8s-scale-testing (still being used for some GKE scale tests)
- k8s-presubmit-scale
- kubemark-scalability-testing
- k8s-e2e-gce-scalability-1-1
- k8s-e2e-gci-gce-scale-1-4
- k8s-jenkins-gci-scalability
- k8s-jenkins-scalability
- k8s-jenkins-blocking-kubemark
- k8s-jenkins-kubemark
- k8s-jkns-clusterloader

cc @kubernetes/sig-testing @kubernetes/sig-scalability-misc @dims @BenTheElder @thockin @amwat

shyamjvs · 2018-08-31T18:44:26Z

/assign
I can drive this overall, and do some of the pieces. But if someone wants to take up any AIs, you're gladly welcome :)

shyamjvs · 2018-08-31T20:12:28Z

Some analysis on projects/jobs that can be moved to boskos:

k8s-jenkins-scalability (used by scalability-canary job - uses ~100 CPU in us-central1-f)
k8s-jenkins-gci-scalability (used by gce-100 job - uses ~100 CPU in us-central-f)
k8s-e2e-gce-scalability-1-1 (used by gce-100-beta job - uses ~100 CPU in us-east1-b)
k8s-e2e-gci-gce-scale-1-4 (used by gce-100-stable1 and gce-100-stable2 jobs - each uses ~100 CPU in us-east1-b)
k8s-jenkins-kubemark (used by following jobs:

kubemark-100 - uses < 20 CPU (so this job can be moved to regular gce boskos pool)
kubemark-small-last-release - uses < 10 CPU (so this job can be moved to regular gce boskos pool)
kubemark-high-density - uses ~100 CPU in us-central1-f

k8s-jenkins-blocking-kubemark (used by kubemark-500 job - uses ~80 CPU)

So we can create a new boskos pool with all the above projects and use it for those jobs (as each of them needs about 100 CPU). Few blockers to clear before that:
- ~~[ ] Move CPU quota in projects 3 and 4 from us-east1-b to us-central1-f (as otherwise the pool would be inconsistent wrt the zone having the quota)~~ (in favour of below option)
- Move CPU quota in projects 1, 2, 5 and 6 from us-central1-f to us-east1-b
- ~~Increase quota for projects 5 and 6 from 100 CPU to a bit higher (sth like 110) so that other jobs using the pool would also be able to run on it~~ (done)

And now, some projects/jobs that can go to boskos right away:

~~k8s-jenkins-gci-kubemark (used by kubemark-canary - uses < 20 CPU)~~ (done)
~~k8s-jkns-clusterloader (used by clusterloader and clusterloader2 jobs - each of which uses < 10 CPU)~~ (done)

shyamjvs · 2018-08-31T20:34:25Z

@amwat @krzysied - Would you be able/interested to pick up some of the AIs above? That'll be quite helpful for the CNCF migration.

amwat · 2018-08-31T20:41:31Z

I can create the boskos pool and update the jobs accordingly.

shyamjvs · 2018-08-31T20:43:14Z

That would be great, thanks. Note that the blockers mentioned above need to be fixed first before doing that.

shyamjvs · 2018-08-31T20:52:31Z

Move CPU quota in projects 3 and 4 from us-east1-b to us-central1-f (as otherwise the pool would be inconsistent wrt the zone having the quota)

Actually, now that I think of it, it's probably better (for the long run) to move the remaining projects' quota in the opposite direction (us-central1-f -> us-east1-b) because:

we have more slack in us-east1-b wrt resources available
when we later split our large monolithic presubmit project into multiple smaller ones, they would all be in us-east1-b (where we've already been granted quota currently). So it'll be easier to achieve consistency. cc @BenTheElder

shyamjvs · 2018-09-17T16:23:38Z

/assign @amwat
(for the boskos part)

fejta-bot · 2018-12-16T17:00:05Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

shyamjvs · 2019-01-07T19:55:07Z

@amwat Would you be able to cross out the checklist items that are done already? (especially those around boskos)

shyamjvs · 2019-01-07T19:55:22Z

/remove-lifecycle stale

fejta-bot · 2019-04-07T20:25:16Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

k8s-ci-robot · 2019-04-07T20:25:16Z

@fejta-bot: Please provide 1 or more arguments. Example usage: /project 0.5.0, /project 0.5.0 To do, /project clear 0.4.0

In response to this:

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2019-04-07T20:25:16Z