wg-k8s-infra: canary prowjobs for sig-scalability #22430

ameukam · 2021-06-05T01:30:00Z

Add canary jobs running on k8s-infra-prow-build for some sig-scalability periodics prowjobs.
Switch from --gcp-project=foo to --gcp-project=k8s-infra-e2e-scale-5k-project

Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

dims · 2021-06-05T01:56:13Z

/approve

k8s-ci-robot · 2021-06-05T01:56:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/wg-k8s-infra/OWNERS~~ [dims]
~~config/testgrids/kubernetes/wg-k8s-infa/OWNERS~~ [ameukam,dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ameukam · 2021-06-05T20:29:36Z

/assign @BenTheElder @spiffxp

ameukam · 2021-06-07T22:26:51Z

xref: kubernetes/k8s.io#1469.
Failures are expected so there is no need of a active babysitting.

spiffxp

/lgtm
/hold
If you expect these to fail and just want to see how, then remove the hold. But I'm not ok with the 5k node jobs as-is.

I would much rather see provisioning of a special 5k node project for the 5k node jobs. Like modify infra/gcp/prow/ensure-e2e-projects.sh to add a k8s-infra-e2e-scale-5k-project to E2E_MANUAL_PROJECTS and then pin the jobs to that.

spiffxp · 2021-06-07T23:02:56Z

config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml

+      - --gcp-master-image=gci
+      - --gcp-node-image=gci
+      - --gcp-node-size=e2-small
+      - --gcp-nodes=5000


We'll need a quota request for this. I'd rather avoid giving all scalability projects this kind of quota, we should pin to a specific project for this.

@spiffxp I'll make the quota requests in k8s-infra-e2e-scale-5k-project. See kubernetes/k8s.io#2225

spiffxp · 2021-06-07T23:03:53Z

config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml

+      - --env=KUBE_DNS_MEMORY_LIMIT=300Mi
+      - --extract=ci/latest-fast
+      - --extract-ci-bucket=k8s-release-dev
+      - --gcp-nodes=5000


Same. This should use the same project as correctness. What's the node type used here?

Took a lot at a successful job of ci-kubernetes-e2e-gce-scale-performance: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1401585162642264064/build-log.txt.
The node type appears to be e2-standard-32.

spiffxp · 2021-06-07T23:04:29Z

config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml

+      - --extract=ci/latest
+      - --gcp-node-image=gci
+      - --gcp-node-size=e2-standard-8
+      - --gcp-nodes=84


Honestly not sure if this will need quota increase or not

No need to increase the quota for this job. the smallest quota in us-east1 is 1250 (for all the projects with type scalability-project)

jkaniuk · 2021-06-08T11:05:05Z

There are various GCP quotas apart from CPU one that need to be lifted to run 5k node jobs (to be copied from legacy project).

Are you tracking this anywhere?

config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml

ameukam · 2021-06-09T12:05:00Z

There are various GCP quotas apart from CPU one that need to be lifted to run 5k node jobs (to be copied from legacy project).

Are you tracking this anywhere?

@jkaniuk For the moment nothing is tracked. Do you have an existing document with the list of quotas that need to be raised ? I can create a new document if not.

jkaniuk · 2021-06-10T12:51:36Z

@jkaniuk For the moment nothing is tracked. Do you have an existing document with the list of quotas that need to be raised ? I can create a new document if not.

Please do, I do not believe we have anything yet.

@wojtek-t, @jprzychodzen

Add e2e projects for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam · 2021-06-11T23:10:14Z

@jkaniuk For the moment nothing is tracked. Do you have an existing document with the list of quotas that need to be raised ? I can create a new document if not.

Please do, I do not believe we have anything yet.

@wojtek-t, @jprzychodzen

@jkaniuk I created https://docs.google.com/spreadsheets/d/1v9ynsUx3pcMJKVaHViuVqxVtRP1OmGIadvw2uro7D7Y/edit#gid=0. Feel free to adjust at your own convenience.

Add e2e project for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

Ref: kubernetes/k8s.io#1469 Add canaries running on `k8s-infra-prow-build` for some sig-scalability periodics prowjobs. Switch from `--gcp-project=foo` to `--gcp-project=k8s-infra-e2e-scale-5k-project` Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

dims · 2021-06-17T22:52:19Z

/lgtm
/hold

please feel free to remove hold if you are ready.

ameukam · 2021-06-18T06:28:59Z

/hold cancel

k8s-ci-robot · 2021-06-18T06:44:11Z

@ameukam: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key wg-k8s-infra-canaries.yaml using file config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml

In response to this:

Add canary jobs running on k8s-infra-prow-build for some sig-scalability periodics prowjobs.
Switch from --gcp-project=foo to --gcp-project=k8s-infra-e2e-scale-5k-project

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Add testgrid dashboard for wg-k8s-infra canaries prowjobs

92d79b3

Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/config Issues or PRs related to code in /config area/jobs area/testgrid sig/testing Categorizes an issue or PR as relevant to SIG Testing. wg/k8s-infra labels Jun 5, 2021

k8s-ci-robot requested review from dims and spiffxp June 5, 2021 01:30

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 5, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2021

k8s-ci-robot assigned BenTheElder and spiffxp Jun 5, 2021

spiffxp reviewed Jun 7, 2021

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jun 7, 2021

mborsz reviewed Jun 8, 2021

View reviewed changes

config/jobs/kubernetes/wg-k8s-infra/wg-k8s-infra-canaries.yaml Outdated Show resolved Hide resolved

ameukam mentioned this pull request Jun 10, 2021

infra/gcp/prow: Add e2e project for sig-scalability kubernetes/k8s.io#2195

Merged

ameukam force-pushed the canaries-special-projects branch from 398e9b8 to 9ee0994 Compare June 16, 2021 19:25

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 16, 2021

k8s-ci-robot assigned dims Jun 17, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 18, 2021

k8s-ci-robot merged commit 44b0738 into kubernetes:master Jun 18, 2021

k8s-ci-robot added this to the v1.22 milestone Jun 18, 2021

ameukam mentioned this pull request Jun 19, 2021

Add a canary job for ci-kubernetes-kubemark-gce-scale #22169

Closed

ameukam mentioned this pull request Jun 28, 2021

Migrate away from google.com gcp project kubernetes-scale kubernetes/k8s.io#2241

Closed

7 tasks

ameukam deleted the canaries-special-projects branch March 7, 2024 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wg-k8s-infra: canary prowjobs for sig-scalability #22430

wg-k8s-infra: canary prowjobs for sig-scalability #22430

ameukam commented Jun 5, 2021 •

edited

Loading

dims commented Jun 5, 2021

k8s-ci-robot commented Jun 5, 2021

ameukam commented Jun 5, 2021

ameukam commented Jun 7, 2021

spiffxp left a comment

spiffxp Jun 7, 2021

ameukam Jun 16, 2021

spiffxp Jun 7, 2021

ameukam Jun 8, 2021

spiffxp Jun 7, 2021

ameukam Jun 8, 2021

jkaniuk commented Jun 8, 2021

ameukam commented Jun 9, 2021 •

edited

Loading

jkaniuk commented Jun 10, 2021

ameukam commented Jun 11, 2021

dims commented Jun 17, 2021

ameukam commented Jun 18, 2021

k8s-ci-robot commented Jun 18, 2021

wg-k8s-infra: canary prowjobs for sig-scalability #22430

wg-k8s-infra: canary prowjobs for sig-scalability #22430

Conversation

ameukam commented Jun 5, 2021 • edited Loading

dims commented Jun 5, 2021

k8s-ci-robot commented Jun 5, 2021

ameukam commented Jun 5, 2021

ameukam commented Jun 7, 2021

spiffxp left a comment

Choose a reason for hiding this comment

spiffxp Jun 7, 2021

Choose a reason for hiding this comment

ameukam Jun 16, 2021

Choose a reason for hiding this comment

spiffxp Jun 7, 2021

Choose a reason for hiding this comment

ameukam Jun 8, 2021

Choose a reason for hiding this comment

spiffxp Jun 7, 2021

Choose a reason for hiding this comment

ameukam Jun 8, 2021

Choose a reason for hiding this comment

jkaniuk commented Jun 8, 2021

ameukam commented Jun 9, 2021 • edited Loading

jkaniuk commented Jun 10, 2021

ameukam commented Jun 11, 2021

dims commented Jun 17, 2021

ameukam commented Jun 18, 2021

k8s-ci-robot commented Jun 18, 2021

ameukam commented Jun 5, 2021 •

edited

Loading

ameukam commented Jun 9, 2021 •

edited

Loading