Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

infra/gcp/prow: Add e2e project for sig-scalability #2195

Merged
merged 1 commit into from
Jun 15, 2021
Merged

infra/gcp/prow: Add e2e project for sig-scalability #2195

merged 1 commit into from
Jun 15, 2021

Conversation

ameukam
Copy link
Member

@ameukam ameukam commented Jun 10, 2021

Add e2e projects for sig-scalability prowjobs that need 5K nodes.
Initial suggestion : kubernetes/test-infra#22430 (review)

Signed-off-by: Arnaud Meukam ameukam@gmail.com

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ labels Jun 10, 2021
@k8s-ci-robot k8s-ci-robot added area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters approved Indicates a PR has been approved by an approver from all required OWNERS files. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jun 10, 2021
@ameukam
Copy link
Member Author

ameukam commented Jun 10, 2021

/assign @spiffxp

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 10, 2021
@ameukam ameukam force-pushed the e2e-projects-scalability-5k-nodes branch from 6510918 to 7aa2e47 Compare June 11, 2021 22:58
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 11, 2021
spiffxp
spiffxp suggested changes Jun 11, 2021
View reviewed changes
Copy link
Member

@spiffxp spiffxp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I left my comment at pending so in case it doesn't come through: I would rather start with just the one project instead of provisioning five projects. That's five times more capacity than we have provisioned today. I'm in no rush to spend even more than we do today without concrete justification

infra/gcp/prow/ensure-e2e-projects.sh Outdated
Comment on lines 73 to 79
# e2e projects with 5000 nodes for scalability jobs
E2E_SCALE_5K_PROJECTS=()
for i in $(seq 1 5); do
E2E_SCALE_5K_PROJECTS+=("$(printf "k8s-infra-e2e-boskos-scale-5k-%02i" "$i")")
done
readonly E2E_SCALE_5K_PROJECTS

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still too many IMO, I'd rather keep us to just the one (k8s-infra-e2e-scale-5k-project) until there's a proven need for more than one

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spiffxp Somehow I missed your comment. 😞
PR updated.

@ameukam ameukam force-pushed the e2e-projects-scalability-5k-nodes branch from 7aa2e47 to db4f84d Compare June 14, 2021 18:57
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 14, 2021
spiffxp
spiffxp suggested changes Jun 15, 2021
View reviewed changes
Copy link
Member

@spiffxp spiffxp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit otherwise lgtm

infra/gcp/infra.yaml Outdated Show resolved Hide resolved
@ameukam
infra/gcp/prow: Add e2e project for sig-scalability
92fb97f 
Add e2e project for sig-scalability prowjobs that need 5K nodes.
Initial suggestion : kubernetes/test-infra#22430 (review)

Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
@ameukam ameukam force-pushed the e2e-projects-scalability-5k-nodes branch from db4f84d to 92fb97f Compare June 15, 2021 20:39
spiffxp
spiffxp approved these changes Jun 15, 2021
View reviewed changes
Copy link
Member

@spiffxp spiffxp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 15, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@spiffxp
Copy link
Member

spiffxp commented Jun 15, 2021

I'll take care of deploying

@k8s-ci-robot k8s-ci-robot merged commit eb35ea4 into kubernetes:main Jun 15, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jun 15, 2021
@spiffxp
Copy link
Member

spiffxp commented Jun 15, 2021 

$ ./ensure-e2e-projects.sh k8s-infra-e2e-scale-5k-project
Ensuring monitoring.prow.k8s.io can scrape k8s-infra-prow-build metrics endpoints
  ./ensure-e2e-projects.sh: line 192: ensure_prow_build_cluster_metrics_endpoints: command not found

Opened #2223 to fix, re-running...

@spiffxp
Copy link
Member

spiffxp commented Jun 15, 2021

Deployed

$ ./ensure-e2e-projects.sh k8s-infra-e2e-scale-5k-project
Ensuring external secrets exist for use by k8s-infra-prow-build-trusted
  Ensuring secret 'cncf-ci-github-token' exists in 'k8s-infra-prow-build-trusted' and is owned by 'k8s-infra-ii-coop@kubernetes.io'
  Updated secret [cncf-ci-github-token].
  Ensuring secret 'snyk-token' exists in 'k8s-infra-prow-build-trusted' and is owned by 'k8s-infra-code-organization@kubernetes.io'
  Updated secret [snyk-token].
Ensuring e2e projects used by prow...
  Configuring e2e project: k8s-infra-e2e-scale-5k-project
    Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/k8s-infra-e2e-scale-5k-project].
    Waiting for [operations/cp.6812659568010387061] to finish...
    ..done.
    billingAccountName: billingAccounts/018801-93540E-22A20E
    billingEnabled: true
    name: projects/k8s-infra-e2e-scale-5k-project/billingInfo
    projectId: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -1,2 +1 @@
    -- member: user:spiffxp@google.com
    -  role: roles/owner
    +[]
    Ensure stale role bindings have been removed from e2e project: k8s-infra-e2e-scale-5k-project
      no stale bindings slated for removal
    Ensuring only APIs necessary for kubernetes e2e jobs to use e2e project: k8s-infra-e2e-scale-5k-project
    plan to enable/disable the following services
    to_enable:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_disable: []
    Operation "operations/acf.p2-783965148085-e3561dac-7684-4ec5-8a4e-087848353038" finished successfully.
    Operation "operations/acf.p2-783965148085-73b68eda-a5fc-45e5-a827-90df93bfa3ef" finished successfully.
    Operation "operations/acf.p2-783965148085-69a4fc7c-d39b-44da-9978-eb05e6bd4abc" finished successfully.
    Operation "operations/acf.p2-783965148085-17922383-9f5a-4e41-b070-fb6d703dc45f" finished successfully.
    Operation "operations/acf.p2-783965148085-de4f0675-e9d1-4b66-8fd6-9ecf662d56cd" finished successfully.
    "INFO: dry-run mode, would run:" gcloud services disable --force oslogin.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force pubsub.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force storage-api.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    WARN: ensure_only_services: after enable/disable cycle, still projects to enable/disable: k8s-infra-e2e-scale-5k-project
    intent:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    enabled:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com
      - storage-component.googleapis.com
    expected:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_enable: []
    to_disable:
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com
    @@ -4,17 +4,23 @@ intent:
       - logging.googleapis.com
       - monitoring.googleapis.com
       - storage-component.googleapis.com
    -enabled: []
    -expected:
    +enabled:
       - compute.googleapis.com
       - containerregistry.googleapis.com
       - logging.googleapis.com
       - monitoring.googleapis.com
    +  - oslogin.googleapis.com
    +  - pubsub.googleapis.com
    +  - storage-api.googleapis.com
       - storage-component.googleapis.com
    -to_enable:
    +expected:
       - compute.googleapis.com
       - containerregistry.googleapis.com
       - logging.googleapis.com
       - monitoring.googleapis.com
       - storage-component.googleapis.com
    -to_disable: []
    +to_enable: []
    +to_disable:
    +  - oslogin.googleapis.com
    +  - pubsub.googleapis.com
    +  - storage-api.googleapis.com
    Empower prow-build service account to edit e2e project: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -2,6 +2,8 @@
       role: roles/editor
     - member: serviceAccount:783965148085@cloudservices.gserviceaccount.com
       role: roles/editor
    +- member: serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com
    +  role: roles/editor
     - member: serviceAccount:service-783965148085@compute-system.iam.gserviceaccount.com
       role: roles/compute.serviceAgent
     - member: serviceAccount:service-783965148085@containerregistry.iam.gserviceaccount.com
    Empower boskos-janitor service account to clean e2e project: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -2,6 +2,8 @@
       role: roles/editor
     - member: serviceAccount:783965148085@cloudservices.gserviceaccount.com
       role: roles/editor
    +- member: serviceAccount:boskos-janitor@k8s-infra-prow-build.iam.gserviceaccount.com
    +  role: roles/editor
     - member: serviceAccount:prow-build@k8s-infra-prow-build.iam.gserviceaccount.com
       role: roles/editor
     - member: serviceAccount:service-783965148085@compute-system.iam.gserviceaccount.com
    Empower k8s-infra-prow-oncall@kubernetes.io to admin e2e project: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -1,3 +1,5 @@
    +- member: group:k8s-infra-prow-oncall@kubernetes.io
    +  role: roles/owner
     - member: serviceAccount:783965148085-compute@developer.gserviceaccount.com
       role: roles/editor
     - member: serviceAccount:783965148085@cloudservices.gserviceaccount.com
    Empower k8s-infra-prow-viewers@kubernetes.io to view specific resources in e2e project: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -1,5 +1,7 @@
     - member: group:k8s-infra-prow-oncall@kubernetes.io
       role: roles/owner
    +- member: group:k8s-infra-prow-viewers@kubernetes.io
    +  role: organizations/758905017065/roles/prow.viewer
     - member: serviceAccount:783965148085-compute@developer.gserviceaccount.com
       role: roles/editor
     - member: serviceAccount:783965148085@cloudservices.gserviceaccount.com
    Empower k8s-infra-sig-scalability-oncall@kubernetes.io to admin e2e project: k8s-infra-e2e-scale-5k-project
    Updated IAM policy for project [k8s-infra-e2e-scale-5k-project].
    @@ -2,6 +2,8 @@
       role: roles/owner
     - member: group:k8s-infra-prow-viewers@kubernetes.io
       role: organizations/758905017065/roles/prow.viewer
    +- member: group:k8s-infra-sig-scalability-oncall@kubernetes.io
    +  role: roles/owner
     - member: serviceAccount:783965148085-compute@developer.gserviceaccount.com
       role: roles/editor
     - member: serviceAccount:783965148085@cloudservices.gserviceaccount.com
    Ensure prow-build prowjobs are able to ssh to instances in e2e project: k8s-infra-e2e-scale-5k-project
    Updated [https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-scale-5k-project].
    @@ -0,0 +1,2 @@
    +prow:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmYxHh/wwcV0P1aChuFLpl28w6DFyc7G5Xrw1F8wH1Re9AdxyemM2bTZ/PhsP3u9VDnNbyOw3UN00VFdumkFLjLf1WQ7Q6rZDlPjlw7urBIvAMqUecY6ae1znqsZ0dMBxOuPXHznlnjLjM5b7O7q5WsQMCA9Szbmz6DsuSyCuX0It2osBTN+8P/Fa6BNh3W8AF60M7L8/aUzLfbXVS2LIQKAHHD8CWqvXhLPuTJ03iSwFvgtAK1/J2XJwUP+OzAFrxj6A9LW5ZZgk3R3kRKr0xT/L7hga41rB1qy8Uz+Xr/PTVMNGW+nmU4bPgFchCK0JBK7B12ZcdVVFUEdpaAiKZ prow
    +prow:prow:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmYxHh/wwcV0P1aChuFLpl28w6DFyc7G5Xrw1F8wH1Re9AdxyemM2bTZ/PhsP3u9VDnNbyOw3UN00VFdumkFLjLf1WQ7Q6rZDlPjlw7urBIvAMqUecY6ae1znqsZ0dMBxOuPXHznlnjLjM5b7O7q5WsQMCA9Szbmz6DsuSyCuX0It2osBTN+8P/Fa6BNh3W8AF60M7L8/aUzLfbXVS2LIQKAHHD8CWqvXhLPuTJ03iSwFvgtAK1/J2XJwUP+OzAFrxj6A9LW5ZZgk3R3kRKr0xT/L7hga41rB1qy8Uz+Xr/PTVMNGW+nmU4bPgFchCK0JBK7B12ZcdVVFUEdpaAiKZ prow
Done

@spiffxp spiffxp mentioned this pull request Jun 15, 2021
Open
@ameukam ameukam mentioned this pull request Jun 15, 2021
Closed
@spiffxp spiffxp mentioned this pull request Jun 16, 2021
Merged
@ameukam
Copy link
Member Author

ameukam commented Jun 18, 2021

Ref #2241

@ameukam ameukam mentioned this pull request Jun 28, 2021
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spiffxp spiffxp spiffxp approved these changes

@chaodaiG chaodaiG Awaiting requested review from chaodaiG

@xmudrii xmudrii Awaiting requested review from xmudrii

Assignees

@spiffxp spiffxp

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.22
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ameukam @k8s-ci-robot @spiffxp