Migrate cli-utils jobs to eks-prow-build-cluster

[rjsadow]

This PR transitions the cli-utils jobs from the default cluster to eks-prow-build-cluster

ref: #29722

[ameukam]

/lgtm
/approve
/hold

@rjsadow Remove the hold when ready. Please keep an eye on those jobs for a few days.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, rjsadow

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes-sigs/cli-utils/OWNERS [ameukam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rjsadow]

/remove-hold

[k8s-ci-robot]

@rjsadow: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key cli-utils-presubmit-master.yaml using file config/jobs/kubernetes-sigs/cli-utils/cli-utils-presubmit-master.yaml

In response to this:

This PR transitions the cli-utils jobs from the default cluster to eks-prow-build-cluster

ref: #29722

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[karlkfi]

This change (probably due to added memory limit) seems to be causing the cli-utils stress tests to fail. Should I just make a PR to increase the memory limit? Where are the node sizes documented?

[sdowell]

cc @rjsadow @ameukam

It appears this change broke the cli-utils stress tests (as @karlkfi mentioned). Could you take a look?

[rjsadow]

@sdowell sorry, i missed that. Yes, I'll dig into it tomorrow morning.

[rjsadow]

@sdowell can you please link some current examples? It looks like there were issues a few days ago but it looks like #30110 fixed it. I'm not seeing any current resource issues in grafana and the current job history seems ok. The couple failures that are there seem to be from tests failing.

[sdowell]

@rjsadow

The common failure mode that I'm seeing is Job execution failed: Pod got deleted unexpectedly. I'm not sure how to diagnose why the Pod is getting deleted without having access to the build cluster myself.

Here's an example where the tests passed but the job failed before posting a status (with the above error): https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cli-utils/622/cli-utils-presubmit-master-stress/1682149784174989312

[rjsadow]

This might be more related to kubernetes/k8s.io#5473. I too have seen those Job execution failed: Pod got deleted unexpectedly off and on but also don't have access to the build cluster myself. We can, if you want, move this job to the GCP cluster for now.

[sdowell]

@rjsadow That would be great if that is okay with you. We're currently blocked on submitting PRs and would prefer not to disable the presubmit if possible.

I appreciate it, I understand this probably makes your job harder with the migration effort. We can move back to the eks cluster once the Node health issue is better understood - does that work for you?