Migrate node-problem-detector jobs to eks-prow-build-cluster

[Vyom-Yadav]

This PR transitions the node-problem-detector jobs from the default cluster to eks-prow-build-cluster

ref: #29722

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Vyom-Yadav / name: Vyom Yadav (d3d851c)

[Vyom-Yadav]

CPU/Memory requests/limits would be added if pull-test-infra-unit-test fails.

[marquiz]

You need to add resource (cpu and mem) requests and limits to the jobs.

Why not update the periodic jobs, too (in kubernetes/node-problem-detector/node-problem-detector-ci.yaml)

[Vyom-Yadav]

You need to add resource (cpu and mem) requests and limits to the jobs.

I have contacted node problem selector folks regarding the limits. Haven't received any response yet, any suggestions?

https://kubernetes.slack.com/archives/CJA25LM6D/p1686373854184919?thread_ts=1686373854.184919&cid=CJA25LM6D

Why not update the periodic jobs, too (in kubernetes/node-problem-detector/node-problem-detector-ci.yaml)

Sure.

[rjsadow]

I have contacted node problem selector folks regarding the limits. Haven't received any response yet, any suggestions?

Since you've not heard anything back from them yet. I would recommend starting with 2 cpu and 4GB memory. We can monitor their resource usage through Kabana (https://monitoring-eks.prow.k8s.io/d/96Q8oOOZk/builds?from=now-24h&to=now) after they're merged and fine tune the capacity requirements later.

[SergeyKanzhelev]

/assign @mmiranda96

[mmiranda96]

Please specify limits. As @rjsadow mentioned, 2 CPU and 4G memory per job should be a good starting point.

Thanks for your work!

/lgtm

[Vyom-Yadav]

I kept the req., limit the same. Prow configures request as:

resources:                                                                   
  requests:                                                                  
    cpu: 250m                                                                
    memory: 1Gi        

Ping me if you want me to change request to prow's request.

[rjsadow]

/lgtm

[Vyom-Yadav]

@marquiz ping

[marquiz]

/lgtm

ping @mmiranda96 @SergeyKanzhelev

[mmiranda96]

/lgtm

[dims]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, marquiz, Vyom-Yadav

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/node-problem-detector/OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@Vyom-Yadav: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key node-problem-detector-ci.yaml using file config/jobs/kubernetes/node-problem-detector/node-problem-detector-ci.yaml
• key node-problem-detector-presubmits.yaml using file config/jobs/kubernetes/node-problem-detector/node-problem-detector-presubmits.yaml

In response to this:

This PR transitions the node-problem-detector jobs from the default cluster to eks-prow-build-cluster

ref: #29722

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-npd-e2e-kubernetes-gce-gci-custom-flags?buildId=1675443561614544896

2023/07/01 14:01:41 main.go:328: Something went wrong: failed to prepare test environment: --provider=gce boskos failed to acquire project: resources not found
Traceback (most recent call last):
  File "/workspace/scenarios/kubernetes_e2e.py", line 723, in <module>
    main(parse_args())
  File "/workspace/scenarios/kubernetes_e2e.py", line 569, in main
    mode.start(runner_args)
  File "/workspace/scenarios/kubernetes_e2e.py", line 228, in start
    check_env(env, self.command, *args)
  File "/workspace/scenarios/kubernetes_e2e.py", line 111, in check_env
    subprocess.check_call(cmd, env=env)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('kubetest', '--dump=/logs/artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--provider=gce', '--cluster=e2e-a52e116b34-75293', '--gcp-network=e2e-a52e116b34-75293', '--extract=ci/latest', '--gcp-node-image=gci', '--gcp-zone=us-west1-b', '--ginkgo-parallel=30', '--test_args=--ginkgo.focus=NodeProblemDetector', '--timeout=60m')' returned non-zero exit status 1.
+ EXIT_VALUE=1
+ set +o xtrace

Some test failed after this PR merged.

[Vyom-Yadav]

@pacoxu From the original issue:

NOTE: if you see any entries under label that says gce skip this job and go to the next time as this is not ready to be moved yet.

The test name contains gce but it's not present in labels, could this be the problem? And from the logs too, I think this belongs to gcp and not eks.

[pacoxu]

I opened #29998. I am not quite sure about that.