Verbose logging of cache and queues

[alculquicondor]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Produces a dump of the cache into the logs at log level 5.

A dump of the cache and queues can be logged on demand using SIGUSR2, regardless of the log level.

Refactored some existing dump methods used for testing to use slices for better performance and simpler formatting.

Which issue(s) this PR fixes:

Fixes #1438 #1437

Special notes for your reviewer:

Does this PR introduce a user-facing change?

At log level 6, the usage of ClusterQueues and cohorts is included in logs.

The status of the internal cache and queues is also logged on demand when a SIGUSR2 is sent to kueue, regardless of the log level.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	aa40831
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65942567ceceed00080e07f3

[tenzen-y]

Generally, lgtm

[tenzen-y]

IIRC, our max log level is 6. So is it better to use V(6) here and Line 37 as well?

kueue/pkg/controller/jobframework/reconciler.go

Line 335 in 2fcdb56

log.V(6).Info("The job is no longer active, clear the workloads admission")

[alculquicondor]

Done

[tenzen-y]

Thanks.
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [alculquicondor,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 755a673fe5d9f86f7e1230d38cd6867928f5419f

[alculquicondor]

Btw, this is how the dump looks:

{"level":"info","ts":"2024-01-02T19:00:43.567480487Z","logger":"dumper","caller":"debugger/debugger.go:59","msg":"Started dump"}
{"level":"info","ts":"2024-01-02T19:00:43.56753785Z","logger":"dumper","caller":"cache/snapshot.go:67","msg":"Found ClusterQueue","clusterQueue":{"name":"cluster-queue"},"cohort":"<none>","resourceGroups":[{"CoveredResources":{"cpu":{},"memory":{}},"Flavors":[{"Name":"default-flavor","Resources":{"cpu":{"Nominal":9000,"BorrowingLimit":null},"memory":{"Nominal":38654705664,"BorrowingLimit":null}}}],"LabelKeys":null}],"usage":{"default-flavor":{"cpu":3000,"memory":629145600}},"workloads":["default/job-sample-job-rphvr-2168d"]}
{"level":"info","ts":"2024-01-02T19:00:43.567775516Z","logger":"dumper","caller":"queue/dumper.go:32","msg":"Found pending and inadmissible workloads in ClusterQueue","clusterQueue":{"name":"cluster-queue"},"pending":[],"inadmissible":[]}
{"level":"info","ts":"2024-01-02T19:00:43.567789748Z","logger":"dumper","caller":"debugger/debugger.go:63","msg":"Ended dump"}

I'm preparing another PR with instructions on how to produce it.

[tenzen-y]

Btw, this is how the dump looks:

{"level":"info","ts":"2024-01-02T19:00:43.567480487Z","logger":"dumper","caller":"debugger/debugger.go:59","msg":"Started dump"}
{"level":"info","ts":"2024-01-02T19:00:43.56753785Z","logger":"dumper","caller":"cache/snapshot.go:67","msg":"Found ClusterQueue","clusterQueue":{"name":"cluster-queue"},"cohort":"<none>","resourceGroups":[{"CoveredResources":{"cpu":{},"memory":{}},"Flavors":[{"Name":"default-flavor","Resources":{"cpu":{"Nominal":9000,"BorrowingLimit":null},"memory":{"Nominal":38654705664,"BorrowingLimit":null}}}],"LabelKeys":null}],"usage":{"default-flavor":{"cpu":3000,"memory":629145600}},"workloads":["default/job-sample-job-rphvr-2168d"]}
{"level":"info","ts":"2024-01-02T19:00:43.567775516Z","logger":"dumper","caller":"queue/dumper.go:32","msg":"Found pending and inadmissible workloads in ClusterQueue","clusterQueue":{"name":"cluster-queue"},"pending":[],"inadmissible":[]}
{"level":"info","ts":"2024-01-02T19:00:43.567789748Z","logger":"dumper","caller":"debugger/debugger.go:63","msg":"Ended dump"}

I'm preparing another PR with instructions on how to produce it.

It's great to see. Thanks.