Use debug logging for scheduler internals

[preetapan]

We currently log an error if preemption is unable to find a suitable set of
allocations to preempt. This commit changes that to debug level since not finding
preemptable allocations is not an error condition.

[notnoop]

lgtm.

Looks like we can remove these logs as well? I see that we've been mostly logging actual errors in scheduler - e.g. we don't log when a node is over-utilized or not meeting constraints (these get propagated through ctx fields:

$ git rev-parse HEAD
7cda14098104adc56386c13ee2ced34952f02cf6
$ git grep iter.ctx.Logger
scheduler/feasible.go:          iter.ctx.Logger().Named("distinct_hosts").Error("failed to get proposed allocations", "error", err)
scheduler/rank.go:                      iter.ctx.Logger().Named("binpack").Error("failed retrieving proposed allocations", "error", err)
scheduler/rank.go:                                      iter.ctx.Logger().Named("binpack").Error("preemption not possible ", "network_resource", ask)
scheduler/rank.go:                                      iter.ctx.Logger().Named("binpack").Error("unexpected error, unable to create network offer after considering preemption", "error", err)
scheduler/rank.go:                                              iter.ctx.Logger().Named("binpack").Error("preemption not possible ", "network_resource", ask)
scheduler/rank.go:                                              iter.ctx.Logger().Named("binpack").Error("unexpected error, unable to create network offer after considering preemption", "error", err)
scheduler/rank.go:                                              iter.ctx.Logger().Named("binpack").Error("preemption not possible", "requested_device", req)
scheduler/rank.go:                                              iter.ctx.Logger().Named("binpack").Error("unexpected error, unable to create device offer after considering preemption", "error", err)
scheduler/rank.go:                      iter.ctx.Logger().Named("job_anti_affinity").Error("failed retrieving proposed allocations", "error", err)
scheduler/spread.go:                            iter.ctx.Logger().Named("spread").Warn("error building spread attributes for task group", "task_group", tgName, "error", errorMsg)

[preetapan]

@notnoop The failed retrieving proposed allocations should stay as Error because that can only happen if there's a state store error, and that's worth being noisy about.

I'll downgrade the other errors to Debug.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.