scheduler: detect and log unexpected scheduling collisions

[lgfa29]

#9506 and others have reported sporadically eval plan failures on job submission.

The root of the problem is the PlanApplier rejecting plans from scheduler workers due to port conflicts. Since the scheduler did not observe the conflict it will likely make the same plan again and the job will never finish scheduling. We need more data on why the port conflict happens in the first place.

This PR doesn't solve the problem, as there are multiple causes and not all are known, but brings the worker scheduler logic closer to the logic used by the PlanApplier during scheduling.

More specifically, one discrepancy we noticed is the BinPackIterator not ignoring potential port collisions, whereas the PlanApplier would not and reject the plan if a collision is detected.

This flow violates the assumption made by AllocsFit:

// If the netIdx is provided, it is assumed that the client has already
// ensured there are no collisions.

The PlanApplier passes a nil netIdx to AllocsFit, while the scheduler worker passes a non-nil value, but also doesn't check for conflicts.

We believe the initial logic behaved like this because, when the iterator is evaluating a node, the netIdx is initialized with its current state, which should never have a collision, but this may not always hold true (corrupt state, incorrect copying values, mutating the state store directly, incorrect upgrade paths etc.).

Since we're not sure what could be causing this issue, this PR also adds additional logging when this problem is detected. To aid debugging, extensive data is collected (full node struct, list of allocations, NetworkIndex etc.). Since this can result in a lot of data, these events are sent back to the server to be centralized and deduplicated by node ID to avoid polluting the agent log.

After these changes, any unexpected port collision will result in the node being skipped, which will allow for evaluations to progress into a healthy node.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.