Add scheduler version enforcement

[dadgar]

@armon For review

[armon]

The major feedback is that the worker needs to detect this and do a very long or permanent back off until the leader changes. While the same leader is running, we will never be able to dequeue work.

[armon]

Something about the uint16 feels dirty...

[dadgar]

It gives us a lot of versions while keeping it low overhead. We will never pass 65536 version bumps.

[armon]

The worker should probably not sit in a hot loop of calling dequeue on this error, e.g. the failure is basically semi-permanent.

[dadgar]

So I was thinking we want it to spam the logs so the operator knows they are doing something bad. On the error we start backing off:

nomad/nomad/worker.go

Lines 146 to 163 in 962f4d4

	REQ:
	// Check if we are paused
	w.checkPaused()
	// Make a blocking RPC
	start := time.Now()
	err := w.srv.RPC("Eval.Dequeue", &req, &resp)
	metrics.MeasureSince([]string{"nomad", "worker", "dequeue_eval"}, start)
	if err != nil {
	if time.Since(w.start) > dequeueErrGrace && !w.srv.IsShutdown() {
	w.logger.Printf("[ERR] worker: failed to dequeue evaluation: %v", err)
	}
	if w.backoffErr(backoffBaselineSlow, backoffLimitSlow) {
	return nil, "", true
	}
	goto REQ
	}
	w.backoffReset()

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.