Better metric name for nomad.broker.total_blocked

[preetapan]

The metric broker.total_blocked is poorly named. It is actually counting the number of pending evals for the same job in Nomad's eval broker.

Proposing renaming this (better name TBD) with a deprecation path. Plan is to log the metric with two names in a point release. The old name will be removed in a subsequent major release with deprecation notes.

[tgross]

This came up again recently while debugging #13407 and having a metric named nomad.broker.total_blocked added significant delay because this metric is easily confused with the evaluation status "blocked" from structs.go#L10668-L10674

const (
	EvalStatusBlocked   = "blocked"
	EvalStatusPending   = "pending"
	EvalStatusComplete  = "complete"
	EvalStatusFailed    = "failed"
	EvalStatusCancelled = "canceled"
)

The eval broker's internal state is never reflected on the Evaluation.Status field, so having the metric name so similar is the point of confusion and should be avoided on a change. Arguably "blocked" is a great word here and it's the EvalStatusBlocked that would be better to change but that's written in countless raft stores so it's not terribly practical to rename. 😬 I'd like to push this forward, so let's bikeshed the new name a bit...

The current stats are (ref eval_broker.go#L836-L879 and the Metrics Reference docs):

metric	description	type
nomad.nomad.broker.total_blocked	Evaluations that are blocked until an existing evaluation for the same job completes	count
nomad.nomad.broker.total_ready	Number of evaluations ready to be processed	count
nomad.nomad.broker.total_unacked	Evaluations dispatched for processing but incomplete	count
nomad.nomad.broker.total_waiting	Count of evals waiting to be enqueued	count
nomad.nomad.broker.<type>_unacked	Count of unacknowledged system evals	count
nomad.nomad.broker.<type>_ready	Count of evals in the ready state	count
nomad.nomad.broker.eval_waiting	Time elapsed with evaluation waiting to be enqueued	time

ready is the term used for the evals that have been enqueued, whereas blocked (here) is "waiting on a previous eval for this same job before it can be enqueued". But we've already used "waiting" for those evals that are on a delay (the metrics reference doc is worded ambiguously here so I'll fix up that as well). So calling these "delayed" would swap the meaning and the name. Ideally we could change both metrics but repurposing a metric name doesn't have a graceful upgrade path!

Some options:

• nomad.nomad.broker.total_not_ready: mirror of "ready"
• nomad.nomad.broker.total_unready: another mirror of "ready"
• nomad.nomad.broker.total_unenqueued: awkwardly named but better than "pending" which is another eval status
• nomad.nomad.broker.total_congested: @tgross rummages around in the thesaurus for antonyms of "enqueued"