set node.StatusUpdatedAt in raft

[notnoop]

Fix a case where node.StatusUpdatedAt was manipulated directly in
memory. This ensures that StatusUpdatedAt is set in raft layer.

Also, this ensures that the field is updated when node drain/eligibility is updated too. Previously, StatusUpdatedAt is only updated at next node heartbeat. Interestingly, the drain case was handled previously in 76b913c but it got dropped at some point, but not sure why.

[schmichael]

Is Node.StatusUpdatedAt purely informational or does logic depend on it?

I ask because it seems like during server upgrades old servers will be setting UpdatedAt=0 which will overwrite Node.StatusUpdatedAt times that are >0. The pathological case being if the time is used for determining client liveness and suddenly during an upgrade 2/3 of the clients get marked as lost.

We could guard the writes with a >0 check, but if it's purely informational I don't think it's worth worrying about as the the times will be corrected once all servers are updated and all clients have heartbeated.

[notnoop]

Is Node.StatusUpdatedAt purely informational or does logic depend on it?

It's purely informational, as it's returned in API. The only place that nomad code reads it is when StateStore.UpsertNode emits a (re-)registration event and marks the timestamp [1].

[1]

nomad/nomad/state/state_store.go

Lines 676 to 690 in c00d6a4

	appendNodeEvents(index, node, []*structs.NodeEvent{
	structs.NewNodeEvent().SetSubsystem(structs.NodeEventSubsystemCluster).
	SetMessage(NodeRegisterEventReregistered).
	SetTimestamp(time.Unix(node.StatusUpdatedAt, 0))})
	}
	node.Drain = exist.Drain // Retain the drain mode
	node.SchedulingEligibility = exist.SchedulingEligibility // Retain the eligibility
	node.DrainStrategy = exist.DrainStrategy // Retain the drain strategy
	} else {
	// Because this is the first time the node is being registered, we should
	// also create a node registration event
	nodeEvent := structs.NewNodeEvent().SetSubsystem(structs.NodeEventSubsystemCluster).
	SetMessage(NodeRegisterEventRegistered).
	SetTimestamp(time.Unix(node.StatusUpdatedAt, 0))

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.