fsm: one-time token expiration should be deterministic

[tgross]

Fixes #13725 for one-time tokens.

When applying a raft log to expire ACL tokens, we need to use a
timestamp provided by the leader so that the result is deterministic
across servers.

[schmichael]

Doesn't this need to get set in CoreScheduler.expiredOneTimeTokenGC? I don't see where this gets set, and core_sched.go is where the other GC operations compute their threshold based on the timetable/local clock before submitting the deletions via raft.

[tgross]

It should be set on the ACL.ExpireOneTimeTokens RPC, which I missed b/c the tests passed.

Most of the core jobs use the timetable block specifically because they don't submit the deletions via raft, but via the RPC. While the leader creates the GC eval, it can be processed on any worker (ref worker.go#L565-L569).

So when I wrote the expire OTT job originally, I hit the same "quiet clusters will have wonky timestamps" problem @jrasell discovered (and I wanted to make transaction logic simple). So I wanted to use the wall clock on the leader to get a more accurate expiration. But I messed that up by not having the timestamp set in the leader's RPC, but in the FSM instead. So we should be setting that on the leader's handler for the RPC and we'll be all set.

[tgross]

Fixed in 9d8138c

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.