…he last index processed so that stale data does not impact the running allocations.

Testing for absence is a rather difficult task without hooks into the code to assert that given code paths have been hit. In this test we simply make sure that a transaction with a lower raft index does not make it to the client and depend on the timeout to make sure that this is the case. If the invariant is broken we expect the test to become either flaky or entirely broken.

I acknowledge there may be a better way to test here, but lacking context this seems to be a minimal attempt to ensure that a similar issue doesn't creep back in. Feedback welcome.

1. error is ignored in WaitForResult
2. existing timeouts in blocking queries cannot be overridden and would make the tests always pass
3. the scheduler seems to ignore raft transactions with lower ids

As such, we would need more infrastructure to write such tests. Do you have an idea is there is a precedent I can use here no to reinvent the wheel? In general we need:

1. To reduce the raft index of scheduler (purge the state or reassign the state: This can be done with regular assignment into a newly created state
2. Induce a timeout on the blocking query to ensure that the query result is passed to client -> This I see no easy way of achieving

Use identifier-style keys for structured logs

Co-authored-by: Michael Schurter <michael.schurter@gmail.com>