Skip to content

Commit

Permalink
test: deflake node drain intergration test (#18171)
Browse files Browse the repository at this point in the history
The `TestDrainer_AllTypes_NoDeadline` test has been flaky. It looks like this
might be because the final update of batch allocations to complete is improperly
updating the state store directly rather than by RPC. If the service jobs have
restarted in the meantime, the `allocClientStateSimulator` will have updated the
index on the allocations table and that will prevent the drainer from
unblocking (and being marked complete) when the batch jobs are written with an
earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally
would be in real code).
  • Loading branch information
tgross committed Aug 14, 2023
1 parent e3986a8 commit a7f43a5
Showing 1 changed file with 10 additions and 6 deletions.
16 changes: 10 additions & 6 deletions nomad/drainer_int_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ import (
)

// allocClientStateSimulator simulates the updates in state from the
// client. allocations that are new on the server get marked with healthy
// deployments, and allocations that are DesiredStatus=stop on the server get
// updates with terminal client status.
// client. service allocations that are new on the server get marked with
// healthy deployments, and service allocations that are DesiredStatus=stop on
// the server get updates with terminal client status.
func allocClientStateSimulator(t *testing.T, errCh chan<- error, ctx context.Context,
srv *Server, nodeID string, logger log.Logger) {

Expand Down Expand Up @@ -529,9 +529,13 @@ func TestDrainer_AllTypes_NoDeadline(t *testing.T) {
new.ClientStatus = structs.AllocClientStatusComplete
updates = append(updates, new)
}
index, _ := store.LatestIndex()
index++
must.NoError(t, store.UpdateAllocsFromClient(structs.MsgTypeTestSetup, index, updates))

batchDoneReq := &structs.AllocUpdateRequest{
Alloc: updates,
WriteRequest: structs.WriteRequest{Region: "global"},
}
err = msgpackrpc.CallWithCodec(codec, "Node.UpdateAlloc", batchDoneReq, &resp)
must.NoError(t, err)

// Wait for the service allocations to be replaced
waitForPlacedAllocs(t, store, n2.ID, 3)
Expand Down

0 comments on commit a7f43a5

Please sign in to comment.