Skip to content

Commit

Permalink
variables: limit rekey eval to half the nack timeout
Browse files Browse the repository at this point in the history
In order to limit how much the rekey job can monopolize a scheduler worker, we
limit how long it can run to 1min before stopping work and emitting a new
eval. But this exactly matches the default nack timeout, so it'll fail the eval
rather than getting a chance to emit a new one.

Set the timeout for the rekey eval to half the configured nack timeout.
  • Loading branch information
tgross committed Nov 1, 2022
1 parent 6b2da83 commit c2719d3
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 1 deletion.
3 changes: 3 additions & 0 deletions .changelog/15102.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
variables: Fixed a bug where a long-running rekey could hit the nack timeout
```
2 changes: 1 addition & 1 deletion nomad/core_sched.go
Original file line number Diff line number Diff line change
Expand Up @@ -1053,7 +1053,7 @@ func (c *CoreScheduler) rotateVariables(iter memdb.ResultIterator, eval *structs
//
// Instead, we'll rate limit RPC requests and have a timeout. If we still
// haven't finished the set by the timeout, emit a new eval.
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute)
ctx, cancel := context.WithTimeout(context.Background(), c.srv.GetConfig().EvalNackTimeout/2)
defer cancel()
limiter := rate.NewLimiter(rate.Limit(100), 100)

Expand Down

0 comments on commit c2719d3

Please sign in to comment.