Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit minimal time poll.Wait* functions can wait between attempts #2543

Merged
merged 2 commits into from
Dec 19, 2023
Merged

Limit minimal time poll.Wait* functions can wait between attempts #2543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e86e12
Limit minimal time poll.Wait* functions can wait between attempts
hairyhum Dec 14, 2023
0b76735
Merge branch 'master' into poll-min-sleep
mergify[bot] Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 4 additions & 8 deletions pkg/poll/poll.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,10 @@ func WaitWithBackoffWithRetries(ctx context.Context, b backoff.Backoff, numRetri
sleep := b.Duration()
if deadline, ok := ctx.Deadline(); ok {
ctxSleep := time.Until(deadline)
sleep = minDuration(sleep, ctxSleep)
// We want to wait for smaller of backoff sleep and context sleep
Copy link
Contributor

@julio-lopez julio-lopez Dec 15, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: IIUC from the PR description the <-t.C option is (often? always?) selected in the select block below when sleep is negative, is that the case?

If the deadline has passed, it seems that a sensible behavior would be to respect the context deadline by returning an error right away and thus not executing the operation.

It seems that with this change, ctx.Done(): is going to be selected below when the context has expired, independent of how large sleep is, so in that case, why not make sleep positive (strictly greater than 0)? Would that be sufficient?
Or even simpler, why not return with an error when ctxSleep <= 0 ?

Copy link
Contributor Author

@hairyhum hairyhum Dec 15, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially this is a race condition. If ctx already expired but did not perfom the cancel yet, so there's nothing in the ctx.Done() channel. At the same time if we have negative timer, t.C will always have a value in the channel.
Just means that positive sleep (in nanoseconds) doesn't really lets the context to finish in my tests.
We could just return a context.DeadlineExceeded error when ctxSleep <= 0, but then execution will continue in race with termination of the context. While technically correct (deadline did exceed), I don't know which consequences this might have.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other hand additional 10ms of wait doesn't really add that much considering default backoff starts with 100ms and goes up from there (so context deadline in 10s of ms doesn't make much sense)

// but it has to be > 0 to give ctx.Done() a chance
// to return below
sleep = max(min(sleep, ctxSleep), 5*time.Millisecond)
}
t.Reset(sleep)
select {
Expand All @@ -101,10 +104,3 @@ func WaitWithBackoffWithRetries(ctx context.Context, b backoff.Backoff, numRetri
}
}
}

func minDuration(a, b time.Duration) time.Duration {
if a < b {
return a
}
return b
}
26 changes: 26 additions & 0 deletions pkg/poll/poll_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ package poll

import (
"context"
"errors"
"fmt"
"runtime"
"testing"
"time"

Expand Down Expand Up @@ -130,6 +132,30 @@ func (s *PollSuite) TestWaitWithBackoffCancellation(c *C) {
c.Check(err, NotNil)
}

func (s *PollSuite) TestWaitWithRetriesTimeout(c *C) {
// There's a better chance of catching a race condition
// if there is only one thread
maxprocs := runtime.GOMAXPROCS(1)
defer runtime.GOMAXPROCS(maxprocs)

f := func(context.Context) (bool, error) {
return false, errors.New("retryable")
}
errf := func(err error) bool {
return err.Error() == "retryable"
}
ctx := context.Background()
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, time.Millisecond)
defer cancel()

backoff := backoff.Backoff{}
backoff.Min = 2 * time.Millisecond
err := WaitWithBackoffWithRetries(ctx, backoff, 10, errf, f)
c.Check(err, NotNil)
c.Assert(err.Error(), Matches, ".*context deadline exceeded*")
}

func (s *PollSuite) TestWaitWithBackoffBackoff(c *C) {
const numIterations = 10
i := 0
Expand Down