Skip to content

Commit

Permalink
Improve github pull request call retries (runatlantis#1810)
Browse files Browse the repository at this point in the history
* Improve github pull request call retries

Retry with fixed 1 second backoff up to 3 retries was added by runatlantis#1131 to
address runatlantis#1019, but the issue continued to show up (runatlantis#1453).

Increase max attempts to 5 and use exponential backoff for a maximum
total retry time of (2^n - n - 1) seconds, which is roughly 30 seconds
for current max attempts n = 5.

Also move the sleep to the top of the loop so that we never sleep
without sending the request again on the last iteration.

* Fix style with gofmt -s
  • Loading branch information
aristocrates authored and krrrr38 committed Dec 16, 2022
1 parent d6b54ec commit 0510b7b
Showing 1 changed file with 8 additions and 5 deletions.
13 changes: 8 additions & 5 deletions server/events/vcs/github_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,10 +292,14 @@ func (g *GithubClient) GetPullRequest(repo models.Repo, num int) (*github.PullRe

// GitHub has started to return 404's here (#1019) even after they send the webhook.
// They've got some eventual consistency issues going on so we're just going
// to retry up to 3 times with a 1s sleep.
numRetries := 3
retryDelay := 1 * time.Second
for i := 0; i < numRetries; i++ {
// to attempt up to 5 times with exponential backoff.
maxAttempts := 5
attemptDelay := 0 * time.Second
for i := 0; i < maxAttempts; i++ {
// First don't sleep, then sleep 1, 3, 7, etc.
time.Sleep(attemptDelay)
attemptDelay = 2*attemptDelay + 1*time.Second

pull, _, err = g.client.PullRequests.Get(g.ctx, repo.Owner, repo.Name, num)
if err == nil {
return pull, nil
Expand All @@ -304,7 +308,6 @@ func (g *GithubClient) GetPullRequest(repo models.Repo, num int) (*github.PullRe
if !ok || ghErr.Response.StatusCode != 404 {
return pull, err
}
time.Sleep(retryDelay)
}
return pull, err
}
Expand Down

0 comments on commit 0510b7b

Please sign in to comment.