Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iam: separate backoffs, add jitter, and increase for conflicts #10786

Closed
wants to merge 1 commit into from
Closed

iam: separate backoffs, add jitter, and increase for conflicts #10786

wants to merge 1 commit into from

Commits on Dec 22, 2021

  1. iam: separate backoffs, add jitter, and increase for conflicts 

    Within our project, we can have multiple instances of terraform
modifying iam policies, and in many cases these instances are kicked off
at exactly the same time. We're running into errors where we exceed the
backoff max (which in reality is 16 seconds, not 30). Also, Google
reccomends that backoffs contain jitter [1] to prevent clients from
retrying all at once in synchronized waves.

This change (1) separates the 3 distinct backoffs used in the iam policy
read-modify-write cycle, (2) introduces jitter on each retry, and (3)
increases the conflict max backoff to 5 minutes.

[1] https://cloud.google.com/iot/docs/how-tos/exponential-backoff
    @stbenjam
    stbenjam committed Dec 22, 2021
    Configuration menu
    Copy the full SHA
    1ad1a77 View commit details
    Browse the repository at this point in the history