Configure Retryer and Max Retry to avoid throttling error #3578
Conversation
Codecov ReportBase: 37.13% // Head: 37.07% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #3578 +/- ##
==========================================
- Coverage 37.13% 37.07% -0.07%
==========================================
Files 136 136
Lines 16643 16703 +60
==========================================
+ Hits 6181 6192 +11
- Misses 9425 9463 +38
- Partials 1037 1048 +11
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
Twsouza
force-pushed
the
aws-retryer-throttling
branch
from
e7fe2ac to
371937c
Compare
There was a problem hiding this comment.
It looks good but i don't think we need the wrapper around the DescribeStack operation.
We can configure the retry logic on all existing session objects.
To make it a bit more DRY you could have a helper function, like
helpers.NewSession() so you don't need to pass the same config over and over
| @@ -379,14 +379,8 @@ func RackSync(rack sdk.Interface, c *stdcli.Context) error { | |||
|
|
|||
| if c.String("name") != "" { | |||
| rname = c.String("name") | |||
| s, err := ss.NewSession(&aws.Config{}) | |||
There was a problem hiding this comment.
Can't we just configure the Retry/Throttling logic in this session instead?
pkg/helpers/aws.go
Outdated
| @@ -71,12 +72,12 @@ func AwsErrorCode(err error) string { | |||
| return "" | |||
| } | |||
|
|
|||
| func CloudformationDescribe(cf cloudformationiface.CloudFormationAPI, stack string) (*cloudformation.Stack, error) { | |||
| func CloudformationDescribe(stack string) (*cloudformation.Stack, error) { | |||
There was a problem hiding this comment.
Same here, looks like cf cloudformationiface.CloudFormationAPI is the provider.Cloudformation struct. We can change the session there.
https://github.com/convox/rack/blob/aws-retryer-throttling/provider/kaws/system.go#L271
https://github.com/convox/rack/blob/master/provider/kaws/kaws.go#L159
It's a good practice to have a retry/throtling logic so it's ok that we configure for the other services too. (in the initializeAwsServices function)
provider/aws/apps.go
Outdated
| @@ -400,7 +401,7 @@ func (p *Provider) cleanup(app *structs.App) error { | |||
| shouldRetry := true | |||
|
|
|||
| for i := 0; i < 60; i++ { | |||
| res, err := p.cloudformation().DescribeStacks(&cloudformation.DescribeStacksInput{ | |||
There was a problem hiding this comment.
Same here, configure session in p;cloudformation().
provider/aws/system.go
Outdated
| @@ -286,7 +287,7 @@ func (p *Provider) SystemInstall(w io.Writer, opts structs.SystemInstallOptions) | |||
| bar.Finish() | |||
| } | |||
|
|
|||
| dres, err := cf.DescribeStacks(&cloudformation.DescribeStacksInput{ | |||
| dres, err := pkgcf.DescribeStacks(&cloudformation.DescribeStacksInput{ | |||
There was a problem hiding this comment.
same here, use session config from L255
provider/aws/system.go
Outdated
| @@ -455,7 +456,9 @@ func (p *Provider) SystemUninstall(name string, w io.Writer, opts structs.System | |||
|
|
|||
| cf := cloudformation.New(session.New(&aws.Config{})) | |||
provider/aws/system.go
Outdated
| @@ -663,7 +666,7 @@ func cloudformationProgress(stack, token string, template []byte, w io.Writer) e | |||
| } | |||
| } | |||
|
|
|||
| dres, err := cf.DescribeStacks(&cloudformation.DescribeStacksInput{ | |||
| dres, err := pkgcf.DescribeStacks(&cloudformation.DescribeStacksInput{ | |||
| @@ -690,14 +693,12 @@ func cloudformationProgress(stack, token string, template []byte, w io.Writer) e | |||
| } | |||
|
|
|||
| func rackDependencies(name string) ([]string, error) { | |||
| cf := cloudformation.New(session.New(&aws.Config{})) | |||
Twsouza
force-pushed
the
aws-retryer-throttling
branch
from
371937c to
850994f
Compare
Configure and use the session on each new session
Twsouza
force-pushed
the
aws-retryer-throttling
branch
from
850994f to
335f39f
Compare
Use MaxRetry on aws.Config and set custom timers on DefaultRetryer. Also, updates Go to 1.17 as `golang.org/x/sys` only allows 2 versions behind the latest one. Go 1.19 was released last month.
Use MaxRetry on aws.Config and set custom timers on DefaultRetryer. Also, updates Go to 1.17 as `golang.org/x/sys` only allows 2 versions behind the latest one. Go 1.19 was released last month.
Use MaxRetry on aws.Config and set custom timers on DefaultRetryer. Also, updates Go to 1.17 as `golang.org/x/sys` only allows 2 versions behind the latest one. Go 1.19 was released last month.
Use MaxRetry on aws.Config and set custom timers on DefaultRetryer. Also, updates Go to 1.17 as `golang.org/x/sys` only allows 2 versions behind the latest one. Go 1.19 was released last month.
Use MaxRetry on aws.Config and set custom timers on DefaultRetryer.
Also, updates Go to 1.17 as
golang.org/x/sysonly allows 2 versions behind the latest one. Go 1.19 was released last month.