Increase AWS pagination size #2009
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
From measurements, AWS by default has pagination of 100 items per page when listing hosted zone resources. This increases the number of requests required to list all our zones, and pushes a hard constraint on the rate limits. From the experiments, it seems that on the server-side, there is a hard limit of 300 elements per page, as per AWS documentation: https://docs.aws.amazon.com/Route53/latest/APIReference/API_ListResourceRecordSets.html > ListResourceRecordSets returns up to 300 resource record sets at a time in ASCII order, > beginning at a position specified by the name and type elements Hence raising the page size from 100 to 300 items would decrease by 3 the number of requests posted to Route53 We even set a higher limit so we can benefit from a lower number of requests if ever AWS increases the hard limit of 300.
k8s-ci-robot
added
cncf-cla: yes
|
/assign @Raffo |
Raffo
reviewed
Mar 18, 2021
provider/aws/aws.go
Outdated
| // when less items are returned, and still paginate accordingly. | ||
| // As we are using the standard AWS client, this should already be compliant. | ||
| // Hence, ifever AWS decides to raise this limit, we will automatically reduce the pressure on rate limits | ||
| MaxItems: aws.String("1000"), |
There was a problem hiding this comment.
I would prefer having this as 300 as it's the limit now.
I would also love to see a constant declared for that instead of a magic number string.
|
/approve |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Raffo, tjamet The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From measurements, AWS by default has pagination of 100 items per
page when listing hosted zone resources.
This increases the number of requests required to list all our zones,
and pushes a hard constraint on the rate limits.
From the experiments, it seems that on the server-side, there is a hard
limit of 300 elements per page, as per AWS documentation:
https://docs.aws.amazon.com/Route53/latest/APIReference/API_ListResourceRecordSets.html
Hence raising the page size from 100 to 300 items would decrease by 3
the number of requests posted to Route53
We even set a higher limit so we can benefit from a lower number of
requests if ever AWS increases the hard limit of 300.
contributes to improving #1293
Checklist
I didn't find a place where to update the documentation