Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Use list_objects_v2 to list objects #155

Closed
pjbull opened this issue Aug 2, 2021 · 1 comment
Closed

S3: Use list_objects_v2 to list objects #155

pjbull opened this issue Aug 2, 2021 · 1 comment
Labels

Comments

@pjbull
Copy link
Member

pjbull commented Aug 2, 2021

The v1 of list_objects can be the cause of some consistency problems. See this answer for context:
https://stackoverflow.com/a/67412931/1692709

We currently use list_objects for non recursive cases:

paginator = self.client.get_paginator("list_objects")

We use the bucket filter in recursive cases:

for o in bucket.objects.filter(Prefix=prefix):

We should replace both code paths with self.client.get_paginator('list_objects_v2'):

Here's the boto3 docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2

@pjbull pjbull added good first issue Good for newcomers S3 labels Aug 2, 2021
@pjbull pjbull removed the good first issue Good for newcomers label Mar 2, 2022
@pjbull
Copy link
Member Author

pjbull commented Dec 19, 2022

Closed by #302

@pjbull pjbull closed this as completed Dec 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant