s3 commands that list buckets will use ListObjectsV2

[drocamor]

This PR makes the AWS CLI use ListObjectsV2 when doing operations in the s3 subcommand like ls, cp, and sync. The S3 API docs recommend using ListObjectsV2 instead of V1.

I updated the code and the tests to use V2. It seems like there are some errors around the tests for requester pays features that I may have introduced, but I don't understand where they are coming from. If you have suggestions on how to fix those I can do that. Thanks!

[jamesls]

The errors come from the fact that botocore has a customization to automatically urlencode/decode params. These customizations are configured per operation (in botocore), so we'd need something similar for ListObjectsV2 now that we're not using ListObjects.

https://github.com/boto/botocore/blob/develop/botocore/handlers.py#L881

We'd also need to do a thorough round of testing just to confirm there are no unintended negative side effects.

Let me know if you have any other questions.

[drocamor]

Ah, ok. I will make a PR for botocore too and comment here when the tests are passing on my branches.

Agreed on the thorough testing.

Thanks!

[drocamor]

OK, I had to make some Requester Pays related changes here and I made the encoding changes here: boto/botocore#1552

[kyleknap]

This looks pretty good. I did a run of the integration tests using this PR and the botocore branch and only got one integration test failure, but it just looks like we need to update this test to use ListObjectsV2 instead of ListObjects in the error message assertion. Otherwise, we would have got a clean run through the integration tests which is awesome.

[codecov-io]

Codecov Report

Merging #3549 into develop will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           develop   #3549      +/-   ##
==========================================
+ Coverage    94.87%   94.9%   +0.03%     
==========================================
  Files          174     175       +1     
  Lines        13369   13416      +47     
==========================================
+ Hits         12684   12733      +49     
+ Misses         685     683       -2

Impacted Files	Coverage Δ
awscli/customizations/s3/filegenerator.py	98.91% <100%> (ø)	⬆️
awscli/customizations/s3/utils.py	97.97% <100%> (ø)	⬆️
awscli/customizations/s3/subcommands.py	97.21% <100%> (ø)	⬆️
awscli/handlers.py	100% <0%> (ø)	⬆️
awscli/customizations/sessionmanager.py	100% <0%> (ø)
awscli/compat.py	91.55% <0%> (+0.06%)	⬆️
awscli/clidocs.py	99.14% <0%> (+0.63%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d965bf7...2a7b96f. Read the comment docs.

[kyleknap]

Looks good to me. Thanks! 🚢

[gaul]

Is there some specific reason to prefer ListObjects V2 over V1? Some S3 implementations like S3Proxy do not implement the former.