fix: pass along continuation token correctly

[zkrising]

When using From -> ListObjectsV2, the marker field is mistakenly translated into start_after instead of continuation_token.

S3 is weird in that it has two ways of specifying pagination; start_after lets you start after a given key, whereas marker (continuation_token) is a fully opaque thing that the server then uses to handle pagination.

Please double check this PR: I am reasonably sure it is correct, but S3 is a complex beast and I may be missing something obvious.

[Nugine]

Thank you!

I'm not very sure about this but the AWS doc says that start-after is the same thing with marker.
(AWS docs are not always correct though)

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html

marker
Marker is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. Marker can be any key in the bucket.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

start-after
StartAfter is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. StartAfter can be any key in the bucket.

continuation-token
ContinuationToken indicates to Amazon S3 that the list is being continued on this bucket with a token. ContinuationToken is obfuscated and is not a real key. You can use this ContinuationToken for pagination of the list results.

Our test suites didn't cover this case.
How can we test this behavior? Or Is there any reference implementation?

[zkrising]

Hi Nugine,

I think you're right here, actually. This PR is wrong.

It looks like ListObjects marker is meant to be a real key name, and ListObjects(v1) doesn't support a continuation-token style parameter.

I think the original code might actually be correct, and I've gotten confused myself.