Add option to start listing at a particular key

[wjones127]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In an object store, we might have a bunch of sequential files being written:

0000001.json
0000002.json
...
0001000.json

We'd like to be able to query for all the "new" files starting at a certain point, skipping all the earlier files.

S3 has a start-after parameter we can use for this. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestParameters
TDB on other systems.

Describe the solution you'd like

Not sure the best way to add the parameter. Does it belong in a new method? Should we introduce a more complex "ListCallBuilder" API?

let list_stream = object_store
     .build_list_call(Some(&prefix))
     .with_start_key("0000999.json")
     .await
     .expect("Error listing files");

Describe alternatives you've considered

Not sure if there is an easier way to do it.

Additional context

[tustvold]

Seems like a useful feature, I'll have a think about what an API for this could look like. I wonder if we should just add a list_opts method that acts as a superset of list_with_delimiter and list 🤔 Similar perhaps to the proposal in #2241

What is support for this like in other object stores, I presume they support it, but I've learnt never to assume anything when it comes to object stores 😅

[wjones127]

It looks like support isn't that wide:

• S3: has start-after (exclusive?)
• GCS: has a startOffset (inclusive)
• Azure Blob Store not supported.
• I don't see any obvious API for local filesystems.

So this is mostly providing a useful optimization for S3 and GCS. There can be a default implementation that just throws out earlier entries. Also, for consistency between S3 and GCS, we would have to make the lower bound exclusive, since that seems to be the S3 behavior.

[rtyler]

Speaking selfishly supporting S3-based optimizations goes a long way given its dominance in the market. Most data workloads I see are on AWS or GCP, so that's great you found a compatible API in GCS @wjones127

[tustvold]

I've created #3973 if we like the interface I can flesh it out

[JHibbard]

The Azure Data Lake Storage Gen2 REST API has endpoints for filesystem list and path list that looks interesting, but the documentation is vague. ADLS-G2 is hierarchical in nature... so some offset/skipping API is likely available somewhere.

filesystem list:

• prefix: Filters results to filesystems within the specified prefix.

path list:

• directory: Filters results to paths within the specified directory. An error occurs if the directory does not exist.