@vercel/blob API - list() - Filtering (before pagination) and possibly sorting too

[ADTC]

In Blob API list() function, please add the ability to filter while maintaining pagination.

list({
  ...options
  filter: null | {
    // one or more of:
    sizeLessThan: number,
    sizeMoreThan: number,
    uploadedLaterThan: Date,
    uploadedEarlierThan: Date,
    pathname: string | Regex,
    contentType: string | string[] | Regex,
    contentDisposition: string | string[] | Regex,
    url: string | Regex,
    downloadUrl: string | Regex,
    cacheControl: string | string[] | Regex,
  }
})

Otherwise the workaround is quite clunky and doesn't work well with pagination. Suppose we need to show only images from a collection of media paginated 100 at a time. Maybe page 1 has only 67 images. Then the list() function will fetch 100 media and the UI displays 67 images on page 1. Then it fetches another 100 images on page 2, only to display maybe 33 images. Maybe page 3 has no images out of 100 media (misleading the user), then page 4 has 87 images out of 100 media.

I hope you get what I mean, and why "filtered, then paginated" results will be better.

I saw that sorting was refused citing technical limitations in #533 which has me worried that this request for filters could also be refused. In the issue, it was said that the:

underlying object storage technology we use doesn't have this feature (and most of them don't: aws s3 doesn't have it)

That is unfortunate, but I believe it could be overcome by having an in-between:

Client-side       Vercel Blob        Vercel Blob            Filtered,
list() call  --->  gets full  --->   applies sort  --->    sorted and
  issued          object list      filter/paginate          paginated
                from underlying           ^              results returned
                 storage tech.     (the in-between)         to client.

If Vercel will not provide this "in-between" natively (which could be hyperfast because it runs internally with the lowest latency), we'll have to build our own version of the "in-between" where sorting and filtering is desired (which wouldn't be as fast, because not only do we have to call list() to get complete blob object lists (file lists), we then also have to call head() each and every one of them to get the additional information (which is another gripe I have with list() because it's missing critical information which can only be obtained by a separate head() call). This has increased the development cost, time cost and network resource cost.

I hope you get the rationale for why it's better to build it in-house and expose it as filter and sort options in list() function.