ObjectStore bulk delete

[thinkharderdev]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Deleting a lot of objects from an ObjectStore can be expensive if you have to do a separate API call for each object. In scenarios where you have to delete a lot of objects in a batch (eg a compaction process) this can be very annoying and resource intensive.

It would be helpful to add a bulk_delete method to ObjectStore that can take advantage of underlying APIs that allow for bulk deletion (eg S3 DeleteObjects API).

The default implementation can just call delete on each path serially if the underlying ObjectStore does not have an API for bulk deletion or we can throw an unimplemented error instead.

Describe the solution you'd like

pub struct BulkDeleteResult {
/// Count of objects successfully deleted
deleted: usize,
/// Errors for individual objects which could not be deleted for some reason,
errors: Vec<(Path,Error)>,
}

Something like

pub trait ObjectStore {
  
  async fn bulk_delete(&self, locations: &[Path]) -> Result<BulkDeleteResult>;

}

Describe alternatives you've considered

Do nothing

Additional context

[tustvold]

We already support this https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#method.delete_stream

[thinkharderdev]

Ugh, not sure how I missed that :(