cloud_storage: more efficient timequery

[jcsp]

Time queries currently scan from the start of a segment, even when doing chunked reads. This makes them an order of magnitude more expensive in some cases, compared with if they just read a single chunk.

This can be fixed by using the timestamps in the remote index that were added for 23.2.

However, the cost of doing a timequery will still be one chunk promotion per partition, which is very expensive if you've got e.g. 1000 partitions in a topic: 16GiB of downloads + disk writes to serve one ListOffsets.

It may also make sense to override the chunk size to something smaller when the reader is created for a time query, and/or to implement a "fuzzy mode" for time queries to tiered storage, and just give the caller the nearest indexed offset rather than a precise answer: many applications will not care if they see a few messages before the timestamp they requested, as long as they see all the timestamps after it. This is already the behavior for compressed batches, where we return the offset of the start of the batch rather than the exact record result.

[dotnwat]

@VladLazar @Lazin is this different than #13125? In that issue one of the observed behaviors is "Observe high delays or missing timestamps" which sounds pretty similar to "more efficient timequery".

[VladLazar]

@VladLazar @Lazin is this different than #13125? In that issue one of the observed behaviors is "Observe high delays or missing timestamps" which sounds pretty similar to "more efficient timequery".

This issues (11801) was more generic than the one Evgeny created. The goal here was to hydrate less data in the general case. 13125 is more about what happens when the timestamps are bad (either due to config batches or user provided ts skew), so it's a more specific case. We had the exact same issues when improving the local storage timequery a while back.