cloud_storage: add timestamps to remote segment index #11802
Conversation
jcsp
added
kind/enhance
jcsp
force-pushed
the
remote-index-timestamps
branch
4 times, most recently
from
54fe19a to
c3d1676
Compare
|
/cdt tests/rptest/scale_tests/tiered_storage* |
| int64_t base_time; | ||
| int64_t last_time; | ||
| std::vector<int64_t> time_write_buf; | ||
| iobuf time_index; |
There was a problem hiding this comment.
do we need to care for version<1> ->version<2> compat?
There was a problem hiding this comment.
We're using serde's default read/write impls, & I believe serde::envelope's default behavior is that if it just runs out of data on decode, it will not populate those later fields (so these values remain at their defaults).
There was a problem hiding this comment.
(I have also updated the base_time and last_time to have explicit initialization -- previously one would have had to use the vector or iobuf to check for whether timestamp fields are set, because the ints would have had undefined contents)
| auto actual = iobuf_to_bytes(ix.to_iobuf()); | ||
| auto ix_buf = ix.to_iobuf(); | ||
| iobuf expected_buf; | ||
| expected_buf.append((const uint8_t*)(expected.c_str()), expected.size()); |
There was a problem hiding this comment.
Yeah, that's a bit neater. Changed.
jcsp
force-pushed
the
remote-index-timestamps
branch
from
c3d1676 to
95e3ec7
Compare
This is a prerequisite to make timequeries seek to the proper chunk, instead of reading from the start of segments. The timestamp we care about is the batch max timestamp, because timequeries will want to find a batch that is definitely before the target timestamp, to start their scan from there. Related: redpanda-data#11801
jcsp
force-pushed
the
remote-index-timestamps
branch
from
95e3ec7 to
955e374
Compare
|
Force push: rebase for merge conflict. |
jcsp
force-pushed
the
remote-index-timestamps
branch
from
955e374 to
b3e5b48
Compare
This is a prerequisite to make timequeries seek
to the proper chunk, instead of reading from the
start of segments.
The timestamp we care about is the batch max timestamp, because timequeries will want to find a batch that is definitely before the target timestamp, to start their scan from there.
Related: #11801
Backports Required
Release Notes