Skip to content

Commit

Permalink
feat: Implementing cross-segment read/write for WAL based on local di…
Browse files Browse the repository at this point in the history
…sk (apache#1556)

## Rationale
Improving WAL based on local disk.

This is a follow-up task for apache#1552.

## Detailed Changes
1. Make MAX_FILE_SIZE configurable.
2. Allocate enough space when creating a segment to avoid remapping when
appending to the segment.​
3. Add `MultiSegmentLogIterator` to enable cross-segment reading.
4. When writing, if the current segment has insufficient space, create a
new segment and write to the new segment.​

## Test Plan
Unit test.
  • Loading branch information
dracoooooo authored and LeslieKid committed Sep 24, 2024
1 parent ac5f1c5 commit 7030f15
Show file tree
Hide file tree
Showing 4 changed files with 666 additions and 243 deletions.
2 changes: 1 addition & 1 deletion src/wal/src/local_storage_impl/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LocalStorageConfig {
pub path: String,
pub max_segment_size: u64,
pub max_segment_size: usize,
pub cache_size: usize,
}

Expand Down
8 changes: 8 additions & 0 deletions src/wal/src/local_storage_impl/record_encoding.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ define_result!(Error);
/// | (u8) | (u32) | (u32) | (u64) | (u64) | (u32) | |
/// +---------+--------+--------+------------+--------------+--------------+-------+
/// ```
#[derive(Debug)]
pub struct Record<'a> {
/// The version number of the record.
pub version: u8,
Expand Down Expand Up @@ -183,6 +184,13 @@ impl RecordEncoding {

// Read length
let length = buf.try_get_u32().context(Decoding)?;
ensure!(
length > 0,
LengthMismatch {
expected: 1usize,
actual: 0usize
}
);

// Ensure the buf is long enough
ensure!(
Expand Down
Loading

0 comments on commit 7030f15

Please sign in to comment.