Persist compaction time window to table manifest

[v0y4g3r]

What problem does the new feature solve?

Everytime when compacting a table, we scan all SST files in level 0 and calculate a suitable time window for compaction.

greptimedb/src/storage/src/compaction/strategy.rs

Line 51 in af1f8d6

let time_bucket = infer_time_bucket(&files);

With the presumption that the workload of a given table remains consistent, we can persist the inferred time window to table manifest to avoid calculating it everytime.

Another benefit of this proposal is that it enforces all SSTs in level 1 to have the same aligned time window, which makes it easier to find compactable SSTs in level 1.

What does the feature do?

Persist inferred time window to table manifest on first compaction and use that window for following compactions.

Implementation challenges

We can add a field compaction_time_window on TableOptions :

greptimedb/src/table/src/requests.rs

Lines 62 to 70 in e17d5a1

	pub struct TableOptions {
	/// Memtable size of memtable.
	pub write_buffer_size: Option<ReadableSize>,
	/// Time-to-live of table. Expired data will be automatically purged.
	#[serde(with = "humantime_serde")]
	pub ttl: Option<Duration>,
	/// Extra options that may not applicable to all table engines.
	pub extra_options: HashMap<String, String>,
	}

We can also allow users to manually specify this option when creating table using WITH clause like:

CREATE TABLE monitor (host STRING,
    ts BIGINT,
    cpu DOUBLE DEFAULT 0,
    memory DOUBLE,
    TIME INDEX (ts),
    PRIMARY KEY(host)
) ENGINE=mito WITH(regions=1, ttl='7days', compaction_time_window='2hours');

[etolbakov]

Hi @v0y4g3r,
I would like to give a go if you don't mind.

[killme2008]

@v0y4g3r Do we finish this job?

[v0y4g3r]

@v0y4g3r Do we finish this job?

Not yet. We still need to to finish the compaction window persistence part. I will resolve this ASAP.

[killme2008]

ping @v0y4g3r