Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for compression of the oplog in case of mergeable operations. #100

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
480edca
oplog compression
Wulf0x67E7 Sep 14, 2021
393255e
forgot about associated const
Wulf0x67E7 Sep 14, 2021
e238b4f
"fixed" loom test
Wulf0x67E7 Sep 15, 2021
0c430ec
test limited compress range
Wulf0x67E7 Sep 15, 2021
04adefa
more tests, found/fixed none removal bug
Wulf0x67E7 Sep 15, 2021
abcf063
optimize none removal when few nones
Wulf0x67E7 Sep 15, 2021
1ca8a2a
preempted possible future bug with none_back_count
Wulf0x67E7 Sep 15, 2021
4f5d9fe
4. optimization makes prev bug fix redundant
Wulf0x67E7 Sep 15, 2021
5bc3699
comments and minor tweaks
Wulf0x67E7 Sep 16, 2021
0aa5900
...reversed tweak to bring rust version back down
Wulf0x67E7 Sep 16, 2021
722c318
fixed missing optimization wghile reversing tweak
Wulf0x67E7 Sep 19, 2021
f22dc65
try_compress(&mut prev, next): better fits model
Wulf0x67E7 Sep 19, 2021
6c7bfd7
rangeify and cleanup
Wulf0x67E7 Sep 19, 2021
3a37d0d
more cleanup
Wulf0x67E7 Sep 19, 2021
c6aa598
One final cleanup and test
Wulf0x67E7 Sep 20, 2021
f0c9966
Apply suggestions from code review
Wulf0x67E7 Sep 26, 2021
cde672f
suggested doc changes
Wulf0x67E7 Sep 26, 2021
f74cadd
unwrap_or_else to expect
Wulf0x67E7 Sep 26, 2021
000d512
inverted do-compression condition
Wulf0x67E7 Sep 26, 2021
cec99e0
factored out compress_insert_op
Wulf0x67E7 Sep 26, 2021
0382332
cleaner sanity checks
Wulf0x67E7 Sep 26, 2021
f49adca
test tweaks
Wulf0x67E7 Sep 26, 2021
13c380e
quickcheck correctness
Wulf0x67E7 Sep 26, 2021
5f58376
heavily rigged criterion benchmark
Wulf0x67E7 Sep 26, 2021
4b61b26
Apply documentation suggestions from code review
Wulf0x67E7 Oct 4, 2021
1ca7bbc
tweaking extend variant branch layout
Wulf0x67E7 Oct 4, 2021
e3ff872
Merge branch 'master' of https://github.com/Wulf0x67E7/left-right
Wulf0x67E7 Oct 4, 2021
5138829
factored out custom retain
Wulf0x67E7 Oct 4, 2021
b852b9c
refined quicktest
Wulf0x67E7 Oct 4, 2021
6ca3be6
make test use factored out retain instead
Wulf0x67E7 Oct 4, 2021
b6cbabe
correcting/tweaking outdated doc
Wulf0x67E7 Oct 4, 2021
14c2c9c
benchmark revamp
Wulf0x67E7 Oct 8, 2021
0016728
changed quickcheck to not blow up miri
Wulf0x67E7 Oct 9, 2021
e2a5b82
fixed bug from false assumptions
Wulf0x67E7 Oct 9, 2021
1524b18
more benches
Wulf0x67E7 Oct 9, 2021
f512288
un-constify benchamrks
Wulf0x67E7 Oct 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,17 @@ pub use crate::read::{ReadGuard, ReadHandle, ReadHandleFactory};

pub mod aliasing;

/// The result of calling [`Absorb::try_compress`](Absorb::try_compress).
#[derive(Debug)]
pub enum TryCompressResult<O> {
/// Returned when [`try_compress`](Absorb::try_compress) was successful.
Compressed,
/// Returned when [`try_compress`](Absorb::try_compress) failed because `prev` and `next` are independent of each other, contains `next`.
Independent(O),
/// Returned when [`try_compress`](Absorb::try_compress) failed because `prev` must precede `next`, contains `next`.
jonhoo marked this conversation as resolved.
Show resolved Hide resolved
Dependent(O),
}

/// Types that can incorporate operations of type `O`.
///
/// This trait allows `left-right` to keep the two copies of the underlying data structure (see the
Expand Down Expand Up @@ -261,6 +272,32 @@ pub trait Absorb<O> {
/// subtly affect results like the `RandomState` of a `HashMap` which can change iteration
/// order.
fn sync_with(&mut self, first: &Self);

/// Range at which [`WriteHandle`] tries to compress the oplog, reset each time a compression succeeds.
///
/// Can be used to avoid having insertion into the oplog be O(oplog.len * ops.len) if it is filled with mainly independent ops.
///
/// Defaults to `0`, which disables compression and allows the usage of an efficient fallback.
const MAX_COMPRESS_RANGE: usize = 0;

/// Try to compress two ops into a single op to optimize the oplog.
///
/// `prev` is the target of the compression and temporarily removed from the oplog, `next` is the op to be inserted.
///
/// A return value of [`TryCompressResult::Compressed`] means the ops were successfully compressed,
/// [`TryCompressResult::Independent`] that while the ops can't be compressed,
/// `next` can safely precede `prev`, which therefore can be simply skipped over,
jonhoo marked this conversation as resolved.
Show resolved Hide resolved
/// and [`TryCompressResult::Dependent`] that they can not be compressed, and `prev` must precede `next`.
///
/// Defaults to [`TryCompressResult::Dependent`], which sub-optimally disables compression.
/// Setting [`Self::MAX_COMPRESS_RANGE`](Absorb::MAX_COMPRESS_RANGE) to `0` is vastly more efficient for that.
Wulf0x67E7 marked this conversation as resolved.
Show resolved Hide resolved
fn try_compress(prev: &mut O, next: O) -> TryCompressResult<O> {
// yes, unnecessary, but: makes it so that prev is not an unused variable
// and really matches the mental model of 'all ops are dependent'.
match prev {
_ => TryCompressResult::Dependent(next),
}
}
}

/// Construct a new write and read handle pair from an empty data structure.
Expand Down
56 changes: 53 additions & 3 deletions src/utilities.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,63 @@ impl Absorb<CounterAddOp> for i32 {
*self += operation.0;
}

fn absorb_second(&mut self, operation: CounterAddOp, _: &Self) {
*self += operation.0;
fn sync_with(&mut self, first: &Self) {
jonhoo marked this conversation as resolved.
Show resolved Hide resolved
*self = *first
}
}

#[cfg(test)]
#[derive(Debug, Eq, PartialEq)]
pub enum CompressibleCounterOp<const MAX_COMPRESS_RANGE: usize> {
Set(i32),
Add(i32),
Sub(i32),
}

fn drop_first(self: Box<Self>) {}
#[cfg(test)]
impl<const MAX_COMPRESS_RANGE: usize> Absorb<CompressibleCounterOp<MAX_COMPRESS_RANGE>> for i32 {
fn absorb_first(
&mut self,
operation: &mut CompressibleCounterOp<MAX_COMPRESS_RANGE>,
_: &Self,
) {
match operation {
CompressibleCounterOp::Set(v) => *self = *v,
CompressibleCounterOp::Add(v) => *self += *v,
CompressibleCounterOp::Sub(v) => *self -= *v,
}
}

fn sync_with(&mut self, first: &Self) {
*self = *first
}

const MAX_COMPRESS_RANGE: usize = MAX_COMPRESS_RANGE;

fn try_compress(
prev: &mut CompressibleCounterOp<MAX_COMPRESS_RANGE>,
next: CompressibleCounterOp<MAX_COMPRESS_RANGE>,
) -> TryCompressResult<CompressibleCounterOp<MAX_COMPRESS_RANGE>> {
match (prev, next) {
(CompressibleCounterOp::Add(prev), CompressibleCounterOp::Add(next)) => {
*prev += next;
TryCompressResult::Compressed
}
(CompressibleCounterOp::Sub(prev), CompressibleCounterOp::Sub(next)) => {
*prev += next;
TryCompressResult::Compressed
}
(CompressibleCounterOp::Add(_), CompressibleCounterOp::Sub(next)) => {
TryCompressResult::Independent(CompressibleCounterOp::Sub(next))
}
Wulf0x67E7 marked this conversation as resolved.
Show resolved Hide resolved
(CompressibleCounterOp::Sub(_), CompressibleCounterOp::Add(next)) => {
TryCompressResult::Independent(CompressibleCounterOp::Add(next))
}
(CompressibleCounterOp::Set(_), next) => TryCompressResult::Dependent(next),
(prev, CompressibleCounterOp::Set(next)) => {
*prev = CompressibleCounterOp::Set(next);
TryCompressResult::Compressed
}
}
}
}
Loading