From ecd5791cda2f579a49b09b972465f64906c51217 Mon Sep 17 00:00:00 2001 From: Tatsuya Kawano Date: Thu, 2 Jan 2025 12:52:34 +0800 Subject: [PATCH 1/4] Doc: Remove leftover mentions of background threads Also add the implementation details chapter to the crate level documentation. --- src/future/cache.rs | 40 +++++++----- src/lib.rs | 146 +++++++++++++++++++++++++++++++++++++++++++- src/sync/cache.rs | 40 +++++++----- src/sync/segment.rs | 40 +++++++----- 4 files changed, 223 insertions(+), 43 deletions(-) diff --git a/src/future/cache.rs b/src/future/cache.rs index efe998b5..2c191d26 100644 --- a/src/future/cache.rs +++ b/src/future/cache.rs @@ -1378,10 +1378,14 @@ where /// Discards all cached values. /// - /// This method returns immediately and a background thread will evict all the - /// cached values inserted before the time when this method was called. It is - /// guaranteed that the `get` method must not return these invalidated values - /// even if they have not been evicted. + /// This method returns immediately by just setting the current time as the + /// invalidation time. `get` and other retrieval methods are guaranteed not to + /// return the entries inserted before or at the invalidation time. + /// + /// The actual removal of the invalidated entries is done as a maintenance task + /// driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// /// Like the `invalidate` method, this method does not clear the historic /// popularity estimator of keys so that it retains the client activities of @@ -1392,15 +1396,21 @@ where /// Discards cached values that satisfy a predicate. /// - /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This - /// method returns immediately and a background thread will apply the closure to - /// each cached value inserted before the time when `invalidate_entries_if` was - /// called. If the closure returns `true` on a value, that value will be evicted - /// from the cache. + /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The + /// closure is called against each cached entry inserted before or at the time + /// when this method was called. If the closure returns `true` that entry will be + /// evicted from the cache. + /// + /// This method returns immediately by not actually removing the invalidated + /// entries. Instead, it just sets the predicate to the cache with the time when + /// this method was called. The actual removal of the invalidated entries is done + /// as a maintenance task driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// - /// Also the `get` method will apply the closure to a value to determine if it - /// should have been invalidated. Therefore, it is guaranteed that the `get` - /// method must not return invalidated values. + /// Also the `get` and other retrieval methods will apply the closure to a cached + /// entry to determine if it should have been invalidated. Therefore, it is + /// guaranteed that these methods must not return invalidated values. /// /// Note that you must call /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures] @@ -1413,8 +1423,10 @@ where /// popularity estimator of keys so that it retains the client activities of /// trying to retrieve an item. /// - /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures - /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled + /// [support-invalidation-closures]: + /// ./struct.CacheBuilder.html#method.support_invalidation_closures + /// [invalidation-disabled-error]: + /// ../enum.PredicateError.html#variant.InvalidationClosuresDisabled pub fn invalidate_entries_if(&self, predicate: F) -> Result where F: Fn(&K, &V) -> bool + Send + Sync + 'static, diff --git a/src/lib.rs b/src/lib.rs index ed8f551c..e0c4aad3 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -42,6 +42,61 @@ //! //! [tiny-lfu]: https://github.com/moka-rs/moka/wiki#admission-and-eviction-policies //! +//! ## Cache Policies +//! +//! When a cache is full, it has to select and evict existing entries to make some +//! room. A cache policy is a strategy to determine which entry to evict. +//! +//! The choice of the cache policy may have a significant impact on the performance +//! of the cache. Because the time for cache misses is usually much greater than the +//! time for cache hits, the miss rate (number of misses per second) has a +//! significant impact on the performance. +//! +//! Moka provides the following policies: +//! +//! - TinyLFU +//! - LRU +//! +//! ### TinyLFU +//! +//! TinyLFU is the default policy of the cache, and will be suitable for most +//! workloads. +//! +//! TinyLFU is a combination of the LRU eviction policy and the LFU admission policy. +//! LRU stands for Least Recently Used, which is very popular in many cache systems. +//! LFU stands for Least Frequently Used. +//! +//! ![The lifecycle of cached entries with TinyLFU][tiny-lfu-image] +//! +//! [tiny-lfu-image]: +//! https://github.com/moka-rs/moka/wiki/images/benchmarks/moka-tiny-lfu.png +//! +//! With TinyLFU policy, the cache will admit a new entry based on its popularity. If +//! the key of the entry is popular, it will be admitted to the cache. Otherwise, it +//! will be rejected. +//! +//! The popularity of the key is estimated by the historic popularity estimator +//! called LFU filter. It is a modified Count-Min Sketch, and it can estimate the +//! frequency of keys with a very low memory footprint (thus the name “tiny”). Note +//! that it tracks not only the keys currently in the cache, but all hit and missed +//! keys. +//! +//! Once the entry is admitted to the cache, it will be evicted based on the LRU +//! policy. It evicts the least recently used entry from the cache. +//! +//! TinyLFU will be suitable for most workloads, such as database, search, and +//! analytics. +//! +//! ### LRU +//! +//! LRU stands for Least Recently Used. +//! +//! With LRU policy, the cache will evict the least recently used entry. It is a +//! simple policy and has been used in many cache systems. +//! +//! LRU will be suitable for recency-biased workloads, such as job queues and event +//! streams. +//! //! # Examples //! //! See the following document: @@ -80,11 +135,100 @@ //! | `sync` | Rust 1.70.0 (June 3, 2022) | //! //! It will keep a rolling MSRV policy of at least 6 months. If the default features -//! with a mondatory features (`future` or `sync`) are enabled, MSRV will be updated +//! with a mandatory features (`future` or `sync`) are enabled, MSRV will be updated //! conservatively. When using other features, MSRV might be updated more frequently, //! up to the latest stable. //! //! In both cases, increasing MSRV is _not_ considered a semver-breaking change. +//! +//! # Implementation Details +//! +//! ## Concurrency +//! +//! The entry replacement algorithms are kept eventually consistent with the +//! concurrent hash table. While updates to the cache are immediately applied to the +//! hash table, recording of reads and writes may not be immediately reflected on the +//! cache policy's data structures. +//! +//! These cache policy structures are guarded by a lock and operations are applied in +//! batches to avoid lock contention. +//! +//! Recap: +//! +//! - The concurrent hash table in the cache is _strong consistent_: +//! - It is a lock-free data structure and immediately applies updates. +//! - It is guaranteed that the inserted entry will become visible immediately to +//! all threads. +//! - The cache policy's data structures are _eventually consistent_: +//! - They are guarded by a lock and operations are applied in batches. +//! - An example of eventual consistency: the `entry_count` method may return an +//! outdated value. +//! +//! ### Bounded Channels +//! +//! In order to hold the recordings of reads and writes until they are applied to the +//! cache policy's data structures, the cache uses two bounded channels, one for +//! reads and the other for writes. Bounded means that a channel have a maximum +//! number of elements that can be stored. +//! +//! These channels are drained when one of the following conditions is met: +//! +//! - The numbers of read or write recordings reach to the configured amounts. +//! - It is currently hard-coded to 64. +//! - Or, the certain time past from the last draining. +//! - It is currently hard-coded to 300 milliseconds. +//! +//! Cache does not have a dedicated thread for draining. Instead, it is done by a +//! user thread. When user code calls certain cache methods, such as `get`, +//! `get_with`, `insert`, and `run_pending_tasks`, the cache checks if the above +//! condition is met, and if so, it will start draining as a part of the method call +//! and apply the recordings to the cache policy's data structures. See [the +//! Maintenance Tasks section](#maintenance-tasks) for more details of applying the +//! recordings. +//! +//! ### When a Bounded Channels is Full +//! +//! Under heavy concurrent operations from clients, draining may not be able to catch +//! up and the bounded channels can become full. In this case, the cache will do one +//! of the followings: +//! +//! - For the read channel, recordings of new reads will be discarded, so that +//! retrievals will never be blocked. This behavior may have some impact to the hit +//! rate of the cache. +//! - For the write channel, updates from clients to the cache will be blocked until +//! the draining task catches up. +//! +//! ## Maintenance Tasks +//! +//! When draining the read and write recordings from the channels, the cache will do +//! the following maintenance tasks: +//! +//! 1. Determine whether to admit an entry to the cache or not, based on its +//! popularity. +//! - If not, the entry is removed from the internal concurrent hash table. +//! 2. Apply the recording of cache reads and writes to the internal data structures +//! for the cache policies, such as the LFU filter, LRU queues, and hierarchical +//! timer wheels. +//! - The hierarchical timer wheels are used for the per-entry expiration policy. +//! 3. When cache's max capacity is exceeded, remove least recently used (LRU) +//! entries. +//! 4. Remove expired entries. +//! 5. Find and remove the entries that have been invalidated by the `invalidate_all` +//! or `invalidate_entries_if` methods. +//! 6. Deliver removal notifications to the eviction listener. (Call the eviction +//! listener closure with the information about the evicted entry) +//! +//! The following cache method calls may trigger the maintenance tasks: +//! +//! - All cache write methods: `insert`, `get_with`, `invalidate`, etc., except for +//! `invalidate_all` and `invalidate_entries_if`. +//! - Some of the cache read methods: `get` +//! - `run_pending_tasks` method, which executes the pending maintenance tasks +//! explicitly. +//! +//! Except `run_pending_tasks` method, the maintenance tasks are executed lazily +//! when one of the conditions in the [Bounded Channels](#bounded-channels) section +//! is met. #[cfg(not(any(feature = "sync", feature = "future")))] compile_error!( diff --git a/src/sync/cache.rs b/src/sync/cache.rs index 6df8773d..6c846f06 100644 --- a/src/sync/cache.rs +++ b/src/sync/cache.rs @@ -1658,10 +1658,14 @@ where /// Discards all cached values. /// - /// This method returns immediately and a background thread will evict all the - /// cached values inserted before the time when this method was called. It is - /// guaranteed that the `get` method must not return these invalidated values - /// even if they have not been evicted. + /// This method returns immediately by just setting the current time as the + /// invalidation time. `get` and other retrieval methods are guaranteed not to + /// return the entries inserted before or at the invalidation time. + /// + /// The actual removal of the invalidated entries is done as a maintenance task + /// driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// /// Like the `invalidate` method, this method does not clear the historic /// popularity estimator of keys so that it retains the client activities of @@ -1672,15 +1676,21 @@ where /// Discards cached values that satisfy a predicate. /// - /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This - /// method returns immediately and a background thread will apply the closure to - /// each cached value inserted before the time when `invalidate_entries_if` was - /// called. If the closure returns `true` on a value, that value will be evicted - /// from the cache. + /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The + /// closure is called against each cached entry inserted before or at the time + /// when this method was called. If the closure returns `true` that entry will be + /// evicted from the cache. + /// + /// This method returns immediately by not actually removing the invalidated + /// entries. Instead, it just sets the predicate to the cache with the time when + /// this method was called. The actual removal of the invalidated entries is done + /// as a maintenance task driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// - /// Also the `get` method will apply the closure to a value to determine if it - /// should have been invalidated. Therefore, it is guaranteed that the `get` - /// method must not return invalidated values. + /// Also the `get` and other retrieval methods will apply the closure to a cached + /// entry to determine if it should have been invalidated. Therefore, it is + /// guaranteed that these methods must not return invalidated values. /// /// Note that you must call /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures] @@ -1693,8 +1703,10 @@ where /// popularity estimator of keys so that it retains the client activities of /// trying to retrieve an item. /// - /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures - /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled + /// [support-invalidation-closures]: + /// ./struct.CacheBuilder.html#method.support_invalidation_closures + /// [invalidation-disabled-error]: + /// ../enum.PredicateError.html#variant.InvalidationClosuresDisabled pub fn invalidate_entries_if(&self, predicate: F) -> Result where F: Fn(&K, &V) -> bool + Send + Sync + 'static, diff --git a/src/sync/segment.rs b/src/sync/segment.rs index f3bd626b..c1d1cf17 100644 --- a/src/sync/segment.rs +++ b/src/sync/segment.rs @@ -503,10 +503,14 @@ where /// Discards all cached values. /// - /// This method returns immediately and a background thread will evict all the - /// cached values inserted before the time when this method was called. It is - /// guaranteed that the `get` method must not return these invalidated values - /// even if they have not been evicted. + /// This method returns immediately by just setting the current time as the + /// invalidation time. `get` and other retrieval methods are guaranteed not to + /// return the entries inserted before or at the invalidation time. + /// + /// The actual removal of the invalidated entries is done as a maintenance task + /// driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// /// Like the `invalidate` method, this method does not clear the historic /// popularity estimator of keys so that it retains the client activities of @@ -519,15 +523,21 @@ where /// Discards cached values that satisfy a predicate. /// - /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This - /// method returns immediately and a background thread will apply the closure to - /// each cached value inserted before the time when `invalidate_entries_if` was - /// called. If the closure returns `true` on a value, that value will be evicted - /// from the cache. + /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The + /// closure is called against each cached entry inserted before or at the time + /// when this method was called. If the closure returns `true` that entry will be + /// evicted from the cache. + /// + /// This method returns immediately by not actually removing the invalidated + /// entries. Instead, it just sets the predicate to the cache with the time when + /// this method was called. The actual removal of the invalidated entries is done + /// as a maintenance task driven by a user thread. For more details, see + /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate + /// level documentation. /// - /// Also the `get` method will apply the closure to a value to determine if it - /// should have been invalidated. Therefore, it is guaranteed that the `get` - /// method must not return invalidated values. + /// Also the `get` and other retrieval methods will apply the closure to a cached + /// entry to determine if it should have been invalidated. Therefore, it is + /// guaranteed that these methods must not return invalidated values. /// /// Note that you must call /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures] @@ -540,8 +550,10 @@ where /// popularity estimator of keys so that it retains the client activities of /// trying to retrieve an item. /// - /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures - /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled + /// [support-invalidation-closures]: + /// ./struct.CacheBuilder.html#method.support_invalidation_closures + /// [invalidation-disabled-error]: + /// ../enum.PredicateError.html#variant.InvalidationClosuresDisabled pub fn invalidate_entries_if(&self, predicate: F) -> Result<(), PredicateError> where F: Fn(&K, &V) -> bool + Send + Sync + 'static, From 8284a079c24251c929576d5dde7e0093b82fa639 Mon Sep 17 00:00:00 2001 From: Tatsuya Kawano Date: Thu, 2 Jan 2025 12:58:14 +0800 Subject: [PATCH 2/4] Fix a Clippy warning on a doc comment --- src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib.rs b/src/lib.rs index e0c4aad3..402483ae 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -221,7 +221,7 @@ //! The following cache method calls may trigger the maintenance tasks: //! //! - All cache write methods: `insert`, `get_with`, `invalidate`, etc., except for -//! `invalidate_all` and `invalidate_entries_if`. +//! `invalidate_all` and `invalidate_entries_if`. //! - Some of the cache read methods: `get` //! - `run_pending_tasks` method, which executes the pending maintenance tasks //! explicitly. From c5c365885f10925634da178b00cffca232ba1c34 Mon Sep 17 00:00:00 2001 From: Tatsuya Kawano Date: Thu, 2 Jan 2025 13:11:51 +0800 Subject: [PATCH 3/4] Update the change log --- CHANGELOG.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38c5e7b8..a49c6501 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,12 @@ Bumped the minimum supported Rust version (MSRV) to 1.70 (June 1, 2023) ### Fixed -- Fixed an occasional panic in an internal `to_std_instant` method when per-entry +- Prevent an occasional panic in an internal `to_std_instant` method when per-entry expiration policy is used. ([#472][gh-issue-0472]) +- Documentation: Removed leftover mentions of background threads. + ([#464][gh-issue-0464]) + - Also added the implementation details chapter to the crate top-level + documentation to explain some internal behavior of the cache. ### Added @@ -19,8 +23,8 @@ Bumped the minimum supported Rust version (MSRV) to 1.70 (June 1, 2023) - Removed `triomphe` crate from the dependency by adding our own internal `Arc` type. ([#456][gh-pull-0456]) - - Our `Arc` is more memory efficient than `std::sync::Arc` or `triomphe::Arc` on - 64-bit platforms as it uses a single `AtomicU32` counter. + - Our `Arc` will be more memory efficient than `std::sync::Arc` or + `triomphe::Arc` on 64-bit platforms as it uses a single `AtomicU32` counter. - Removed needless traits along with `async-trait` usage. ([#445][gh-pull-0445], by [@Swatinem][gh-Swatinem]) @@ -909,6 +913,7 @@ The minimum supported Rust version (MSRV) is now 1.51.0 (Mar 25, 2021). [gh-zonyitoo]: https://github.com/zonyitoo [gh-issue-0472]: https://github.com/moka-rs/moka/issues/472/ +[gh-issue-0464]: https://github.com/moka-rs/moka/issues/464/ [gh-issue-0412]: https://github.com/moka-rs/moka/issues/412/ [gh-issue-0385]: https://github.com/moka-rs/moka/issues/385/ [gh-issue-0329]: https://github.com/moka-rs/moka/issues/329/ From cd2b3b3242765090c2443d683c02bc6e75cc21c2 Mon Sep 17 00:00:00 2001 From: Tatsuya Kawano Date: Thu, 2 Jan 2025 13:38:51 +0800 Subject: [PATCH 4/4] Fix a typo in the README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index afa01122..3e845ca5 100644 --- a/README.md +++ b/README.md @@ -473,7 +473,7 @@ Moka's minimum supported Rust versions (MSRV) are the followings: | `sync` | Rust 1.70.0 (June 3, 2022) | It will keep a rolling MSRV policy of at least 6 months. If the default features with -a mondatory features (`future` or `sync`) are enabled, MSRV will be updated +a mandatory features (`future` or `sync`) are enabled, MSRV will be updated conservatively. When using other features, MSRV might be updated more frequently, up to the latest stable.