Merge pull request #479 from moka-rs/fix-doc-about-pending-tasks

Documentation: Remove leftover mentions of background threads

CHANGELOG.md

@@ -7,8 +7,12 @@ Bumped the minimum supported Rust version (MSRV) to 1.70 (June 1, 2023)
 
 ### Fixed
 
-- Fixed an occasional panic in an internal `to_std_instant` method when per-entry
+- Prevent an occasional panic in an internal `to_std_instant` method when per-entry
   expiration policy is used. ([#472][gh-issue-0472])
+- Documentation: Removed leftover mentions of background threads.
+  ([#464][gh-issue-0464])
+    - Also added the implementation details chapter to the crate top-level
+      documentation to explain some internal behavior of the cache.
 
 ### Added
 
@@ -19,8 +23,8 @@ Bumped the minimum supported Rust version (MSRV) to 1.70 (June 1, 2023)
 
 - Removed `triomphe` crate from the dependency by adding our own internal `Arc` type.
   ([#456][gh-pull-0456])
-    - Our `Arc` is more memory efficient than `std::sync::Arc` or `triomphe::Arc` on
-      64-bit platforms as it uses a single `AtomicU32` counter.
+    - Our `Arc` will be more memory efficient than `std::sync::Arc` or
+      `triomphe::Arc` on 64-bit platforms as it uses a single `AtomicU32` counter.
 - Removed needless traits along with `async-trait` usage. ([#445][gh-pull-0445], by
   [@Swatinem][gh-Swatinem])
 
@@ -909,6 +913,7 @@ The minimum supported Rust version (MSRV) is now 1.51.0 (Mar 25, 2021).
 [gh-zonyitoo]: https://github.com/zonyitoo
 
 [gh-issue-0472]: https://github.com/moka-rs/moka/issues/472/
+[gh-issue-0464]: https://github.com/moka-rs/moka/issues/464/
 [gh-issue-0412]: https://github.com/moka-rs/moka/issues/412/
 [gh-issue-0385]: https://github.com/moka-rs/moka/issues/385/
 [gh-issue-0329]: https://github.com/moka-rs/moka/issues/329/

README.md

@@ -473,7 +473,7 @@ Moka's minimum supported Rust versions (MSRV) are the followings:
 | `sync`   | Rust 1.70.0 (June 3, 2022) |
 
 It will keep a rolling MSRV policy of at least 6 months. If the default features with
-a mondatory features (`future` or `sync`) are enabled, MSRV will be updated
+a mandatory features (`future` or `sync`) are enabled, MSRV will be updated
 conservatively. When using other features, MSRV might be updated more frequently, up
 to the latest stable.
 

src/future/cache.rs

@@ -1378,10 +1378,14 @@ where
 
     /// Discards all cached values.
     ///
-    /// This method returns immediately and a background thread will evict all the
-    /// cached values inserted before the time when this method was called. It is
-    /// guaranteed that the `get` method must not return these invalidated values
-    /// even if they have not been evicted.
+    /// This method returns immediately by just setting the current time as the
+    /// invalidation time. `get` and other retrieval methods are guaranteed not to
+    /// return the entries inserted before or at the invalidation time.
+    ///
+    /// The actual removal of the invalidated entries is done as a maintenance task
+    /// driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
     /// Like the `invalidate` method, this method does not clear the historic
     /// popularity estimator of keys so that it retains the client activities of
@@ -1392,15 +1396,21 @@ where
 
     /// Discards cached values that satisfy a predicate.
     ///
-    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This
-    /// method returns immediately and a background thread will apply the closure to
-    /// each cached value inserted before the time when `invalidate_entries_if` was
-    /// called. If the closure returns `true` on a value, that value will be evicted
-    /// from the cache.
+    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The
+    /// closure is called against each cached entry inserted before or at the time
+    /// when this method was called. If the closure returns `true` that entry will be
+    /// evicted from the cache.
+    ///
+    /// This method returns immediately by not actually removing the invalidated
+    /// entries. Instead, it just sets the predicate to the cache with the time when
+    /// this method was called. The actual removal of the invalidated entries is done
+    /// as a maintenance task driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
-    /// Also the `get` method will apply the closure to a value to determine if it
-    /// should have been invalidated. Therefore, it is guaranteed that the `get`
-    /// method must not return invalidated values.
+    /// Also the `get` and other retrieval methods will apply the closure to a cached
+    /// entry to determine if it should have been invalidated. Therefore, it is
+    /// guaranteed that these methods must not return invalidated values.
     ///
     /// Note that you must call
     /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures]
@@ -1413,8 +1423,10 @@ where
     /// popularity estimator of keys so that it retains the client activities of
     /// trying to retrieve an item.
     ///
-    /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures
-    /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
+    /// [support-invalidation-closures]:
+    ///     ./struct.CacheBuilder.html#method.support_invalidation_closures
+    /// [invalidation-disabled-error]:
+    ///     ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
     pub fn invalidate_entries_if<F>(&self, predicate: F) -> Result<PredicateId, PredicateError>
     where
         F: Fn(&K, &V) -> bool + Send + Sync + 'static,

src/lib.rs

@@ -42,6 +42,61 @@
 //!
 //! [tiny-lfu]: https://github.com/moka-rs/moka/wiki#admission-and-eviction-policies
 //!
+//! ## Cache Policies
+//!
+//! When a cache is full, it has to select and evict existing entries to make some
+//! room. A cache policy is a strategy to determine which entry to evict.
+//!
+//! The choice of the cache policy may have a significant impact on the performance
+//! of the cache. Because the time for cache misses is usually much greater than the
+//! time for cache hits, the miss rate (number of misses per second) has a
+//! significant impact on the performance.
+//!
+//! Moka provides the following policies:
+//!
+//! - TinyLFU
+//! - LRU
+//!
+//! ### TinyLFU
+//!
+//! TinyLFU is the default policy of the cache, and will be suitable for most
+//! workloads.
+//!
+//! TinyLFU is a combination of the LRU eviction policy and the LFU admission policy.
+//! LRU stands for Least Recently Used, which is very popular in many cache systems.
+//! LFU stands for Least Frequently Used.
+//!
+//! ![The lifecycle of cached entries with TinyLFU][tiny-lfu-image]
+//!
+//! [tiny-lfu-image]:
+//!     https://github.com/moka-rs/moka/wiki/images/benchmarks/moka-tiny-lfu.png
+//!
+//! With TinyLFU policy, the cache will admit a new entry based on its popularity. If
+//! the key of the entry is popular, it will be admitted to the cache. Otherwise, it
+//! will be rejected.
+//!
+//! The popularity of the key is estimated by the historic popularity estimator
+//! called LFU filter. It is a modified Count-Min Sketch, and it can estimate the
+//! frequency of keys with a very low memory footprint (thus the name “tiny”). Note
+//! that it tracks not only the keys currently in the cache, but all hit and missed
+//! keys.
+//!
+//! Once the entry is admitted to the cache, it will be evicted based on the LRU
+//! policy. It evicts the least recently used entry from the cache.
+//!
+//! TinyLFU will be suitable for most workloads, such as database, search, and
+//! analytics.
+//!
+//! ### LRU
+//!
+//! LRU stands for Least Recently Used.
+//!
+//! With LRU policy, the cache will evict the least recently used entry. It is a
+//! simple policy and has been used in many cache systems.
+//!
+//! LRU will be suitable for recency-biased workloads, such as job queues and event
+//! streams.
+//!
 //! # Examples
 //!
 //! See the following document:
@@ -80,11 +135,100 @@
 //! | `sync`   | Rust 1.70.0 (June 3, 2022) |
 //!
 //! It will keep a rolling MSRV policy of at least 6 months. If the default features
-//! with a mondatory features (`future` or `sync`) are enabled, MSRV will be updated
+//! with a mandatory features (`future` or `sync`) are enabled, MSRV will be updated
 //! conservatively. When using other features, MSRV might be updated more frequently,
 //! up to the latest stable.
 //!
 //! In both cases, increasing MSRV is _not_ considered a semver-breaking change.
+//!
+//! # Implementation Details
+//!
+//! ## Concurrency
+//!
+//! The entry replacement algorithms are kept eventually consistent with the
+//! concurrent hash table. While updates to the cache are immediately applied to the
+//! hash table, recording of reads and writes may not be immediately reflected on the
+//! cache policy's data structures.
+//!
+//! These cache policy structures are guarded by a lock and operations are applied in
+//! batches to avoid lock contention.
+//!
+//! Recap:
+//!
+//! - The concurrent hash table in the cache is _strong consistent_:
+//!     - It is a lock-free data structure and immediately applies updates.
+//!     - It is guaranteed that the inserted entry will become visible immediately to
+//!       all threads.
+//! - The cache policy's data structures are _eventually consistent_:
+//!     - They are guarded by a lock and operations are applied in batches.
+//!     - An example of eventual consistency: the `entry_count` method may return an
+//!       outdated value.
+//!
+//! ### Bounded Channels
+//!
+//! In order to hold the recordings of reads and writes until they are applied to the
+//! cache policy's data structures, the cache uses two bounded channels, one for
+//! reads and the other for writes. Bounded means that a channel have a maximum
+//! number of elements that can be stored.
+//!
+//! These channels are drained when one of the following conditions is met:
+//!
+//! - The numbers of read or write recordings reach to the configured amounts.
+//!     - It is currently hard-coded to 64.
+//! - Or, the certain time past from the last draining.
+//!     - It is currently hard-coded to 300 milliseconds.
+//!
+//! Cache does not have a dedicated thread for draining. Instead, it is done by a
+//! user thread. When user code calls certain cache methods, such as `get`,
+//! `get_with`, `insert`, and `run_pending_tasks`, the cache checks if the above
+//! condition is met, and if so, it will start draining as a part of the method call
+//! and apply the recordings to the cache policy's data structures. See [the
+//! Maintenance Tasks section](#maintenance-tasks) for more details of applying the
+//! recordings.
+//!
+//! ### When a Bounded Channels is Full
+//!
+//! Under heavy concurrent operations from clients, draining may not be able to catch
+//! up and the bounded channels can become full. In this case, the cache will do one
+//! of the followings:
+//!
+//! - For the read channel, recordings of new reads will be discarded, so that
+//!   retrievals will never be blocked. This behavior may have some impact to the hit
+//!   rate of the cache.
+//! - For the write channel, updates from clients to the cache will be blocked until
+//!   the draining task catches up.
+//!
+//! ## Maintenance Tasks
+//!
+//! When draining the read and write recordings from the channels, the cache will do
+//! the following maintenance tasks:
+//!
+//! 1. Determine whether to admit an entry to the cache or not, based on its
+//!    popularity.
+//!    - If not, the entry is removed from the internal concurrent hash table.
+//! 2. Apply the recording of cache reads and writes to the internal data structures
+//!    for the cache policies, such as the LFU filter, LRU queues, and hierarchical
+//!    timer wheels.
+//!    - The hierarchical timer wheels are used for the per-entry expiration policy.
+//! 3. When cache's max capacity is exceeded, remove least recently used (LRU)
+//!    entries.
+//! 4. Remove expired entries.
+//! 5. Find and remove the entries that have been invalidated by the `invalidate_all`
+//!    or `invalidate_entries_if` methods.
+//! 6. Deliver removal notifications to the eviction listener. (Call the eviction
+//!    listener closure with the information about the evicted entry)
+//!
+//! The following cache method calls may trigger the maintenance tasks:
+//!
+//! - All cache write methods: `insert`, `get_with`, `invalidate`, etc., except for
+//!   `invalidate_all` and `invalidate_entries_if`.
+//! - Some of the cache read methods: `get`
+//! - `run_pending_tasks` method, which executes the pending maintenance tasks
+//!    explicitly.
+//!
+//! Except `run_pending_tasks` method, the maintenance tasks are executed lazily
+//! when one of the conditions in the [Bounded Channels](#bounded-channels) section
+//! is met.
 
 #[cfg(not(any(feature = "sync", feature = "future")))]
 compile_error!(

src/sync/cache.rs

@@ -1658,10 +1658,14 @@ where
 
     /// Discards all cached values.
     ///
-    /// This method returns immediately and a background thread will evict all the
-    /// cached values inserted before the time when this method was called. It is
-    /// guaranteed that the `get` method must not return these invalidated values
-    /// even if they have not been evicted.
+    /// This method returns immediately by just setting the current time as the
+    /// invalidation time. `get` and other retrieval methods are guaranteed not to
+    /// return the entries inserted before or at the invalidation time.
+    ///
+    /// The actual removal of the invalidated entries is done as a maintenance task
+    /// driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
     /// Like the `invalidate` method, this method does not clear the historic
     /// popularity estimator of keys so that it retains the client activities of
@@ -1672,15 +1676,21 @@ where
 
     /// Discards cached values that satisfy a predicate.
     ///
-    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This
-    /// method returns immediately and a background thread will apply the closure to
-    /// each cached value inserted before the time when `invalidate_entries_if` was
-    /// called. If the closure returns `true` on a value, that value will be evicted
-    /// from the cache.
+    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The
+    /// closure is called against each cached entry inserted before or at the time
+    /// when this method was called. If the closure returns `true` that entry will be
+    /// evicted from the cache.
+    ///
+    /// This method returns immediately by not actually removing the invalidated
+    /// entries. Instead, it just sets the predicate to the cache with the time when
+    /// this method was called. The actual removal of the invalidated entries is done
+    /// as a maintenance task driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
-    /// Also the `get` method will apply the closure to a value to determine if it
-    /// should have been invalidated. Therefore, it is guaranteed that the `get`
-    /// method must not return invalidated values.
+    /// Also the `get` and other retrieval methods will apply the closure to a cached
+    /// entry to determine if it should have been invalidated. Therefore, it is
+    /// guaranteed that these methods must not return invalidated values.
     ///
     /// Note that you must call
     /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures]
@@ -1693,8 +1703,10 @@ where
     /// popularity estimator of keys so that it retains the client activities of
     /// trying to retrieve an item.
     ///
-    /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures
-    /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
+    /// [support-invalidation-closures]:
+    ///     ./struct.CacheBuilder.html#method.support_invalidation_closures
+    /// [invalidation-disabled-error]:
+    ///     ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
     pub fn invalidate_entries_if<F>(&self, predicate: F) -> Result<PredicateId, PredicateError>
     where
         F: Fn(&K, &V) -> bool + Send + Sync + 'static,

src/sync/segment.rs

@@ -503,10 +503,14 @@ where
 
     /// Discards all cached values.
     ///
-    /// This method returns immediately and a background thread will evict all the
-    /// cached values inserted before the time when this method was called. It is
-    /// guaranteed that the `get` method must not return these invalidated values
-    /// even if they have not been evicted.
+    /// This method returns immediately by just setting the current time as the
+    /// invalidation time. `get` and other retrieval methods are guaranteed not to
+    /// return the entries inserted before or at the invalidation time.
+    ///
+    /// The actual removal of the invalidated entries is done as a maintenance task
+    /// driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
     /// Like the `invalidate` method, this method does not clear the historic
     /// popularity estimator of keys so that it retains the client activities of
@@ -519,15 +523,21 @@ where
 
     /// Discards cached values that satisfy a predicate.
     ///
-    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. This
-    /// method returns immediately and a background thread will apply the closure to
-    /// each cached value inserted before the time when `invalidate_entries_if` was
-    /// called. If the closure returns `true` on a value, that value will be evicted
-    /// from the cache.
+    /// `invalidate_entries_if` takes a closure that returns `true` or `false`. The
+    /// closure is called against each cached entry inserted before or at the time
+    /// when this method was called. If the closure returns `true` that entry will be
+    /// evicted from the cache.
+    ///
+    /// This method returns immediately by not actually removing the invalidated
+    /// entries. Instead, it just sets the predicate to the cache with the time when
+    /// this method was called. The actual removal of the invalidated entries is done
+    /// as a maintenance task driven by a user thread. For more details, see
+    /// [the Maintenance Tasks section](../index.html#maintenance-tasks) in the crate
+    /// level documentation.
     ///
-    /// Also the `get` method will apply the closure to a value to determine if it
-    /// should have been invalidated. Therefore, it is guaranteed that the `get`
-    /// method must not return invalidated values.
+    /// Also the `get` and other retrieval methods will apply the closure to a cached
+    /// entry to determine if it should have been invalidated. Therefore, it is
+    /// guaranteed that these methods must not return invalidated values.
     ///
     /// Note that you must call
     /// [`CacheBuilder::support_invalidation_closures`][support-invalidation-closures]
@@ -540,8 +550,10 @@ where
     /// popularity estimator of keys so that it retains the client activities of
     /// trying to retrieve an item.
     ///
-    /// [support-invalidation-closures]: ./struct.CacheBuilder.html#method.support_invalidation_closures
-    /// [invalidation-disabled-error]: ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
+    /// [support-invalidation-closures]:
+    ///     ./struct.CacheBuilder.html#method.support_invalidation_closures
+    /// [invalidation-disabled-error]:
+    ///     ../enum.PredicateError.html#variant.InvalidationClosuresDisabled
     pub fn invalidate_entries_if<F>(&self, predicate: F) -> Result<(), PredicateError>
     where
         F: Fn(&K, &V) -> bool + Send + Sync + 'static,