[fix][ml] Fix race conditions in RangeCache

[lhotari]

Motivation

The RangeCache class contains several race conditions which cause instability.

When one thread removes the entry and another one uses it, that will become a problem.

The cacheEvictionIntervalMs setting is 10 ms by default. This results in the
RangeCache.evictLEntriesBeforeTimestamp method getting called about 100 times per second.
The default expiration is managedLedgerCacheEvictionTimeThresholdMillis which is 1000 ms by default.

It's also possible that 2 threads remove the entry at the same time. ManagedLedgerImpl.invalidateEntriesUpToSlowestReaderPosition will result in calls to RangeCache.removeRange method. These calls happen independently of the RangeCache.evictLEntriesBeforeTimestamp calls so there's a chance for race conditions.

Modifications

• Add checks to validate that the value hasn't already been released before it is removed.
  • add extra .retain() and .release() calls
  • add a way to validate that the value matches the key
    • this is to prevent a possible race condition where the entry has already been recycled and contains another valid entry which is for a different key
• Make all removals using the remove(key, value) method so that removals remove the correct value exactly once
  • Add a wrapper object for the value to ensure this so that exact identity match is used for the removal.
  • The wrapper instance is recycled to reduce the overhead.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.45833% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 73.27%. Comparing base (bbc6224) to head (2a6c2b1).
Report is 321 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22789      +/-   ##
============================================
- Coverage     73.57%   73.27%   -0.31%     
- Complexity    32624    32648      +24     
============================================
  Files          1877     1889      +12     
  Lines        139502   141659    +2157     
  Branches      15299    15543     +244     
============================================
+ Hits         102638   103800    +1162     
- Misses        28908    29844     +936     
- Partials       7956     8015      +59     

Flag	Coverage Δ
inttests	27.46% <76.04%> (+2.88%)	⬆️
systests	24.82% <72.91%> (+0.50%)	⬆️
unittests	72.27% <86.45%> (-0.58%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
.../org/apache/bookkeeper/mledger/impl/EntryImpl.java	78.31% <0.00%> (-0.17%)	⬇️
...org/apache/bookkeeper/mledger/util/RangeCache.java	89.31% <87.36%> (-6.09%)	⬇️

... and 367 files with indirect coverage changes

[lhotari]

Previous work in RangeCache to prevent race conditions: #15707 and #21302.

[merlimat]

Very nice work

[lhotari]

This PR contained a few issues. I have a follow up PR #22814 to address the issues. Please review