[fix][broker] Avoid block markDeletePosition forward when skip lost entries

[hanmz]

Motivation

The configuration autoSkipNonRecoverableData is designed to turn this feature on if we can accept partial data loss. When some entries is lost, the broker will still work. cursor.readPosition will move to next position for skip these positions. But now we have this problem: cursor.readPosition can move to next positition, but the cursor.markDeletePosition can not forward.

for example:

1. topic-1 has some entries: [1:1],[1:2],[1:3],[1:4],[1:5],[1:6],[1:7],[1:8]
2. entry [1,4] and [1,5] lost
3. cursor read entry from ledger, then send message [1:1],[1:2],[1:3],[1:6],[1:7] to consumer. (Current: readPosition=[1:8], markDeletePosition=[1:1])
4. consumer individual ack [1:1],[1:2],[1:3],[1:6],[1:7] (Current: readPosition=[1:8], markDeletePosition=[1:3])
5. We can't delete the entry [1:4] and [1:5] because these lost. So markDeletePosition will stay at [1:3] and can not move forward to [1:8]

The root cause is: When entries lost, lost entries not be delete from cursor.

Modifications

When skip lost entries, delete these entries from cursor.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: hanmz#4

[Jason918]

overall LGTM.

[Jason918]

LGTM

[hanmz]

/pulsarbot rerun-failure-checks

[hanmz]

/pulsarbot rerun-failure-checks

[github-actions]

The pr had no activity for 30 days, mark with Stale label.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 68.42105% with 6 lines in your changes missing coverage. Please review.

Project coverage is 74.45%. Comparing base (bbc6224) to head (5d8f8e3).
Report is 792 commits behind head on master.

Files with missing lines	Patch %	Lines
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	72.22%	2 Missing and 3 partials ⚠️
...rg/apache/bookkeeper/mledger/impl/OpReadEntry.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21210      +/-   ##
============================================
+ Coverage     73.57%   74.45%   +0.87%     
- Complexity    32624    35113    +2489     
============================================
  Files          1877     1945      +68     
  Lines        139502   147483    +7981     
  Branches      15299    16276     +977     
============================================
+ Hits         102638   109802    +7164     
- Misses        28908    29217     +309     
- Partials       7956     8464     +508     

Flag	Coverage Δ
inttests	27.26% <0.00%> (+2.68%)	⬆️
systests	24.36% <0.00%> (+0.04%)	⬆️
unittests	73.84% <68.42%> (+1.00%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/bookkeeper/mledger/impl/OpReadEntry.java	83.49% <0.00%> (-0.82%)	⬇️
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	80.06% <72.22%> (+0.76%)	⬆️

... and 672 files with indirect coverage changes

[hanmz]

If you have time, please help me take a look, thank you very much. @Technoboy- @poorbarcode