-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAFKA-12571: Eliminate LeaderEpochFileCache constructor dependency on logEndOffset #10426
KAFKA-12571: Eliminate LeaderEpochFileCache constructor dependency on logEndOffset #10426
Conversation
cc @junrao @dhruvilshah3 @satishd: this PR is ready for review. |
61ccacc
to
4208aad
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for the PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kowshik : Thanks for the PR. LGTM. Waiting for the tests to pass.
4208aad
to
c699987
Compare
The rebase has started the Jenkins run now. We can wait for the tests to pass. |
c699987
to
b5d41d2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @kowshik for the PR, LGTM.
@kowshik : Do you know why the tests timed out? Thanks. |
4bd5bc1
to
e01f613
Compare
e01f613
to
53f8433
Compare
@junrao Jenkins doesn't seem to be starting up tests after I rebased this morning. I'm not sure what's going on. I'll try again today. |
@junrao The jenkins tests have finished. I checked the test failures, they seem to be unrelated to this PR:
|
… logEndOffset (apache#10426) This PR is a precursor to the recovery logic refactor work (KAFKA-12553). Problems: For refactoring the recovery logic (KAFKA-12553), we would like to move the logic to initialize LeaderEpochFileCache out of the Log class and into a separate static function. In the future, once we successfully initialize LeaderEpochFileCache outside Log, we will be able pass it as a dependency into both the Log recovery module and Log class constructor. However, currently the LeaderEpochFileCache constructor takes a dependency on logEndOffset (via a callback), which poses the following problems: Blocks the instantiation of LeaderEpochFileCache outside Log class. Because, outside Log the logEndOffset is unavailable to be passed into LeaderEpochFileCache constructor. As a result, this situation blocks the recovery logic (KAFKA-12553) refactor work. It turns out the logEndOffset dependency is used only in 1 of the LeaderEpochFileCache methods: LeaderEpochFileCache.endOffsetFor, and just for 1 particular case. Therefore, it is overkill to pass it in the constructor as a dependency. Also a callback is generally not a neat way to access dependencies and it poses code readability problems too. Solution: This PR modifies the code such that we only pass the logEndOffset as a parameter into LeaderEpochFileCache.endOffsetFor whenever the method is called, thus eliminating the constructor dependency. This will also unblock the recovery logic refactor work (KAFKA-12553). Tests: I have modified the existing tests to suit the above refactor. Reviewers: Dhruvil Shah <dhruvil@confluent.io>, Jun Rao <junrao@gmail.com>
This PR is a precursor to the recovery logic refactor work (KAFKA-12553).
Problems:
For refactoring the recovery logic (KAFKA-12553), we would like to move the logic to initialize
LeaderEpochFileCache
out of theLog
class and into a separate static function. In the future, once we successfully initializeLeaderEpochFileCache
outsideLog
, we will be able pass it as a dependency into both theLog
recovery module andLog
class constructor. However, currently theLeaderEpochFileCache
constructor takes a dependency onlogEndOffset
(via a callback), which poses the following problems:LeaderEpochFileCache
outsideLog
class. Because, outsideLog
thelogEndOffset
is unavailable to be passed intoLeaderEpochFileCache
constructor. As a result, this situation blocks the recovery logic (KAFKA-12553) refactor work.logEndOffset
dependency is used only in 1 of theLeaderEpochFileCache
methods:LeaderEpochFileCache.endOffsetFor
, and just for 1 particular case. Therefore, it is overkill to pass it in the constructor as a dependency. Also a callback is generally not a neat way to access dependencies and it poses code readability problems too.Solution:
This PR modifies the code such that we only pass the
logEndOffset
as a parameter intoLeaderEpochFileCache.endOffsetFor
whenever the method is called, thus eliminating the constructor dependency. This will also unblock the recovery logic refactor work (KAFKA-12553).Tests:
I have modified the existing tests to suit the above refactor.