Use initial revision to fence leader updates in partition leaders table #9300
Conversation
mmaslankaprv
force-pushed
the
fix-metadata-dissemination-revision
branch
from
ba378d0 to
07b46c7
Compare
mmaslankaprv
force-pushed
the
fix-metadata-dissemination-revision
branch
2 times, most recently
from
044ecf8 to
9a257f8
Compare
| @@ -86,6 +91,7 @@ class partition_leaders_table { | |||
| void update_partition_leader( | |||
| const model::ntp&, | |||
| model::revision_id, | |||
| model::initial_revision_id, | |||
There was a problem hiding this comment.
I wonder if using model::initial_revision_id is correct here. AFAIR we introduced this to remember the revision of the topic that we restored from S3, which can be from a different cluster.
There was a problem hiding this comment.
the initial_revision changes every time the whole topic is deleted, and this is what we want to fence the leadership table updates with
| @@ -86,6 +91,7 @@ class partition_leaders_table { | |||
| void update_partition_leader( | |||
| const model::ntp&, | |||
| model::revision_id, | |||
There was a problem hiding this comment.
Do we need both revision_id and initial_revision_id fields? Perhaps letting health monitor and metadata dissemination service choose which revision to use for the update will be less invasive?
There was a problem hiding this comment.
i moved this into partition_leaders_table to be able to precisely control the moment when we switch from revision to initial revision as we must be certain that the revision is not updated with revision_id, after we enabled the initial revision feature as it is always greater than equal to initial_revision
mmaslankaprv
force-pushed
the
fix-metadata-dissemination-revision
branch
4 times, most recently
from
ee1ae50 to
9563860
Compare
Partition leaders table uses partition revision to fence leader metadata update. Introduced feature enabling use of partition initial revision in feature table. Signed-off-by: Michal Maslanka <michal@redpanda.com>
When propagating leadership information Redpanda include a partition revision id to fence the updates coming from older instance of a partition with the same ntp. The revision that is used in the leader table is increased every time the partition configuration change. Added an initial revision id to the leadership metadata. Initial revision changes when partition is recreated but not when it is moved. Signed-off-by: Michal Maslanka <michal@redpanda.com>
…ates Using `initial_revision_id` to fence leadership updates instead of using a revision coming from configuration id. In current implementation leadership update might have been lost due to the fact that the partition log wasn't fully up to date with the leader hence the revision of configuration wasn't yet updated. Using initial revision that changes only when topic is deleted and then created again but not when the partition moves make the implementation correct as the fencing will only happen for partition being part of different topic instances. Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
fix-metadata-dissemination-revision
branch
from
9563860 to
9a0a9ea
Compare
Using
initial_revision_idto fence leadership updates instead of usinga revision coming from configuration id. In current implementation
leadership update might have been lost due to the fact that the
partition log wasn't fully up to date with the leader hence the revision
of configuration wasn't yet updated. Using initial revision that changes
only when topic is deleted and then created again but not when the
partition moves make the implementation correct as the fencing will only
happen for partition being part of different topic instances.
Backports Required
Release Notes
Improvements