Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raft: recover from situations where majority of replicas are lost #9096

Closed
bharathv opened this issue Feb 24, 2023 · 3 comments · Fixed by #9785
Closed

raft: recover from situations where majority of replicas are lost #9096

bharathv opened this issue Feb 24, 2023 · 3 comments · Fixed by #9785
Assignees
Labels
kind/enhance New feature or request

Comments

@bharathv
Copy link
Contributor

bharathv commented Feb 24, 2023

Support unclean leader election from a minority replica. This is useful when majority of the replicas are gone (eg: lost nodes) but a user intends to promote (one of) the minority replicas into a leader and up-replicate from there.

The solution needs to handle fencing of lost nodes if they come back up.

@bharathv bharathv changed the title Recover single replica partitions to 3x replication raft: recover from situations where majority of replicas are lost Feb 24, 2023
@bharathv bharathv self-assigned this Feb 28, 2023
@bharathv bharathv added the kind/enhance New feature or request label Mar 3, 2023
@rystsov
Copy link
Contributor

rystsov commented Mar 8, 2023

We should make sure that the recovered partition (with a potential data loss) won't block the tx coordinator if it was involved in an ongoing transaction - #9330

@rystsov
Copy link
Contributor

rystsov commented Mar 8, 2023

We already have some API for that, probably we should add a script to list the transactions on the coordinator, then check the partitions / consumer groups, find the ones which don't have the matching ongoing txes and exclude them from the coordinator.

@bpraseed
Copy link

bpraseed commented Mar 24, 2023

@mattschumpert - this is flagged from chaos testing.
https://docs.google.com/document/d/110nU8vDI8YpQ25IHvNlEuTTK69zplzfkzWeA-TH4LyE/edit#heading=h.yxgertgn7rj8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhance New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants