-
Notifications
You must be signed in to change notification settings - Fork 501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(blocksync)!: don't block in blocksync if our voting power is blocking the chain #3406
Conversation
69ccf42
to
0160866
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure about this workaround.
We should now whether we should run block sync outside the protocol. But, ok, it works. But by changing the block Reactor constructor, we breaking a lot of code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would approve, but the >=1/3 vs >2/3 question remains open.
See associated comment (line 515).
.changelog/unreleased/bug-fixes/3406-blocksync-dont-stall-if-blocking-chain.md
Outdated
Show resolved
Hide resolved
.changelog/unreleased/bug-fixes/3406-blocksync-dont-stall-if-blocking-chain.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Daniel <daniel.cason@informal.systems>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job!
Is this safe to backport to 0.38/v1 (post-rc1 v1)? Or better not to because it's too significant behavior change? Not trying to imply here we should backport; in favor of limiting backporting actually. |
I think it's definitely safe to backport, it can't really affect mainnets as you need one Val w/ over 1/3 to do anything. (And it only helps users right now if it's on the 38 line) |
We need to find a solution for v0.37.x too... |
To me, it's a bug, so unless there is big risk identified I'd backport it. Besides, this is clearly holding teams back, which are on |
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems> (cherry picked from commit bd95579) # Conflicts: # internal/blocksync/reactor.go
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems> (cherry picked from commit bd95579) # Conflicts: # .changelog/v0.38.3/bug-fixes/3406-blocksync-dont-stall-if-blocking-chain.md # blocksync/reactor.go # blocksync/reactor_test.go # node/node.go
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems> (cherry picked from commit bd95579) # Conflicts: # blocksync/reactor_test.go # internal/blocksync/reactor.go # node/node.go # node/setup.go
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems>
…king the chain (backport #3406) (#3420) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3406 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: Daniel <daniel.cason@informal.systems>
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems>
…king the chain (#3406) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Daniel <daniel.cason@informal.systems>
…ing the chain (backport #3406) (#3421) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3406 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: Daniel <daniel.cason@informal.systems>
…ing the chain (backport #3406) (#3422) Partially addresses #3415 The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height. However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers. Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following: * _I am a node and I am starting... shall I run blocksync?_ * _Well, looks like I have 1/3 of the voting power (or more) at my current height... so there's no way the chain could advance in my absence... so I don't need to blocksync"_ Explanation of commits: * Commit 1: `e2e` testbed reproducing the issue * Commit 2: commit with a trivial change to trigger `e2e` tests. Check the error: ❌ next to the commit hash (3fb1057) * Commit 3: Tentative fix. Although there is a ❌ next to the commit hash (16a46ea), if you click on it, you'll see that `e2e` are passing now. * Commit 4: revert commit2 * Commit 5: Move the check for "local node is blocking the chain" outside the pool, as suggested by @cason * Commit 6: Fixed unit tests All further commits: addressing other comments and tidying up the code --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3406 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: Daniel <daniel.cason@informal.systems>
Contributes to #3415 This is mainly refactoring to simplify `onlyValidatorIsUs` and `localNodeBlocksTheChain` (since the latter implies the former). It is a follow-up of #3406 (this is the part of #3406 that doesn't need to be backported) --- #### PR checklist - ~[ ] Tests written/updated~ - ~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~ - ~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~ --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
Partially addresses #3415
The a node has no peers, blocksync gets stuck without switching to consesnus, because it needs info from other peers to have an idea of maximum height.
However, there is an edge case (mainly when testing) where a validator might have >2/3 of the voting power and other validators are not started. In this case, we know we are blocking the chain, so we don't need to stay in blockchain if the only condition is that we don't have peers.
Moreover, in order to block a chain, 1/3 of the voting power is enough, so the reasoning of this fix is the following:
Explanation of commits:
e2e
testbed reproducing the issuee2e
tests. Check the error: ❌ next to the commit hash (3fb1057)e2e
are passing now.All further commits: addressing other comments and tidying up the code
PR checklist
.changelog
(we use unclog to manage our changelog)[ ] Updated relevant documentation (docs/
orspec/
) and code comments