failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block

[akshar]

failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block

Description

• We pulled in latest develop changes to one of our validator and are seeing this error.

Your environment

• OS and version: Ubuntu 20
• version of the Polygon Edge: dd1d9af9d602fe1c88ce473c09259a038d07a16d
• branch that causes this issue: develop

Steps to reproduce

• Create a validator node with latest develop changes

Logs

May 11 14:57:23 ip-172-31-4-5 main[579319]: 2022-05-11T14:57:23.511Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block"

we suspect #513 this PR might have introduced this error

[dbrajovic]

Hey @akshar , thank you for raising the issue.

#513 is what introduced the issue most likely. The returned value from QuorumSize() is different than before for the same size of the validator set. In effect, this means that previously sealed blocks (verified under the old calculation) cannot be verified with the new calculation during bulk sync.

1. Do you have a public JSON-RPC endpoint ?
2. How many validators are in the network ?
3. What is block number that cannot be verified ?
4. If possible, provide the output of eth_getBlockByNumber so I can see who are the validators that committed the seals

[akshar]

Hey @akshar , thank you for raising the issue.

#513 is what introduced the issue most likely. The returned value from QuorumSize() is different than before for the same size of the validator set. In effect, this means that previously sealed blocks (verified under the old calculation) cannot be verified with the new calculation during bulk sync.

1. Do you have a public JSON-RPC endpoint ?

• https://rpc.toronto.sx.technology

2. How many validators are in the network ?

• 5

[VALIDATORS]
ADDRESS
0xA3a585c6d59CCE6aAe7035e8df48b3327cC8BE54
0x7EFF16a34e3433182D636488bc97919b10283F37
0x3e64F88C6C7a1310236B242180c0Ba1409d10F4d
0x9cde4F11EA57163023A9a8a5dF499b27813639EB
0x6c230a5f8dd3413aA04e0860dD225F887b36f101

3. What is block number that cannot be verified ?

May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.307Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=4014669
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.307Z [DEBUG] polygon.consensus.syncer: sync up to block: from=4014670 to=4016340
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.313Z [INFO]  polygon.blockchain: write block: num=4014670 parent=0xb525cb4c5deffdbc069b6215819281d7d86cee0fdba0f3bc78ec3d490cfce6e5
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.314Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block"
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.345Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=4014669
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.345Z [DEBUG] polygon.consensus.syncer: sync up to block: from=4014670 to=4016340
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.355Z [INFO]  polygon.blockchain: write block: num=4014670 parent=0xb525cb4c5deffdbc069b6215819281d7d86cee0fdba0f3bc78ec3d490cfce6e5
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.357Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block"
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.380Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=4014669

May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.380Z [DEBUG] polygon.consensus.syncer: sync up to block: from=4014670 to=4016340
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.390Z [INFO]  polygon.blockchain: write block: num=4014670 parent=0xb525cb4c5deffdbc069b6215819281d7d86cee0fdba0f3bc78ec3d490cfce6e5

May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.392Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block"
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.414Z [DEBUG] polygon.consensus.syncer: fork found: ancestor=4014669

May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.414Z [DEBUG] polygon.consensus.syncer: sync up to block: from=4014670 to=4016340

May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.424Z [INFO]  polygon.blockchain: write block: num=4014670 parent=0xb525cb4c5deffdbc069b6215819281d7d86cee0fdba0f3bc78ec3d490cfce6e5
May 12 21:17:11 ip-172-31-11-88 main[312344]: 2022-05-12T21:17:11.426Z [ERROR] polygon.consensus.ibft: failed to bulk sync: err="failed to write bulk sync blocks: failed to verify the header: not enough seals to seal block"

• we were consistently getting this error on all validator nodes (all of them had develop changes)

4. If possible, provide the output of eth_getBlockByNumber so I can see who are the validators that committed the seals

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "parentHash": "0xb525cb4c5deffdbc069b6215819281d7d86cee0fdba0f3bc78ec3d490cfce6e5",
        "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
        "miner": "0x0000000000000000000000000000000000000000",
        "stateRoot": "0x04812af12359f85e0ac0b7209d951471ccd7e932f6cb346150b7b31b4440e8bc",
        "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
        "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
        "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
        "difficulty": "0x3d424e",
        "totalDifficulty": "0x3d424e",
        "size": "0x3a1",
        "number": "0x3d424e",
        "gasLimit": "0x3938700",
        "gasUsed": "0x0",
        "timestamp": "0x627d3c29",
        "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f90179f86994a3a585c6d59cce6aae7035e8df48b3327cc8be54947eff16a34e3433182d636488bc97919b10283f37943e64f88c6c7a1310236b242180c0ba1409d10f4d949cde4f11ea57163023a9a8a5df499b27813639eb946c230a5f8dd3413aa04e0860dd225f887b36f101b84111f5423531cf6e67c029b4953f6aaaca6e5aa918b315ec582f1704b7ce2b732b5a810af0bcb3f04495a8d2c69403aa7e777722c22a9c2e6838fa87e57faab08201f8c9b841c0e28bb611ee16cba559f10c821eaeb22e135dc58ca13d2e7fcbe8c886721058489fc55a5c9b8fac51fa5b6177d2a979aa4b0e3c582e907a80471bdab6cac86301b8417269fdfda39d5b267701d1c032ea69d2d4fe34671f0ad0ae7e04079ea942249955367649b40127bfe47f732452edae2de43c017199e1b1a35355a3216f240ea401b8417fa48ff637af1c085da5394a2d7c664293e213373d0676a0c1c474ec949583ce0d0b5703e4520b506e9f163bea8808fd99185d9540e43b3420bd220dfd430ccd01",
        "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
        "nonce": "0x0000000000000000",
        "hash": "0x4ab55a70e523e514ea5cd5e97a57ed46f6fda62aa3991062738b541a9fefd82a",
        "transactions": [],
        "uncles": []
    }
}

• we see a new PR related to quorum. Is it related to this issue ? If yes, can we get some context ? @dbrajovic

[dbrajovic]

@akshar

Yes, the linked PR is intended to resolve this issue for all chains that previously sealed blocks under the old Quorum size calculation. If you look at the original PR's additional comments, you'll notice the difference in required number of seals (old Q vs new Q). In other words, starting from a certain block height, the number of seals that are required for a block to reach consensus is more constrained (more seals required).

In #549 , a flag is introduced to specify the block number at which the new calculation was used for the required seals. Essentially, if you specify this value (or a greater one), any new node entering the network should be able to sync up with the entire chain - each block is verified with the number of seals appropriate with the quorum size calculation used at the time.

If you pull fix/ibft-quorum-optimal and test it out (before it is merged to develop), please let us know if it resolves your issue.

[akshar]

So, we pulled hotfix/legacy-quorum changes to all the validator nodes and set the quorum to a future block using ./main ibft quorum --chain genesis.json --from <future-block>. We observed no errors once the blocknumber passed the quorumSizeBlockNum. @dbrajovic

[dbrajovic]

Hey @akshar ,

PR #549 was merged with how-to explanation in the Additional comments section.

If you see the issue persists, please open up a new issue. 🙏