GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))

[xtony77]

version:

• subgame-runtime v3.0.0

2022-03-08 09:29:45  Detected prevote equivocation in the finality worker: Equivocation { round_number: 18715, identity: Public(72ddcbb2ef0324ba25f97e07fc9c214e4e48c23dbc82731cc4146903c89bf9d8 (5EfKCjVE...)), first: (Prevote { target_hash: 0xfbf2f1a30e88dbd75a51230e1854900db1e9be4b07b4259cb4b39b4e093c6e3e, target_number: 3708928 }, Signature(6a02a874c7bb9d944b0e9b6355c5c353687aae396bf4eec3a0bd2ad4b0342331d8a1be307b51a37495b5dc73317027cc004c13d4fdf7aa7a19d484aaffa03d0e)), second: (Prevote { target_hash: 0xb43ab04edc95a8336daf1d44e2afccbee5953c399b853f6b570f13a7db9284ea, target_number: 3708927 }, Signature(5855318cb72103c02e62fa192359b64cd0987580f797cfad365aa813516f21a28cc4126ff582cf07642e6c73aa09729fd00a29e3896d62945a50e9f933ef210d)) }
2022-03-08 09:29:45  Detected precommit equivocation in the finality worker: Equivocation { round_number: 18715, identity: Public(3e2e90b6a429f3b045e884f7dc1118da906455295764f4507dd0a733abd02f41 (5DUEfARo...)), first: (Precommit { target_hash: 0xfbf2f1a30e88dbd75a51230e1854900db1e9be4b07b4259cb4b39b4e093c6e3e, target_number: 3708928 }, Signature(adc4ebae43121129a922f22da97b1544b9128bebcab7af04e8c5a44e86b498a21e252a6ebb4e9bbde2ee4c55ba3dd800cc93d410ff89ceaa7e3bbf5e28b5b30c)), second: (Precommit { target_hash: 0xb43ab04edc95a8336daf1d44e2afccbee5953c399b853f6b570f13a7db9284ea, target_number: 3708927 }, Signature(3360baa3f942b245acd935cc989fc6de937bc33fd233f3c31686d64402dc226d3d28b5de9a42179ac02b591ea16d02ef9618b7e7441bba7fdffe23cb758c6602)) }
2022-03-08 09:29:45  Detected precommit equivocation in the finality worker: Equivocation { round_number: 18715, identity: Public(72ddcbb2ef0324ba25f97e07fc9c214e4e48c23dbc82731cc4146903c89bf9d8 (5EfKCjVE...)), first: (Precommit { target_hash: 0xfbf2f1a30e88dbd75a51230e1854900db1e9be4b07b4259cb4b39b4e093c6e3e, target_number: 3708928 }, Signature(3f6f4a07dd8f9812b4c62c3f57da640acff9d4598f6b119fc7756a8e311874f714d7a19dc5ddea8c9d573690d335438c6ef9550e7a2741518ee7cd14d393590f)), second: (Precommit { target_hash: 0xb43ab04edc95a8336daf1d44e2afccbee5953c399b853f6b570f13a7db9284ea, target_number: 3708927 }, Signature(17e145b6c14b28cbce1851e1a562235d45e9caef4cb98c5e3e38d2b7b41dbc9a5575f82d997e65f99c3619097512230695c15a37b236109ce028d9b232c55804)) }
2022-03-08 09:29:46  GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))
2022-03-08 09:29:46  Essential task `grandpa-voter` failed. Shutting down service.
Error: Service(Other("Essential task failed."))

I restart the node fixed it, but break too long babe stop work now. 😢
Does anyone know what happened? How can I avoid this error next time?

[xtony77]

BABE stop work and error is:

2022-03-08 18:26:17  Timeout fired waiting for transaction pool at block #3708933. Proceeding with production.
2022-03-08 18:26:18  🎁 Prepared block for proposing at 3708934 [hash: 0x033d5298c8363dbad4022bbcdf988c68133ec141660cbe9b4871fa578409afb6; parent_hash: 0xbd09…634f; extrinsics (1): [0x73de…bdf2]]
2022-03-08 18:26:18  🔖 Pre-sealed block for proposal at 3708934. Hash now 0xb853c4330d88a0414f64b82203e77af2212045b1f55070717ea317670704f765, previously 0x033d5298c8363dbad4022bbcdf988c68133ec141660cbe9b4871fa578409afb6.
2022-03-08 18:26:18  Error with block built on 0xbd09cb0b2d29eb450d11b629a2f0b03e441ea28b091434b704fe1a4a433a634f: ClientImport("Unexpected epoch change")

We are trying the following, does anyone have a better way besides purge the chain?

• fork-off-substrate
• SUBCOMMANDS: export-blocks, import-blocks update: this is invalid.

[bkchr]

CC @andresilva

[andresilva]

Could you provide more details regarding what actually caused this? I assume that you deployed validators with duplicate keys or something like that which led to equivocations. How many authorities do you have and how many of those were equivocating?

Recovering BABE from that state is an elaborate and manual process, you can find some details here: #4464

[xtony77]

We have 12 validators. I have confirmed that the keys are not duplicated, but some validators use the base-path folder copied by other validators in order to quickly catch up with the block height. Could it be related to this?
Only this information so far, not sure what's going on?

[andresilva]

Yes, I assume that is also copying the keystore and hence duplicating keys.

[xtony77]

So if change new validators without duplicating keys, can this be fixed and won't happen again?

[andresilva]

There are two problems here:

• GRANDPA got broken because you ran multiple validators with the same keys. This led to more than 1/3 of the validators equivocating which breaks the protocol.
• There were no blocks authored for an entire epoch, which breaks the BABE protocol and leads to "unexpected epoch change".

Both these things were operational errors so there is nothing to be fixed from our side.

[techwizard210]

Everyone. Please help me.

When I run my substrate node server by running this command "./target/release/node-template --dev --ws-external --rpc-external", server works well and start generating blocks first. But after few hours, my server is been stopped and also stopped generating blocks automatically. How can I solve this problem? Here is Terminal Error.

Best Regards.