Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

[Kusama] Upgrade Validator node from 0.8.8 to 0.8.11 meets Encountered consensus error #1305

Closed
jasl opened this issue Jun 23, 2020 · 1 comment

Comments

@jasl
Copy link
Contributor

jasl commented Jun 23, 2020

  • Ubuntu 18.04 latest, 2 Core 8G memory
  • Download Linux binary from GitHub releases page
  • Kusama chain
  • Validator role

Start command:

#!/bin/bash

WORK_PATH=$(dirname $(readlink -f "$0"))
NODE_NAME="my-kusama-validator-1"
CHAIN_NAME="kusama"
RESERVED_NODES="/ip4/10.172.0.13/tcp/30333/p2p/12D3KooWCqsof9wiHJgc62nKz2a8gaP9PSyeKMi8jEcgLB93r5b4 /ip4/10.178.0.3/tcp/30333/p2p/12D3KooWCyh5bS5AVroXvuUJdnByDeGv3akpPTVB6GMNWuiVDBwL"

$WORK_PATH/polkadot \
  --chain $CHAIN_NAME \
  --base-path $WORK_PATH \
  --database paritydb \
  --wasm-execution Compiled \
  --name $NODE_NAME \
  --reserved-only \
  --reserved-nodes $RESERVED_NODES \
  --validator \
  "$@"

I'm trying upgrade our Validator node from 0.8.8 to 0.8.11 by

  • stop polkadot
  • replace old binary with 0.8.11
  • start polkadot

After upgrade, I found in output, every blocks has Encountered consensus error: CannotSign([222, 116, 165, 60, 193, 167, 108, 252, 157, 96, 136, 134, 50, 17, 203, 98, 195, 74, 135, 1, 188, 51, 69, 201, 83, 51, 64, 41, 149, 222, 218, 102], "An unknown keystore error occurred: No such file or directory (os error 2)") log

e.g.

2020-06-23 00:09:24 Starting parachain attestation session on top of parent 0xcfdc1662ff14dfdb70aa69fc6e33743193061a26b3a54c7c1bb0f96505abc86e. Local parachain duty is Some(LocalDuty { validation: Relay, index: 173 })
2020-06-23 00:09:25 ✨ Imported #2864801 (0xff5d…9219)
2020-06-23 00:09:25 ✨ Imported #2864801 (0xcdef…6ad0)
2020-06-23 00:09:25 💤 Idle (2 peers), best: #2864801 (0xcfdc…c86e), finalized #2864798 (0x3e45…b17c), ⬇ 84.4kiB/s ⬆ 79.3kiB/s
2020-06-23 00:09:30 🙌 Starting consensus session on top of parent 0xcfdc1662ff14dfdb70aa69fc6e33743193061a26b3a54c7c1bb0f96505abc86e
2020-06-23 00:09:30 🎁 Prepared block for proposing at 2864802 [hash: 0x3ee2ca8555d3e023316ead3878c5c86de53d332e1271b8e2f7e207f55e0732ed; parent_hash: 0xcfdc…c86e; extrinsics (2): [0x4d4e…3728, 0xcf52…810d]]
2020-06-23 00:09:30 Encountered consensus error: CannotSign([222, 116, 165, 60, 193, 167, 108, 252, 157, 96, 136, 134, 50, 17, 203, 98, 195, 74, 135, 1, 188, 51, 69, 201, 83, 51, 64, 41, 149, 222, 218, 102], "An unknown keystore error occurred: No such file or directory (os error 2)")

I think this is not good, so I quickly downgrade to 0.8.10, the output looks normal, I haven't try 0.8.10 to 0.8.11 yet.

I've some question:

  • Is this error a bug? or my mistake when upgrading polkadot
  • I'm running my validator in this situation for minutes, will my validator getting slash?
@bkchr
Copy link
Member

bkchr commented Jun 23, 2020

The node works as expected, but prints this error per block.

TDLR: We have fixed it on master and you should not use 0.8.11 at the moment. Sorry!

@bkchr bkchr closed this as completed Jun 23, 2020
svyatonik added a commit that referenced this issue Mar 21, 2022
f220d2fcca Polkadot staging update (#1356)
02fd3d497c fix parse_transaction on Rialto+Millau (#1360)
bc191fd9a2 update parity-scale-codec to 3.1.2 (#1359)
a37226e79c update chain versions (#1358)
ff5d539fcb Update Substrate/Polkadot/Cumulus references (#1353)
1581f60cd5 Support dedicated lanes for pallets (#962)
0a7ccf5c57 ignore more "increase" alerts that are sometimes signalling NoData at startup (#1351)
31165127cc added no_stack_overflow_when_decoding_nested_call_during_dispatch test (#1349)
7000619eb8 replace From<>InboundLaneApi with direct storage reads (#1348)
515df10ccc added alerts for relay balances (#1347)
b56f6a87de Mortal conversion rate updater transactions (#1257)
20f2f331ec edition = "2021" (#1346)
99147d4f75 update regex to 1.5.5 (#1345)
686191f379 use DecodeLimit when decoding incoming calls (#1344)
a70c276006 get rid of '[No Data] Messages from Millau to Rialto are not being delivered' warnings (#1342)
01f29b8ac1 fix conversion rate metric in dashboards (#1341)
51c3bf351f Increase rate from metric when estimating fee (#1340)
3bb9c4f68f fix generator scripts to be consistent with updatedrelay output (#1339)
0475a1667b fixed mess with conversion rates (#1338)
d8fdd7d716 synchronize relay cli changes and token swap generator script (#1337)
6e928137a5 fix conversion rate override in token swap (#1336)
62d4a4811d override conversion rate in tokens swap generator (#1335)
ed9e1c839c fi typo in generator script (#1334)
3254b5af7a Override conversion rate when computing message fee (#1261)
66df68b5b8 Revert "Revert "override conversion rate in estimate-message-fee RPC (#1189)" (#1275)" (#1333)
0ca6fc6ef8 fix clippy issues (#1332)
5414b2fffb Reinitialize bridge relay subcommand (#1331)
a63d95ba7d removed extra *_RUNTIME_VERSION consts from relay code (#1330)
59fb18a310 fix typo in alert expression (#1329)
a6267a47ee Using-same-fork metric for finality and complex relay (#1327)
88d684d37e use mortal transactions in transaction resubmitter (#1326)
8ff88b6844 impl Decode for SignedExtensions (otherwise transaction resubmitter panicks) (#1325)
1ed09854f0 Encode and estimate Rococo/Wococo/Kusama/Polkadot messages (#1322)
ddb4517e13 Add some tests to check integrity of chain constants + bridge configuration (#1316)
bdeedb7ab9 Fix issues from cargo deny (#1311)
d3d79d01e0 expose fee multiplier metrics in messages relay (#1312)
c8b3f0ea16 Endow relayer account at target chain in message benchmarks (#1310)
f51ecd92b6 fix benchmarks before using it in Polkadot/Kusama/Rococo runtimes (#1309)
6935c619ad increase relay balance guard limits for Polkadot<>Kusama bridge (#1308)
7e31834c66 Fix mandatory headers scanning in on-demand relay (#1306)
92ddc3ea7a Polkadot-staging update (#1305)
3787193a31 fix session length of Rococo and Wococo (#1304)
eb468d29c0 Revert nightly docker pin (#1301)
e2d4c073e1 Use raw balance value if tokenDecimals property is missing (#1299)
108f4b29d1 Fix ss58 prefixes of Polkadot, Kusama and Westend used by relay (#1298)
64fbd2705e bump chain spec versions (#1297)
5707777b86 Bump Substrate/Polkadot/Cumulus refs (#1295)
29eecdf1fa Merge pull request #1294 from paritytech/polkadot-staging-update
1f0c05368e Relay balance metrics (#1291)
6356bb90b3 when messages pallet is halted, relay shall not submit messages delivery/confirmation transactions (#1289)
800dc2df8d when GRANDPA pallet is halted, relay shall not submit finality transactions (#1288)
3dd8e4f936 disable BEEFY allerts for Rialto (#1285)
f58fed7380 support version mode cli options in send-message subcommand (#1284)
3aac448da3 reuse polkadot-service code (#1273)
2bdbb651e1 replace latest_confirmed_nonce runtime APIs with direct storage reads (#1282)
5f9c6d241f move "common" code of messages pallet benchmarks helpers to the common library (#1281)
173d2d8229 Merge pull request #1280 from paritytech/polkadot-staging-update
8b9c4ec16d do not start spec_version guard when version mode is set to auto (#1278)
e98d682de2 removed extra messages benchmarks (#1279)
c730e25b61 Move benchmarks from Rialto to Millau (#1277)
54146416e7 Merge pull request #1276 from paritytech/polkadot-staging-update
df70118174 Merge branch 'master' into polkadot-staging-update
ed7def64c4 Revert "override conversion rate in estimate-message-fee RPC (#1189)" (#1275)
38c6c3a49f Use "production" floating tag when uilding docker image from version git tags (#1272)
ded9ff6dbb Replace InboundLaneApi::latest_received_nonce with direct storage read (#1269)
f704a741ee Polkadot staging update (#1270)
8c65f0d7ab verify that GRANDPA pallet is not initialized before submitting initialization transaction (#1267)
e7e83d8944 remove OutboundLaneApi::latest_received_nonce (#1262)
9f4b34acf1 bump rococo version (#1263)
82c08c5a87 read latest_generated_nonce directly from storage (#1260)
50ffb5dd08 override conversion rate in estimate-message-fee RPC (#1189)
467ca5ef59 move storage keys computation to primitivs (#1254)
4f9884066b remporary use pinned bridges-ci image in Dockerfile (#1258)
edfcb74e00 Change submit transaction spec_version and transaction_version query from chain (#1248)
4009d970d0 pin bridges-ci image (#1256)
65e51b5e1c decrease startup sleep to 5s for relays and to 120s for generators + remove curl (#1251)
3bc74355d9 Add missing RPC APIs to rialto parachain node (#1250)
80c9429284 Bump relay version to 1.0.0 (#1249)
9ead06af2a runtimes: fix call_size() test (#1245)
4fc8a29357 Use same endowed accounts set on dev/local chains (#1244)
fed54371c2 Refactor message relay helpers (#1234)
a15b4faae7 post-merge build fix (#1243)
52232d8d54 Fix transactions mortality (#1196)
c07bba931f Expose prometheus BEEFY metrics and add them to grafana dashboard (#1242)
f927775bd5 Refactor finality relay helpers (#1220)
7bf76f14a8 Update Rococo/Wococo version + prepare relay for Rococo<>Wococo bridge (#1241)
e860fecd04 Enable offchain indexing for Rialto/Millau nodes (#1239)
04d4d1c6b4 Enable Beefy debug logs in test deployment (#1237)
cd771f1089 Fix storage parameter name computation (#1238)
816ddd2dd2 Integrate BEEFY with Rialto & Millau runtimes (#1227)
d94b62b1ac update dependencies (#1229)
98eb9ee13d Add mut support (#1232)
ffef6f89f9 fixed set_operational in GRANDPA pallet (#1226)
bd2f8bfbd7 Add CODEOWNERS file (#1219)
6b5cf2b591 Unify metric names (#1209)
d1541e797e remove abandoned exchange relay (#1217)
39140d0b34 Remove unused `relays/headers` (#1216)
9bc071d42b Remove unused PoA<>Substrate bridge (#1210)
877e8d01e3 Fix UI deployment. (#1211)
6cd5775ebe Add `AtLeast32BitUnsigned` for MessageLance::SourceChainBalance (#1207)

git-subtree-dir: bridges
git-subtree-split: f220d2fccabbf141101d19456ecb4e3576a1d797
paritytech-processbot bot pushed a commit that referenced this issue Mar 21, 2022
* Squashed 'bridges/' changes from 1602249f0a..f220d2fcca

f220d2fcca Polkadot staging update (#1356)
02fd3d497c fix parse_transaction on Rialto+Millau (#1360)
bc191fd9a2 update parity-scale-codec to 3.1.2 (#1359)
a37226e79c update chain versions (#1358)
ff5d539fcb Update Substrate/Polkadot/Cumulus references (#1353)
1581f60cd5 Support dedicated lanes for pallets (#962)
0a7ccf5c57 ignore more "increase" alerts that are sometimes signalling NoData at startup (#1351)
31165127cc added no_stack_overflow_when_decoding_nested_call_during_dispatch test (#1349)
7000619eb8 replace From<>InboundLaneApi with direct storage reads (#1348)
515df10ccc added alerts for relay balances (#1347)
b56f6a87de Mortal conversion rate updater transactions (#1257)
20f2f331ec edition = "2021" (#1346)
99147d4f75 update regex to 1.5.5 (#1345)
686191f379 use DecodeLimit when decoding incoming calls (#1344)
a70c276006 get rid of '[No Data] Messages from Millau to Rialto are not being delivered' warnings (#1342)
01f29b8ac1 fix conversion rate metric in dashboards (#1341)
51c3bf351f Increase rate from metric when estimating fee (#1340)
3bb9c4f68f fix generator scripts to be consistent with updatedrelay output (#1339)
0475a1667b fixed mess with conversion rates (#1338)
d8fdd7d716 synchronize relay cli changes and token swap generator script (#1337)
6e928137a5 fix conversion rate override in token swap (#1336)
62d4a4811d override conversion rate in tokens swap generator (#1335)
ed9e1c839c fi typo in generator script (#1334)
3254b5af7a Override conversion rate when computing message fee (#1261)
66df68b5b8 Revert "Revert "override conversion rate in estimate-message-fee RPC (#1189)" (#1275)" (#1333)
0ca6fc6ef8 fix clippy issues (#1332)
5414b2fffb Reinitialize bridge relay subcommand (#1331)
a63d95ba7d removed extra *_RUNTIME_VERSION consts from relay code (#1330)
59fb18a310 fix typo in alert expression (#1329)
a6267a47ee Using-same-fork metric for finality and complex relay (#1327)
88d684d37e use mortal transactions in transaction resubmitter (#1326)
8ff88b6844 impl Decode for SignedExtensions (otherwise transaction resubmitter panicks) (#1325)
1ed09854f0 Encode and estimate Rococo/Wococo/Kusama/Polkadot messages (#1322)
ddb4517e13 Add some tests to check integrity of chain constants + bridge configuration (#1316)
bdeedb7ab9 Fix issues from cargo deny (#1311)
d3d79d01e0 expose fee multiplier metrics in messages relay (#1312)
c8b3f0ea16 Endow relayer account at target chain in message benchmarks (#1310)
f51ecd92b6 fix benchmarks before using it in Polkadot/Kusama/Rococo runtimes (#1309)
6935c619ad increase relay balance guard limits for Polkadot<>Kusama bridge (#1308)
7e31834c66 Fix mandatory headers scanning in on-demand relay (#1306)
92ddc3ea7a Polkadot-staging update (#1305)
3787193a31 fix session length of Rococo and Wococo (#1304)
eb468d29c0 Revert nightly docker pin (#1301)
e2d4c073e1 Use raw balance value if tokenDecimals property is missing (#1299)
108f4b29d1 Fix ss58 prefixes of Polkadot, Kusama and Westend used by relay (#1298)
64fbd2705e bump chain spec versions (#1297)
5707777b86 Bump Substrate/Polkadot/Cumulus refs (#1295)
29eecdf1fa Merge pull request #1294 from paritytech/polkadot-staging-update
1f0c05368e Relay balance metrics (#1291)
6356bb90b3 when messages pallet is halted, relay shall not submit messages delivery/confirmation transactions (#1289)
800dc2df8d when GRANDPA pallet is halted, relay shall not submit finality transactions (#1288)
3dd8e4f936 disable BEEFY allerts for Rialto (#1285)
f58fed7380 support version mode cli options in send-message subcommand (#1284)
3aac448da3 reuse polkadot-service code (#1273)
2bdbb651e1 replace latest_confirmed_nonce runtime APIs with direct storage reads (#1282)
5f9c6d241f move "common" code of messages pallet benchmarks helpers to the common library (#1281)
173d2d8229 Merge pull request #1280 from paritytech/polkadot-staging-update
8b9c4ec16d do not start spec_version guard when version mode is set to auto (#1278)
e98d682de2 removed extra messages benchmarks (#1279)
c730e25b61 Move benchmarks from Rialto to Millau (#1277)
54146416e7 Merge pull request #1276 from paritytech/polkadot-staging-update
df70118174 Merge branch 'master' into polkadot-staging-update
ed7def64c4 Revert "override conversion rate in estimate-message-fee RPC (#1189)" (#1275)
38c6c3a49f Use "production" floating tag when uilding docker image from version git tags (#1272)
ded9ff6dbb Replace InboundLaneApi::latest_received_nonce with direct storage read (#1269)
f704a741ee Polkadot staging update (#1270)
8c65f0d7ab verify that GRANDPA pallet is not initialized before submitting initialization transaction (#1267)
e7e83d8944 remove OutboundLaneApi::latest_received_nonce (#1262)
9f4b34acf1 bump rococo version (#1263)
82c08c5a87 read latest_generated_nonce directly from storage (#1260)
50ffb5dd08 override conversion rate in estimate-message-fee RPC (#1189)
467ca5ef59 move storage keys computation to primitivs (#1254)
4f9884066b remporary use pinned bridges-ci image in Dockerfile (#1258)
edfcb74e00 Change submit transaction spec_version and transaction_version query from chain (#1248)
4009d970d0 pin bridges-ci image (#1256)
65e51b5e1c decrease startup sleep to 5s for relays and to 120s for generators + remove curl (#1251)
3bc74355d9 Add missing RPC APIs to rialto parachain node (#1250)
80c9429284 Bump relay version to 1.0.0 (#1249)
9ead06af2a runtimes: fix call_size() test (#1245)
4fc8a29357 Use same endowed accounts set on dev/local chains (#1244)
fed54371c2 Refactor message relay helpers (#1234)
a15b4faae7 post-merge build fix (#1243)
52232d8d54 Fix transactions mortality (#1196)
c07bba931f Expose prometheus BEEFY metrics and add them to grafana dashboard (#1242)
f927775bd5 Refactor finality relay helpers (#1220)
7bf76f14a8 Update Rococo/Wococo version + prepare relay for Rococo<>Wococo bridge (#1241)
e860fecd04 Enable offchain indexing for Rialto/Millau nodes (#1239)
04d4d1c6b4 Enable Beefy debug logs in test deployment (#1237)
cd771f1089 Fix storage parameter name computation (#1238)
816ddd2dd2 Integrate BEEFY with Rialto & Millau runtimes (#1227)
d94b62b1ac update dependencies (#1229)
98eb9ee13d Add mut support (#1232)
ffef6f89f9 fixed set_operational in GRANDPA pallet (#1226)
bd2f8bfbd7 Add CODEOWNERS file (#1219)
6b5cf2b591 Unify metric names (#1209)
d1541e797e remove abandoned exchange relay (#1217)
39140d0b34 Remove unused `relays/headers` (#1216)
9bc071d42b Remove unused PoA<>Substrate bridge (#1210)
877e8d01e3 Fix UI deployment. (#1211)
6cd5775ebe Add `AtLeast32BitUnsigned` for MessageLance::SourceChainBalance (#1207)

git-subtree-dir: bridges
git-subtree-split: f220d2fccabbf141101d19456ecb4e3576a1d797

* fix compilation warnings
pepyakin added a commit that referenced this issue May 4, 2022
d2faa6b2600 Update spec_version for Rococo (#1387)
b3d701124fe Remove support for encoded-call messaging from relay and runtime integration code (#1376)
7f1c4af6650 fix copypaste (#1386)
4e195205ae2 re-enable BEEFY alarms for Rialto (#1385)
072ae865d6b fix for "`/root` is not writable." during deployments startup (#1384)
3ab1810b071 fix daily gitlab build (#1383)
3317b8a6811 Update Substrate/Polkadot refs for latest BEEFY + xcm-v3 capability (#1381)
ebfa9f2fef8 remove vault from ci (#1382)
82136eb42e3 Switch to gav-xcm-v3 branch to be able to test bridges + XCMv3 integration (#1378)
aa8896475b6 Revert "mention encoded-calls-messaging tag"
80c0f7ee05d mention encoded-calls-messaging tag
c7c6f0ce5e8 Revert "add api data() for inbound_lane (#1373)" (#1375)
6ef6bcc0169 FinalityEngine in substrate relay (#1374)
82698e3e082 add api data() for inbound_lane (#1373)
74a48878f86 pub use WeightInfo in Grandpa + Messsages pallets (#1370)
2cc27b7abb5 Update Substrate/Polkadot/Cumulus references (#1364)
9f3ffcd59c7 [ci] Use bridges-ci:production image in the pipeline (#1365)
61766e31f2e Few typos and clippy fixes (#1362)
REVERT: f220d2fccab Polkadot staging update (#1356)
REVERT: 92ddc3ea7a8 Polkadot-staging update (#1305)
REVERT: 29eecdf1fa1 Merge pull request #1294 from paritytech/polkadot-staging-update
REVERT: 173d2d82297 Merge pull request #1280 from paritytech/polkadot-staging-update
REVERT: 54146416e7f Merge pull request #1276 from paritytech/polkadot-staging-update
REVERT: df701181745 Merge branch 'master' into polkadot-staging-update
REVERT: f704a741ee8 Polkadot staging update (#1270)
REVERT: 1602249f0a2 Enable Beefy debug logs in test deployment (#1237)
REVERT: c61d240b474 Fix storage parameter name computation (#1238)
REVERT: 96d3808e88f Integrate BEEFY with Rialto & Millau runtimes (#1227)
REVERT: f75a1bdd9ba update dependencies (#1229)
REVERT: 957da038547 Add mut support (#1232)
REVERT: 8062637289f fixed set_operational in GRANDPA pallet (#1226)
REVERT: 14b36ca4eef Add CODEOWNERS file (#1219)
REVERT: 3bec15766f6 Unify metric names (#1209)
REVERT: 0e839d2423e remove abandoned exchange relay (#1217)
REVERT: 2c91c6815cf Remove unused `relays/headers` (#1216)
REVERT: 80b1e65db82 Remove unused PoA<>Substrate bridge (#1210)
REVERT: f36f76fc2a7 Fix UI deployment. (#1211)
REVERT: fc0b65365bb Add `AtLeast32BitUnsigned` for MessageLance::SourceChainBalance (#1207)

git-subtree-dir: bridges
git-subtree-split: d2faa6b2600fea77d121f3c0767cf59211e597a3
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants