Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node crashes while trying to report equivocation #892

Closed
fkbenjamin opened this issue Dec 13, 2021 · 1 comment
Closed

Node crashes while trying to report equivocation #892

fkbenjamin opened this issue Dec 13, 2021 · 1 comment

Comments

@fkbenjamin
Copy link

One of our validators is in a crash loop after trying to report an equivocation. Here are some logs of when the issue started:

Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ✨ Imported #10500464 (0x56a2…b0b1)
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ♻️  Reorg on #10500464,0x56a2…b0b1 to #10500464,0x95d1…65c7, common ancestor #10500463,0x0904…6b2f
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ✨ Imported #10500464 (0x95d1…65c7)
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 rejecting unsigned report equivocation transaction because it is not local/in-block.
Dec 13 10:25:44  polkadot[1091]: 2021-12-13 10:25:44 💤 Idle (37 peers), best: #10500464 (0x95d1…65c7), finalized #10500460 (0x8ffe…098c), ⬇ 4.8MiB/s ⬆ 2.8MiB/s
Dec 13 10:25:48  polkadot[1091]: 2021-12-13 10:25:48 ✨ Imported #10500465 (0x01e1…297f)
Dec 13 10:25:49  polkadot[1091]: 2021-12-13 10:25:49 💤 Idle (37 peers), best: #10500465 (0x01e1…297f), finalized #10500461 (0xa7ef…afaa), ⬇ 3.4MiB/s ⬆ 1.4MiB/s
Dec 13 10:25:50  polkadot[1091]: 2021-12-13 10:25:50 ✨ Imported #10500464 (0xcaa5…5185)
Dec 13 10:25:54  polkadot[1091]: 2021-12-13 10:25:54 💤 Idle (37 peers), best: #10500465 (0x01e1…297f), finalized #10500461 (0xa7ef…afaa), ⬇ 2.0MiB/s ⬆ 2.1MiB/s
Dec 13 10:25:54  polkadot[1091]: 2021-12-13 10:25:54 ✨ Imported #10500466 (0x0a6c…9367)
Dec 13 10:25:57  polkadot[1091]: 2021-12-13 10:25:57 GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))
Dec 13 10:25:57  polkadot[1091]: 2021-12-13 10:25:57 Essential task `grandpa-voter` failed. Shutting down service.
Dec 13 10:25:58  polkadot[1091]: Error:
Dec 13 10:25:58  polkadot[1091]:    0: Other: Essential task failed.
Dec 13 10:25:58  polkadot[1091]: Backtrace omitted.
Dec 13 10:25:58  polkadot[1091]: Run with RUST_BACKTRACE=1 environment variable to display it.
Dec 13 10:25:58  polkadot[1091]: Run with RUST_BACKTRACE=full to include source snippets.

Since then the node is in a crash loop trying to submit the report:

Dec 13 10:29:58  systemd[1]: Started Kusama Validator.
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ----------------------------
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 This chain is not in any way
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59       endorsed by the
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59      KUSAMA FOUNDATION
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ----------------------------
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 Parity Polkadot
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ✌️  version 0.9.13-7d8f00b90c-x86_64-linux-gnu
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ❤️  by Parity Technologies <admin@parity.io>, 2017-2021
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 📋 Chain specification: Kusama
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 🏷 Node name: 
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 👤 Role: AUTHORITY
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 💾 Database: RocksDb at /home/<user>/.local/share/polkadot/chains/ksmcc3/db/full
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ⛓  Native runtime: kusama-9130 (parity-kusama-0.tx8.au2)
Dec 13 10:30:11  polkadot[1162451]: 2021-12-13 10:30:11 🏷 Local node identity is: <id>
Dec 13 10:30:11  polkadot[1162451]: 2021-12-13 10:30:11 🔍 Discovered new external address for our node: /ip4/<IP>/tcp/55032/p2p/<id>>
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 📦 Highest known block at #10500466
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 〽️ Prometheus exporter started at 0.0.0.0:9615
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 Listening for new connections on 127.0.0.1:9944.
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 👶 Starting BABE Authorship worker
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 Detected precommit equivocation in the finality worker: Equivocation { round_number: 3321, identity: Public(5f3fa3e3eaf17017ee2175>
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 (offchain call) Error submitting a transaction to the pool: Pool(InvalidTransaction(InvalidTransaction::Stale))
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 Error submitting equivocation report: ()
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 Essential task `grandpa-voter` failed. Shutting down service.
Dec 13 10:30:40  polkadot[1162451]: Error:
Dec 13 10:30:40  polkadot[1162451]:    0: Other: Essential task failed.
Dec 13 10:30:40  polkadot[1162451]: Backtrace omitted.
Dec 13 10:30:40  polkadot[1162451]: Run with RUST_BACKTRACE=1 environment variable to display it.
Dec 13 10:30:40  polkadot[1162451]: Run with RUST_BACKTRACE=full to include source snippets.
Dec 13 10:30:40  systemd[1]: kusama.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 10:30:40  systemd[1]: kusama.service: Failed with result 'exit-code'.

After asking for some advise in the Kusama Validator launch, we tried to restart the node without the validator flag to see if that help it sync further blocks but it still tries to submit the report and crashes.
@joslee7410
Copy link

I think we met with the same issue #893 ? XD

@Sophia-Gold Sophia-Gold transferred this issue from paritytech/polkadot Aug 24, 2023
@bkchr bkchr closed this as completed Sep 5, 2023
claravanstaden added a commit to Snowfork/polkadot-sdk that referenced this issue Dec 8, 2023
@claravanstaden @vgeddes
@doubledup @alistair-singh @yrong
Beacon Fuzz testing (paritytech#907)
60a82e0 
* Improve contracts

* More improvements to contracts

* Fix parachain build in proxy-contracts branch (paritytech#889)

* Upgrade ssz_rs crate. (paritytech#880)

* Upgrade ssz_rs crate.

* Upgrade ssz_rs crate.

---------

Co-authored-by: claravanstaden <Cats 4 life!>

* Bump nixpkgs to use its geth package (paritytech#885)

* Bump nixpkgs to use its geth package

* Use source config in PolkadotListener

* Match field order in struct def

* Move info log to relay creation

* Remove unused channel

* Whitespace

* Add context to errors

* Add logs

* Bump node & pnpm in workflow

* Rename locals

* Remove unused variable

* Add Troubleshooting README section

* Fix up .envrc-example files

* Add note about pure shells

* Update cumulus submodule (paritytech#886)

* Inbound queue benchmarks (paritytech#876)

* Start with inbound channel benchmarks.

* Add method to set execution header storage for benchmark test.

* Working on benchmarks

* Basic working version

* Cleanup

* Removes cleanup.

* Adds some comments for Alistair.

* Adds branch name.

* Makes note

* Test transactions

* Cleaning up beacon client deps.

* Clean up comments.

* Tests cleanup.

* Fixes non-benchmark test runs.

* Cleanup.

* Update fixtures and generates benchmarks.

* Revert relayer logs.

* Cleanup BenchmarkHelper impl and inbound queue dependencies.

* fmt

* Cleanup imports.

* Cleanup imports.

* Touch

* Adds weights in inbound queue pallet.

* Fix tests.

* Update cumulus.

---------

Co-authored-by: claravanstaden <Cats 4 life!>

* Fix parachain build

* Move BalanceOf outside of pallet

* remove benchmark for non existing method

* downgrade cargo.lock to match cumulus

* fix benchmarks

---------

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: Alistair Singh <alistair.singh7@gmail.com>

* forge install: openzeppelin-contracts

v4.9.2

* sno-472 rebased (paritytech#890)

* Halting & resuming bridge pallets

* Ignore .env

* Remove .env

* Some polish

* Set owner of bridge pallets

* Upgrade ssz_rs crate. (paritytech#880)

* Upgrade ssz_rs crate.

* Upgrade ssz_rs crate.

---------

Co-authored-by: claravanstaden <Cats 4 life!>

* Update cumulus

* Bump nixpkgs to use its geth package (paritytech#885)

* Bump nixpkgs to use its geth package

* Use source config in PolkadotListener

* Match field order in struct def

* Move info log to relay creation

* Remove unused channel

* Whitespace

* Add context to errors

* Add logs

* Bump node & pnpm in workflow

* Rename locals

* Remove unused variable

* Add Troubleshooting README section

* Fix up .envrc-example files

* Add note about pure shells

* Relax RANDAO_COMMIT_DELAY for local setup

* Update cumulus

* Update cumulus submodule (paritytech#886)

* Update cumulus

* Inbound queue benchmarks (paritytech#876)

* Start with inbound channel benchmarks.

* Add method to set execution header storage for benchmark test.

* Working on benchmarks

* Basic working version

* Cleanup

* Removes cleanup.

* Adds some comments for Alistair.

* Adds branch name.

* Makes note

* Test transactions

* Cleaning up beacon client deps.

* Clean up comments.

* Tests cleanup.

* Fixes non-benchmark test runs.

* Cleanup.

* Update fixtures and generates benchmarks.

* Revert relayer logs.

* Cleanup BenchmarkHelper impl and inbound queue dependencies.

* fmt

* Cleanup imports.

* Cleanup imports.

* Touch

* Adds weights in inbound queue pallet.

* Fix tests.

* Update cumulus.

---------

Co-authored-by: claravanstaden <Cats 4 life!>

* Fix test

* Fix parachain build

* Move BalanceOf outside of pallet

* remove benchmark for non existing method

* downgrade cargo.lock to match cumulus

* fix benchmarks

* Clara/sno 552 (paritytech#887)

* Spacing

* Spacing

* Undo typo.

* Minor updates.

* Adds comment about IrrelevantUpdate.

* One more comment.

* Update error name.

---------

Co-authored-by: claravanstaden <Cats 4 life!>

* Halting & resuming bridge pallets (paritytech#883)

* Halting & resuming bridge pallets

* Ignore .env

* Remove .env

* Some polish

* Set owner of bridge pallets

* Update cumulus

* Relax RANDAO_COMMIT_DELAY for local setup

* Update cumulus

* Update cumulus

* Fix test

* Fix Warnings

* Fix test

* Fix build & format

* Fix benchmark test

* Check for duplicate versions of substrate and polkadot (paritytech#891)

* modified pre-commit

* fixes

* testing

* testing

* testing

* fixed tests

* Format

* Some fix

* Update cumulus

* Update cumulus

---------

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>
Co-authored-by: Alistair Singh <alistair.singh7@gmail.com>

* Proxy contracts Tests fixes (paritytech#892)

* fixed tests

* warnings and imports

* rustfmt

* updated cumulus

* Revert rustfmt (paritytech#896)

* Revert "rustfmt"

This reverts commit b83cec7929cdcc6ac972a2b1cad3a0e1fde81870.

* reverted xcm-builder

* Create agent (paritytech#895)

* base

* removed location conversion

* completed implementation

* remove xcm-builder

* update cumulus

* update cumulus

* use contains_key

* Fix openzeppelin-contracts submodule

---------

Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>

* improve API for sending tokens

* Messy working version.

* Cleanup.

* More cleanup.

* Rollback unnecessary changes.

* Cleanup whitespace.

* Fix tests.

* Fuzz the submit method.

* Revert rebase oopsies.

* More fuzzing.

* More fuzzing.

* Last extrinsic.

* Remove unnecessary dependency.

* Revert readme.

* Fix tests and feature issues.

* Cleanup types.

* More cleanup.

* Call extrinsics directly. Adds readme.

* Adds CI.

* cd to correct dir

* Update CI.

* Correct nightly param.

* Remove runs.

* Own impl for SyncCommittee.

* Remove rng.
Cleans up workflow.

* Last cleanup.

* Update parachain/pallets/ethereum-beacon-client/fuzz/Cargo.toml

Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>

* Adds cargo-fuzz to init script.

* PR comment changes.

* Update parachain/pallets/ethereum-beacon-client/fuzz/src/impls.rs

Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>

* Update rust-toolchain.toml

Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>

* Update .github/workflows/parachain.yml

Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>

* Less runs for shorter Github actions.

---------

Co-authored-by: Vincent Geddes <vincent.geddes@hey.com>
Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com>
Co-authored-by: Alistair Singh <alistair.singh7@gmail.com>
Co-authored-by: Ron <yrong1997@gmail.com>
Co-authored-by: claravanstaden <Cats 4 life!>
helin6 pushed a commit to boolnetwork/polkadot-sdk that referenced this issue Feb 5, 2024
@tgmichel
Use runtime api for base fee (paritytech#892)
3cdb214
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Mar 26, 2024
@HCastano
Bump jsonrpsee to Alpha.3 (paritytech#892)
1610f86 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Mar 27, 2024
@HCastano
Bump jsonrpsee to Alpha.3 (paritytech#892)
e77a969 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@HCastano
Bump jsonrpsee to Alpha.3 (paritytech#892)
74abbe3 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
5721293 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
c689c85 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
d1e83cc 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
5ad94d9 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
89c64c5 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
deaf9b1 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
9c81bb6 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
52b2d7a 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
fb21157 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
8101ff5 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 10, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
9769599 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 10, 2024
@HCastano @serban300
Bump jsonrpsee to Alpha.3 (paritytech#892)
0ada801 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
bkchr pushed a commit that referenced this issue Apr 10, 2024
@HCastano @bkchr
Bump jsonrpsee to Alpha.3 (#892)
b569f20 
* Bump `jsonrpsee` to `alpha.3`

* Arc-ify `WsClient`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bkchr @fkbenjamin @joslee7410