Stage 1 block verification failed for 5c00…6234: Block(DifficultyOutOfBounds(OutOfBounds { min: None, max: Some(340282366920938463463374607431768211455), found: 340282366920938463463374607431768211456 }))

[Jacob-H5]

Hi, I have a private net with 2 sealer and 4 non-sealing nodes (6 nodes). The non-sealing nodes erorred with the following:

Stage 1 block verification failed for 5c00…6234: Block(DifficultyOutOfBounds(OutOfBounds { min: None, max: Some(340282366920938463463374607431768211455), found: 340282366920938463463374607431768211456 }))

and lost connection two minutes later. After around 15 minutes they reconnected but didn't import any blocks. Deleted the chainfolder on one of the nodes and tried to resync, but the same error occured.

here is my spec:

{
  "name": "POA",
  "engine": {
    "authorityRound": {
      "params": {
        "gasLimitBoundDivisor": "0x400",
        "maxCodeSize": 98304,
        "stepDuration": "1",
        "validators" : {
          "list": [
            "0x0063608C83BAEA08A4b3d33a56C82a63e3e39D6D",
            "0x23ce2f32d13b0e7ef15f93367eb22aa3d7d529e0"
          ]
        }
      }
    }
  },
  "params": {
    "maximumExtraDataSize": "0x20",
    "minGasLimit": "0x1999999999999998",
    "networkID" : "0x76C"
  },
  "genesis": {
    "seal": {
      "authorityRound": {
        "step": "0x0",
        "signature": "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
      }
    },
    "difficulty": "0x20000",
    "gasLimit": "0x1999999999999998"
  },
  "accounts": {
    "0x0000000000000000000000000000000000000001": { "balance": "1", "builtin": { "name": "ecrecover", "pricing": { "linear": { "base": 3000, "word": 0 } } } },
    "0x0000000000000000000000000000000000000002": { "balance": "1", "builtin": { "name": "sha256", "pricing": { "linear": { "base": 60, "word": 12 } } } },
    "0x0000000000000000000000000000000000000003": { "balance": "1", "builtin": { "name": "ripemd160", "pricing": { "linear": { "base": 600, "word": 120 } } } },
    "0x0000000000000000000000000000000000000004": { "balance": "1", "builtin": { "name": "identity", "pricing": { "linear": { "base": 15, "word": 3 } } } },
    "0x0063608C83BAEA08A4b3d33a56C82a63e3e39D6D": { "balance": "160693804425899027554196209234116260252220299378279283530137"},
    "0x00df0ECb56beaB2C51d1C99eCe07e395F73A36fC": { "balance": "160693804425899027554196209234116260252220299378279283530137"},
    "0x23ce2f32d13b0e7ef15f93367eb22aa3d7d529e0": { "balance": "160693804425899027554196209234116260252220299378279283530137"}

  }
}

I added "validateScoreTransition": 1000000 to the spec as a quick workaround.

[keorn]

What are the Parity versions on each of the nodes?

[Jacob-H5]

1.6.6 Beta on one sealing and one non-sealing. The rest is 1.7.0 Unstable of the 12th or 13th of April.

[keorn]

Unstable has been changing the difficulty calculation. I would not recommend running it on the sealing nodes (especially without adjusting the chain spec). You can keep your workaround and make sure to upgrade all your nodes to latest beta before the transition block occurs.

[5chdn]

Yes, 1.7 is no release and receives updates daily. If you plan to stick with nightly builds, make sure you recompile daily, or switch to beta.

[deanstef]

I get the same error after some transactions between nodes.

Stage 1 block verification failed for 6ba6…24a9: Block(DifficultyOutOfBounds(OutOfBounds { min: None, max: Some(340282366920938463463374607431768211455), found: 340282366920938463463374607431768211456 }))

The node who fail can't accept the block and the network goes down. I'm stressing my PoA network with high load with tsung.
Any idea?

[keorn]

Are you using the latest 1.7 release?

[deanstef]

I'm using Parity/v1.7.0-unstable/x86_64-linux-gnu/rustc1.20.0-nightly on each node. I get Parity from git bt cloning the branch v1.7.0 .
In this version i don't use --force-sealing, maybe this is the problem?

[Ochir]

yes, the same bug was reproduced twice with 1.7.0-beta

[Ochir]

As far as I know, @MasX also have had this issue.

[deanstef]

Is with --force-sealing the bug resolved?

[Ochir]

I also don't use "force-sealing" option as well. I will try to use 1.6.6, let see if problem exists with this version.

[deanstef]

Before the 1.7.0 official release i was using 1.6.9 and it works well with and without 'force sealing', but i'm really interested on the new version.

[Ochir]

Issue was reproduced with 1.6.7. The version 1.6.6 works well. However, it is still unclear how to clearly to reproduce this bug. It looks like it happens after some error with network and further disconnect between PoA nodes. I keep further investigation.

[Ochir]

alright, I reproduced this issue with 1.7.2. It happened after several disconnects between 2 nodes and snapshot was created.

[Ochir]

@5chdn what info can I collect to help with resolving this bug? I'm able to reproduce it each time.

[deanstef]

@Ochir how can you reproduce the bug? Just with several network interruptions?

[Ochir]

No, after load testing. It usually happens when 5000 blocks are imported. чт, 5 окт. 2017 г., 17:01 Stefano De Angelis <notifications@github.com>:

…

@Ochir <https://github.com/ochir> how can you reproduce the bug? Just with several network interruptions? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5689 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADFcZ5gFxEG9ISAoto3le7TK2GZx_9mdks5spOEjgaJpZM4Nk4io> .

[Ochir]

As I had already written before, the version 1.6.6 works well after heavy load.

[5chdn]

this happens if two validator nodes are using the same key to author blocks. make sure to use different keys per validator node as described in the wiki.