Transactions causes split with Parity

[soc1c]

This:

> personal.unlockAccount(eth.coinbase)
Unlock account 0x451122a4f0b6c0876d4b28643b04f53435182f71
Passphrase: 
true
> {from:eth.coinbase, to:'0x8b282d2123e77b9bc4b83eb732175e0406f3354a', value: web3.toWei(100, "ether"), gas:21000});
"0xda80a9ecc8d151a09bf715df035d3b91733ff668c78775e478c22d29b92460da"
> 

Caused this:

2019-06-11 20:11:38  Imported #10981 0x52e6…5f5d (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:11:40  Imported #10982 0xbbd9…4fe9 (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:12:08     3/25 peers   371 KiB chain 4 MiB db 0 bytes queue 539 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:12:10  Stage 5 block verification failed for #10983 (0x39b2…14c4)
Error: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x3519d32b9b94bb3c471361ba8d2b34352cb67c47ad0691a387daa995dfc0f347, found: 0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2 })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
2019-06-11 20:12:10  
Bad block detected: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x3519d32b9b94bb3c471361ba8d2b34352cb67c47ad0691a387daa995dfc0f347, found: 0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2 })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
RLP: f90272f901fba0bbd92f55c08cf2eb89f0a96e70205259538c8f4c6b649762e394b1cdaa9c4fe9a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347943df1fc3a01d8a15a6aa4c95bf12f7631619996f1a0d551430df607393d31ff064ac39a52f0ca79f0bc6e7f53275b85214cf0e9ab01a08d1a812bcce43beddffa59165f59487e04604625cc6dc776753459267638c057a03519d32b9b94bb3c471361ba8d2b34352cb67c47ad0691a387daa995dfc0f347b90100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008317ba32822ae78347e7c4825208845cffeedc80a0d0154ad7318ff84466056ed40353f0f136b3fcdefb81ceea7d4dbbc6622c535b885b9f661ffde1b8aaf871f86f808504a817c800825208948b282d2123e77b9bc4b83eb732175e0406f3354a89056bc75e2d6310000080820a95a0cebcb1a24546dac41c2c949cbde4b4981891d2a38e27702eb73c541d6812980da04205d0dd175a42a3062d055277ac5290c40e45ce8c2710b7b812abfc1efccd5ec0
Header: Header { parent_hash: 0xbbd92f55c08cf2eb89f0a96e70205259538c8f4c6b649762e394b1cdaa9c4fe9, timestamp: 1560276700, number: 10983, author: 0x3df1fc3a01d8a15a6aa4c95bf12f7631619996f1, transactions_root: 0x8d1a812bcce43beddffa59165f59487e04604625cc6dc776753459267638c057, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, extra_data: [], state_root: 0xd551430df607393d31ff064ac39a52f0ca79f0bc6e7f53275b85214cf0e9ab01, receipts_root: 0x3519d32b9b94bb3c471361ba8d2b34352cb67c47ad0691a387daa995dfc0f347, log_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, gas_used: 21000, gas_limit: 4712388, difficulty: 1554994, seal: [[160, 208, 21, 74, 215, 49, 143, 248, 68, 102, 5, 110, 212, 3, 83, 240, 241, 54, 179, 252, 222, 251, 129, 206, 234, 125, 77, 187, 198, 98, 44, 83, 91], [136, 91, 159, 102, 31, 253, 225, 184, 170]], hash: Some(0x39b26e69c709287ebec0007440cc5c1ad73b06a98f4777fe8dc73aaca3d114c4) }
Uncles: 
Transactions:[Tx 0] UnverifiedTransaction { unsigned: Transaction { nonce: 0, gas_price: 20000000000, gas: 21000, action: Call(0x8b282d2123e77b9bc4b83eb732175e0406f3354a), value: 100000000000000000000, data: [] }, v: 2709, r: 93509840040241650265440443168223529970091543351406337441313959923251951081485, s: 29862923765667133073992970208953758916573727840587042244033021498471560826206, hash: 0xda80a9ecc8d151a09bf715df035d3b91733ff668c78775e478c22d29b92460da }

2019-06-11 20:12:38     0/25 peers   371 KiB chain 4 MiB db 0 bytes queue 539 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:13:08     0/25 peers   371 KiB chain 4 MiB db 0 bytes queue 539 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:13:38     0/25 peers   371 KiB chain 4 MiB db 0 bytes queue 539 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:14:08     0/25 peers   371 KiB chain 4 MiB db 0 bytes queue 539 KiB sync  RPC:  0 conn,    0 req/s,    0 µs

[soc1c]

might be config issue if it also happens before block 100

openethereum/parity-ethereum#5850

[soc1c]

eip98 is no longer relevant here

it does not happen before block 100, so it must be an atlantis issue

2019-06-11 20:46:12     1/25 peers   63 KiB chain 23 KiB db 0 bytes queue 17 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:46:28  Imported #61 0xe748…35aa (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:46:31  Imported #65 0xb092…91ff (18 txs, 0.38 Mgas, 1 ms, 2.49 KiB) + another 3 block(s) containing 20 tx(s)
2019-06-11 20:46:36  Imported #66 0x862a…d418 (9 txs, 0.19 Mgas, 0 ms, 1.50 KiB)
2019-06-11 20:46:37  Imported #67 0xbba6…8beb (22 txs, 0.46 Mgas, 1 ms, 2.93 KiB)
2019-06-11 20:46:38  Imported #68 0xdba9…1844 (4 txs, 0.08 Mgas, 0 ms, 0.95 KiB)
2019-06-11 20:46:42  Imported #70 0x717c…f89d (1 txs, 0.02 Mgas, 0 ms, 0.61 KiB) + another 1 block(s) containing 7 tx(s)
2019-06-11 20:46:42     1/25 peers   73 KiB chain 27 KiB db 0 bytes queue 43 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:46:43  Imported #71 0x396d…0029 (13 txs, 0.27 Mgas, 0 ms, 1.94 KiB)
2019-06-11 20:46:44  Imported #72 0x9a5b…a042 (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)

[soc1c]

But reproducible after block 100

2019-06-11 20:47:33  Imported #99 0xa5d4…ee65 (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:47:37  Imported #100 0xcca5…9983 (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:47:38  Imported #101 0x7ea6…a0d8 (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:47:42     1/25 peers   95 KiB chain 34 KiB db 0 bytes queue 43 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-06-11 20:47:44  Imported #102 0x4e2b…abbb (0 txs, 0.00 Mgas, 0 ms, 0.50 KiB)
2019-06-11 20:47:45  Stage 5 block verification failed for #103 (0x6b7e…935d)
Error: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x1998b42e5af412e623cee43c76eb7a47e25a5b07dd8198c64fafbeb9bea04b0e, found: 0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2 })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
2019-06-11 20:47:45  
Bad block detected: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x1998b42e5af412e623cee43c76eb7a47e25a5b07dd8198c64fafbeb9bea04b0e, found: 0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2 })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
RLP: f90271f901f9a04e2be51755baee42478b60abf51b5370deda8107b726e73c3c10b3e28cd4abbba01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794451122a4f0b6c0876d4b28643b04f53435182f71a0f202249a50b9955187322f9c84727042f3a0faf316680ea4096ef19918769d73a0b65c3a94aac74b1b3cb199f3331e2a5bcea5cf30aed6435ac8ed4995f361d178a01998b42e5af412e623cee43c76eb7a47e25a5b07dd8198c64fafbeb9bea04b0eb9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000830219f36783350123825208845cfff75080a03abec07a93236acd94e7c639f761819669c0b0e1516a2d270e48ebcde8a0b6ab8871ff5627c5d9d530f872f87081ea8504a817c800825208948b282d2123e77b9bc4b83eb732175e0406f3354a89056bc75e2d6310000080820a95a0136b2f6372d1e5ab7ad0ee64159a259afb6a35fc29ba2682022d6eb5c3fa5462a04d99a1211bd2a974ac6679faf0473622bfc6298dde6701672dab1f6a43db449ec0
Header: Header { parent_hash: 0x4e2be51755baee42478b60abf51b5370deda8107b726e73c3c10b3e28cd4abbb, timestamp: 1560278864, number: 103, author: 0x451122a4f0b6c0876d4b28643b04f53435182f71, transactions_root: 0xb65c3a94aac74b1b3cb199f3331e2a5bcea5cf30aed6435ac8ed4995f361d178, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, extra_data: [], state_root: 0xf202249a50b9955187322f9c84727042f3a0faf316680ea4096ef19918769d73, receipts_root: 0x1998b42e5af412e623cee43c76eb7a47e25a5b07dd8198c64fafbeb9bea04b0e, log_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, gas_used: 21000, gas_limit: 3473699, difficulty: 137715, seal: [[160, 58, 190, 192, 122, 147, 35, 106, 205, 148, 231, 198, 57, 247, 97, 129, 150, 105, 192, 176, 225, 81, 106, 45, 39, 14, 72, 235, 205, 232, 160, 182, 171], [136, 113, 255, 86, 39, 197, 217, 213, 48]], hash: Some(0x6b7eec1dfceb0ddadcadd115f567da750f51b5c36ab42a3ba2977f135ab9935d) }
Uncles: 
Transactions:[Tx 0] UnverifiedTransaction { unsigned: Transaction { nonce: 234, gas_price: 20000000000, gas: 21000, action: Call(0x8b282d2123e77b9bc4b83eb732175e0406f3354a), value: 100000000000000000000, data: [] }, v: 2709, r: 8783323822218315635313448271360551632141882761663824069174154501233646916706, s: 35099529015592771173124026529989677943990664517189602133222110446678791898270, hash: 0x987df200a5c422e656225548f1c88a531232c217622df5810630cf7942d0abe1 }

[austinabell]

that is strange that the unverified transaction would be from a Call, which wasn't changed during Atlantis and is tested through the Ethereum tests. I wonder if there is an implicit transition that we haven't implemented, like EIP 684, that isn't specifically handled in Parity?

Now looking a bit into it, it seems there may have to be a transition defined in these backwards compatible clients to disable EIP 684 and other unsupported functionality in Atlantis.

[austinabell]

it can't be EIP 649, can it?

[soc1c]

no

[austinabell]

The only reason I think it could be a config discrepancy is because the Ethereum tests submodule all pass (Except for EIP 684 and noted edge case). I will continue to look into it and run my own tests.

[soc1c]

Yes, there is still a chance this is a config issue. I really can't think of anything though

Tried setting up a new testnet but somehow failed to make the clients even talked to each other. Need a break 💥

[austinabell]

I have meticulously gone through config and compared it to existing parity config and I can't see anything that I think is wrong, am very confident our client's config is set up correctly, and went through each commit's changes to anything around Call and did not find any functional differences or implementations. Just an update, I'll start trying to get creative to find out what exactly is going wrong.

[soc1c]

Have you seen this?

multi-geth/multi-geth#103

we can run multi geth on kensington now, this makes it easier to determine which client is wrong

[austinabell]

sounds good! I will look into it in about an hour

[austinabell]

{from:eth.coinbase, to:'0x8b282d2123e77b9bc4b83eb732175e0406f3354a', value: web3.toWei(100, "ether"), gas:21000});

Is this a typo or copy paste issue that eth.sendTransaction( was missing from here?

[austinabell]

{from:eth.coinbase, to:'0x8b282d2123e77b9bc4b83eb732175e0406f3354a', value: web3.toWei(100, "ether"), gas:21000});

Is this a typo or copy paste issue that eth.sendTransaction( was missing from here?

nvm, it omits on long lines

[soc1c]

So, Multi-Geth using the Parity configuration also fails to process these blocks

ERROR[06-13|21:15:44.366] 
########## BAD BLOCK #########
Chain config: {ChainID: 1337 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: <nil> Byzantium: <nil> Disposal: 0 ECIP1017: 2000000 EIP160: 0 ECIP1010PauseBlock: 0 ECIP1010Length: 2000000 Constantinople: <nil> ConstantinopleFix: <nil> Engine: ethash}

Number: 23642
Hash: 0xc9cea5a3186c627b1651ca8ff326f2085940bf606f7bc16a1930d939790711e7
	 0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x01f34b8ea39303d73dadbea62755b606fb180334a04f3f2247fe66afbf251e47 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 


Error: invalid receipt root hash (remote: 67f06bb1eef45fe901f8bf48761c026220e22b45fdfad86a8a5c348b9b349cbe local: 056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2)
##############################
 
WARN [06-13|21:16:18.395] Synchronisation failed, dropping peer    peer=9e7800bd369cf694 err="retrieved hash chain is invalid"

> debug.getBadBlocks()
[{
    block: {
      difficulty: "0x1300fd",
      extraData: "0x",
      gasLimit: "0x47e7c4",
      gasUsed: "0x5208",
      hash: "0xc9cea5a3186c627b1651ca8ff326f2085940bf606f7bc16a1930d939790711e7",
      logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      miner: "0xe04a533c9d7cc59a7420fe8688df4f8ebdadc163",
      mixHash: "0xf62e509298cd13145ab35888a3774c4d5f54cf95250e6cd1154c355900223a30",
      nonce: "0x21eca18f599cf0ae",
      number: "0x5c5a",
      parentHash: "0x5d49123dc3d1cd8890ded2787876af98e13d4ec08759f5cb558f559f460d757c",
      receiptsRoot: "0x67f06bb1eef45fe901f8bf48761c026220e22b45fdfad86a8a5c348b9b349cbe",
      sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      size: "0x274",
      stateRoot: "0xd12c554f09410e92bea5512cab9bf10a86667efe09d72a5c1542d491354e0c50",
      timestamp: "0x5d02a0dd",
      transactions: [{...}],
      transactionsRoot: "0xbc3c40a523a1790d193592a511441731be631b3196d0423c02871ef786c2c210",
      uncles: []
    },
    hash: "0xc9cea5a3186c627b1651ca8ff326f2085940bf606f7bc16a1930d939790711e7",
    rlp: "0xf90271f901fba05d49123dc3d1cd8890ded2787876af98e13d4ec08759f5cb558f559f460d757ca01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794e04a533c9d7cc59a7420fe8688df4f8ebdadc163a0d12c554f09410e92bea5512cab9bf10a86667efe09d72a5c1542d491354e0c50a0bc3c40a523a1790d193592a511441731be631b3196d0423c02871ef786c2c210a067f06bb1eef45fe901f8bf48761c026220e22b45fdfad86a8a5c348b9b349cbeb9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000831300fd825c5a8347e7c4825208845d02a0dd80a0f62e509298cd13145ab35888a3774c4d5f54cf95250e6cd1154c355900223a308821eca18f599cf0aef870f86e018504a817c800825208948b282d2123e77b9bc4b83eb732175e0406f3354a880de0b6b3a764000080820a96a002f48b933389cd57ab784747929eb4be20426a5ed3073c797ee52eac44624e21a0024b4a290496a0de36307646f1539d1b08b3e360657750927296f0b7051ffc71c0"
}]

[meowsbits]

so multi-geth and parity are behaving in agreement about the "bad block" (for them it is not a bad block?)?

have these experiments always been when sending the tx from getc? or does getc also report a bad block when receiving a block with an equivalent tx? (in some of these snippets above, it is not clear to me which client is reporting or doing what, sorry)

[meowsbits]

and this

Transactions:[Tx 0] UnverifiedTransaction { unsigned: Transaction { nonce: 0, gas_price: 2

makes me wonder if the transaction is being properly signed? is the nonce being incremented properly? is eip658 engaged, and properly including the status in the receipt?

[meowsbits]

invalid receipt root hash

[meowsbits]

this might be why

// ApplyTransaction attempts to apply a transaction to the given state database
// and uses the input parameters for its environment.
//
// ApplyTransactions returns the generated receipts and vm logs during the
// execution of the state transition phase.
func ApplyTransaction(config *ChainConfig, bc *BlockChain, gp *GasPool, statedb *state.StateDB, header *types.Header, tx *types.Transaction, usedGas *big.Int) (*types.Receipt, vm.Logs, *big.Int, error) {
	tx.SetSigner(config.GetSigner(header.Number))

	_, gas, failed, err := ApplyMessage(NewEnv(statedb, config, bc, tx, header), tx, gp)
	if err != nil {
		return nil, nil, nil, err
	}

	// Update the state with pending changes
	usedGas.Add(usedGas, gas)
	receipt := types.NewReceipt(statedb.IntermediateRoot(false).Bytes(), usedGas)
	receipt.TxHash = tx.Hash()
	receipt.GasUsed = new(big.Int).Set(gas)
	if MessageCreatesContract(tx) {
		from, _ := tx.From()
		receipt.ContractAddress = crypto.CreateAddress(from, tx.Nonce())
	}

	logs := statedb.GetLogs(tx.Hash())
	receipt.Logs = logs
	receipt.Bloom = types.CreateBloom(types.Receipts{receipt})
	if failed {
		receipt.Status = types.TxFailure
	} else {
		receipt.Status = types.TxSuccess
	}

	glog.V(logger.Debug).Infoln(receipt)

	return receipt, logs, gas, err
}

receipt := types.NewReceipt(statedb.IntermediateRoot(false).Bytes(), usedGas)

where false should be eip161

[soc1c]

so multi-geth and parity are behaving in agreement about the "bad block" (for them it is not a bad block?)?

Yes, a block with a normal transaction signed by Getc is handled the same way in Multi Geth and Parity Ethereum as "bad block" (see above).

have these experiments always been when sending the tx from getc? or does getc also report a bad block when receiving a block with an equivalent tx? (in some of these snippets above, it is not clear to me which client is reporting or doing what, sorry)

I also tested it the other way around. A block with tx signed by Multi Geth causes the same invalid receipt root hash in Getc.

[austinabell]

this might be why
...

@meowsbits Sorry I had not checked on this previously, this, along with the actual problem causing the consensus issue had already been fixed, see PR referenced. Changes have been tested internally and on Kensington and this version has stayed in sync with Kensington since the change

[meowsbits]

@austinabell I'm not sure I can concur on the "already been fixed" clause, when the current PR addressing this is still open, but glad to see that it is being resolved.

[austinabell]

@austinabell I'm not sure I can concur on the "already been fixed" clause, when the current PR addressing this is still open, but glad to see that it is being resolved.

okay, sorry on my poor choice of words :) I really just meant to be succinct to create the connection to the change that solves the specific issue of the thread as to not cause any confusion for any reader.