Invalid state root in Nethermind v1.29.0 during gas-benchmark

[ohko4711]

Description

While running gas-benchmark, we encountered a failure with Nethermind v1.29.0. Configuration details are as follows:

• Nethermind version: v1.29.0
• chainspec.json and docker-compose.yaml configurations used:
  https://github.com/NethermindEth/gas-benchmarks/tree/main/scripts/nethermind

Error Log

09 Oct 07:21:59 | Processed block 1 (0x9e82dc...6a1887) is invalid:
09 Oct 07:21:59 | - hash: expected 0x6d19e4482a609433ee2dd418b57edd32946665766866ab550b80dbf9e0c8eef7, got 0x9e82dcd4f465b9eb3a9f6decbcde4aaede6ba90d3099b0866456343ba96a1887
09 Oct 07:21:59 | - state root: expected 0xe8d3a308a0d3fdaeed6c196f78aad4f9620b571da6dd5b886e7fa5eba07c83e0, got 0x4a627ed18a234f5e245e4a464965fb41a79285c3183ee6012fed47f4d385b7fd
09 Oct 07:21:59 | - block extra data : 4e65746865726d696e64, UTF8: Nethermind
09 Oct 07:21:59 | Rejected invalid block 1 (0x6d19e4482a609433ee2dd418b57edd32946665766866ab550b80dbf9e0c8eef7),  ExtraData: Nethermind, reason: invalid block after processing
09 Oct 07:21:59 | Suggested block TD: 1, Suggested block IsPostMerge True, Block TD: 1, Block IsPostMerge True
09 Oct 07:21:59 | Encountered exception Nethermind.Blockchain.InvalidBlockException: InvalidStateRoot: State root in header does not match.
   at Nethermind.Consensus.Processing.BlockProcessor.ValidateProcessedBlock(Block suggestedBlock, ProcessingOptions options, Block block, TxReceipt[] receipts) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 267
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 251
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 134 while processing blocks.

Steps to Reproduce

1. Start the Nethermind client using the configuration mentioned above.

2. Send a transaction similar to:

   /nethermind/tools/Nethermind.Tools.Kute/bin/Release/net8.0/Nethermind.Tools.Kute -i tests/GasLimit/GasLimit_40M.txt -s /tmp/jwtsecret -r results-20241009-1.29/nethermind_response_1_GasLimit_40M.txt -a http://localhost:8551
   
   

Additional Information

• v1.28.0 runs normally with the same input.
• Comparing trace-level logs between v1.28.0 and v1.29.0, we noticed that v1.29.0 introduced an eip4788ContractAddress account. nethermind-1.28.log-processed.txt,nethermind-1.29.log-processed.txt

• We suspect this issue might be related to the changes introduced in this PR: https://github.com/NethermindEth/nethermind/pull/7252/files#diff-d2b418bccac416a8665d22299f898d301ac5035c9f6df964ffb7778c8387c74b

Observations

1. The issue occurs with only a change in the Nethermind version number, suggesting a hash calculation problem.
2. Other clients do not exhibit this issue with the same input.
3. Given that the changes between v1.28 and v1.29 are significant, it's challenging to pinpoint the exact cause. However, since our benchmark input payload remains unchanged, we believe there might be a bug introduced in a commit after the v1.28 release.

We kindly request the Nethermind development team to help investigate and locate the source of this issue.

[MarekM25]

Thank you for very detailed report.

@yeerke could you check it? My intuition is telling me that this check might solve the issue.

if (!stateProvider.AccountExists(eip4788Account))
            return;