westend: Gossamer fails with Storage root must match that calculated at block #14576856

[EclesioMeloJunior]

Describe the bug

• After Gossamer detecting the runtime upgrade

2024-01-18T17:30:43Z INFO 🔄 detected runtime code change, upgrading with block 0x0283b22f28a02825206b5cfd1668e117c4c1d3b49ec02a5b4731aff2c3ac1c40 from previous code hash 0xac5c0d2e187812ca20d6ea0c91fcc67d7eb4feb33edbfd4ddc0abf93bd4ac1b6 to new code hash 0x6eb47852c4f8f8b1ec2cb6084564f88c353f5d6d3a5edd1c85dcd744847652cb... pkg=state

It fails with:

January 18, 2024 at 13:30 (UTC-4:00) | 2024-01-18T17:30:45Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 14576856: running runtime function: wasm error: unreachable | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
-- | -- | -- | --
January 18, 2024 at 13:30 (UTC-4:00) | wasm stack trace: | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00) | .rust_begin_unwind(i32) | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00) | ._ZN4core9panicking9panic_fmt17h32de9c76c9d5eb0cE(i32,i32) | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00) | ._ZN15frame_executive104Executive$LT$System$C$Block$C$Context$C$UnsignedValidator$C$AllPalletsWithSystem$C$COnRuntimeUpgrade$GT$13execute_block17h8d959aeed6ee6a68E(i32) | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00) | .Core_execute_block(i32,i32) i64 pkg=sync | 22d1a73da4c241c39ca263c523451b1f | gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00) | 2024-01-18T17:30:45Z CRITICAL target=runtime message=panicked at 'Storage root must match that calculated.', /home/builder/cargo/git/checkouts/substrate-7e08433d4c370a21/5a281ce/frame/executive/src/lib.rs:602:9 ext_logging_log_version_1 pkg=runtime module=wazero

January 18, 2024 at 13:30 (UTC-4:00)	2024-01-18T17:30:45Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 14576856: running runtime function: wasm error: unreachable	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	wasm stack trace:	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	.rust_begin_unwind(i32)	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	._ZN4core9panicking9panic_fmt17h32de9c76c9d5eb0cE(i32,i32)	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	._ZN15frame_executive104Executive$LT$System$C$Block$C$Context$C$UnsignedValidator$C$AllPalletsWithSystem$C$COnRuntimeUpgrade$GT$13execute_block17h8d959aeed6ee6a68E(i32)	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	.Core_execute_block(i32,i32) i64 pkg=sync	[22d1a73da4c241c39ca263c523451b1f](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/22d1a73da4c241c39ca263c523451b1f?region=us-east-2)	gossamer-westend
January 18, 2024 at 13:30 (UTC-4:00)	2024-01-18T17:30:45Z CRITICAL target=runtime message=panicked at 'Storage root must match that calculated.', /home/builder/cargo/git/checkouts/substrate-7e08433d4c370a21/5a281ce/frame/executive/src/lib.rs:602:9 ext_logging_log_version_1 pkg=runtime module=wazero

[EclesioMeloJunior]

@dimartiro noticed that Gossamer was performing a V1 behavior over the entire trie when the runtime was calling the function ext_storage_root_2, but that is not the expected behavior since the same trie can have nodes written using V0 and nodes written using V1 and Gossamer should only hash sub-values greater than 32 for the nodes whose were written using V1, even though there were other node's sub-values in the trie greater than 32 but written on V0 (this is called hybrid mode), and any access to these V0 nodes will migrate them to V1.

Trie migration pallet

The trie-migration-pallet, as explained in this issue: paritytech/substrate#10073, is a method that facilitates the process of migrating, re-writing the same value to the same key but this time with state_version set to 1. This does not migrate the entire trie since this will require read-write many keys/values, instead the migration pallet migrates few values per block.

How we validated the hypothesis

Gossamer has a delta tracker that stores key-values inserted by the runtime while interacting with ext_storage_set_version_1 host function marking those nodes as v1 nodes, then when the runtime called the ext_storage_root_2 we checked for the v1 in the node, that was enough to validate and check that this is the right behavior