sidechain targetA sync fails with HeaderAncestryMismatch

[brenzi]

when testing with polkadot-sdk-v1.8.0, syncing the relaychain doesn't work

worker tries to import block 208 on top of 204, skips intermediate blocks

./integritee-service -c -u ws://172.17.0.1 --target-a-parentchain-rpc-url ws://172.17.0.1 --target-a-parentchain-rpc-port 9999 run --skip-ra --dev --shielding-target target_a &> worker1-rococo.log

[+] Performing a clean reset of the worker
[+] Purge all files from previous runs
[load_pce ../pce_wrapper.cpp:195] Error, call sgx_create_enclave for PCE fail [load_pce], SGXError:200e.
[load_pce ../pce_wrapper.cpp:195] Error, call sgx_create_enclave for PCE fail [load_pce], SGXError:200e.
Worker Config: Config { integritee_rpc_url: "ws://172.17.0.1", integritee_rpc_port: "9944", target_a_parentchain_rpc_url: Some("ws://172.17.0.1"), target_a_parentchain_rpc_port: Some("9999"), target_b_parentchain_rpc_url: None, target_b_parentchain_rpc_port: None, worker_ip: "127.0.0.1", trusted_external_worker_address: None, trusted_worker_port: "2000", untrusted_external_worker_address: None, untrusted_worker_port: "2001", mu_ra_external_address: None, mu_ra_port: "3443", enable_metrics_server: false, metrics_server_port: "8787", untrusted_http_port: "4545", data_dir: "/home/ubuntu/worker/bin", run_config: Some(RunConfig { skip_ra: true, dev: true, shard: None, teeracle_update_interval: None, reregister_teeracle_interval: None, marblerun_base_url: None, shielding_target: Some(TargetA) }) }
[+] Initialize the shard
Successfully initialized shard "2gGAJSVo2EMWMN9bdrxtduCkWvCiG9vBKBfzreaLAF2L"
Integritee Worker for sidechain v0.12.11
  DCAP is enabled
  Production Mode is disabled
  EVM is disabled
MRENCLAVE=2gGAJSVo2EMWMN9bdrxtduCkWvCiG9vBKBfzreaLAF2L
MRENCLAVE in hex "18ea7af839ef4f6879a3a96ec2ffb9b5018f2b265213a2274b71d9312da6dc3d"
MU-RA server listening on 127.0.0.1:3443
Enclave account 5HMWuJkiKmGbzLpKz7DM6QL2LKV8xVQH2nNprfnrLzGU63LF 
[+] Trusted RPC direct invocation server listening on 127.0.0.1:2000
[+] Untrusted RPC server is spawned on: 127.0.0.1:2001 listening to peer sidechain block broadcasts
[Integritee] last synced parentchain block: 0
[!] skipping remote attestation. Registering enclave without attestation report.
[+] Send register enclave extrinsic
[+] send extrinsic: bootstrap funding Enclave from Alice's funds
[+] Enclave registered at block number: 94, hash: 0x6b30ecd7c6fed221dd0fb1662f573258b3abf4fc3ea11d1605db42258483564d
We are the primary worker on this shard and the shard is untouched. Will initialize it
[Integritee] last synced parentchain block: 0
[Integritee:SCV] Finished initializing light client, syncing integritee parentchain...
[Integritee] synced block number: #94
Initializing parentchain TargetA with url: ws://172.17.0.1:9999
[+] Found `WorkerForShard` on parentchain state: AnySigner::Known(MultiSigner::Ed25519(e9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b (5HMWuJki...)))
[+] send extrinsic: bootstrap funding Enclave from Alice's funds
[TargetA] last synced parentchain block: 0
[TargetA] Finished initializing light client, syncing parentchain...
[L1Event:TargetA] Subscribing to selected events
[TargetA] initializing proxied shard vault account now
[L1Event:TargetA] Balances::Endowed
[L1Event:TargetA] BalanceTransfer { from: e9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b (5HMWuJki...), to: 174ea04725b10a72437cf1ba8935d33be290eac92cac8c53720ce1b8d175af56 (5CbGKF3F...), amount: 96534457233 }
[TargetA] initialized shard vault account: : 5CbGKF3FPTm8PYzZ6eqgA6WF85UNLEHbaYnxyXE4TYpdYxEp
[Integritee:SCV] starting block production
[+] Spawning thread for sidechain block production
[L1Event:Integritee] Subscribing to selected events
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
[L1Event:Integritee] EnclaveBridge::ShardConfigUpdated
[L1Event:TargetA] Balances::Reserved
[L1Event:TargetA] Proxy::ProxyAdded
[Sidechain] propose block 1 summary: executed 4, failed 0, from 4 in queue in 699ms
Syncing Parentchains: Integritee: 97 TargetA: Some(207), TargetB: None, Sidechain: 1
[Sidechain] imported block (number: 1, tcalls: 4, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[2024-03-14T13:49:29.025Z ERROR integritee_service::ocall_bridge::worker_on_chain_ocall] Could not send extrinsic to Integritee: Ok("\"0xe1028400e9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b00c92d42ade311064fcc504b7d435b0f09c85d1755c751199cf0ca2521825492248bce7128ab093f2a5aee79eef2b7c53f58bd85fdcd4c71c7c03a7bfe85a10a0b001c00350018ea7af839ef4f6879a3a96ec2ffb9b5018f2b265213a2274b71d9312da6dc3d01000000000000001500000000000000b00cd3a8ff3740727df94ab90b36e4524f5b3108019fb1641669ee7bd013bd07\""), error: RpcClient(SerdeJson(Error("invalid type: null, expected a (both 0x-prefixed or not) hex string or byte array containing 32 bytes", line: 1, column: 4)))
[Sidechain] propose block 2 summary: executed 0, failed 0, from 0 in queue in 681ms
[Sidechain] imported block (number: 2, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 3 summary: executed 0, failed 0, from 0 in queue in 688ms
[Sidechain] imported block (number: 3, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 4 summary: executed 0, failed 0, from 0 in queue in 689ms
[Sidechain] imported block (number: 4, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 5 summary: executed 0, failed 0, from 0 in queue in 692ms
[Sidechain] imported block (number: 5, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 6 summary: executed 0, failed 0, from 0 in queue in 689ms
[Sidechain] imported block (number: 6, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 7 summary: executed 0, failed 0, from 0 in queue in 692ms
[Sidechain] imported block (number: 7, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[Sidechain] propose block 8 summary: executed 0, failed 0, from 0 in queue in 692ms
[L1Event:TargetA] Session::NewSession
[Sidechain] imported block (number: 8, tcalls: 0, author: 0xe9ff76844d59e9540d8239a55f87041e01734d5af0fe5a1d005c7eb3aedd4a8b), based on parentchain block 97
[L1Event:TargetA] Grandpa::NewAuthorities
[L1Event:TargetA] Session::NewSession
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
�[0m�[38;5;8m[�[0m2024-03-14T13:49:36.044284Z �[0m�[1m�[31mERROR�[0m itc_parentchain_light_client::light_validation�[0m�[38;5;8m]�[0m header ancestry mismatch! last imported was block nr 204 with hash 0xeaaf1d986d99ce96443319634a56875b81e86f7ccb6a50f5727b8cb68f75d76b, attempting to import nr 208 with hash 0x20e14158c2770ce91c97f10693298fdb1152155a01d8734b2f5c1d07f4fb91eb and ancestor 0x8d82a1351270ffc882525a2a984634e3e7272b4c7848600b4a535c782cf4eff7
�[0m�[38;5;8m[�[0m2024-03-14T13:49:36.045302Z �[0m�[1m�[31mERROR�[0m itc_parentchain_block_importer::block_importer�[0m�[38;5;8m]�[0m [TargetA] Header submission to light client failed for block number 208 and hash 0x20e14158c2770ce91c97f10693298fdb1152155a01d8734b2f5c1d07f4fb91eb: HeaderAncestryMismatch
[Sidechain] propose block 9 summary: executed 0, failed 0, from 0 in queue in 682ms

[brenzi]

noteworthy: if we start the worker like this:
./integritee-service -c -u ws://172.17.0.1 run --skip-ra --dev &> worker1-rococo.log
we don't have an issue

our parachain collator is based on polkadsot-sdk 1.6.0 and that's also the last version that was successfully tested for the relaychain shielding

[brenzi]

same error when running zombienet with polkadot v1.6.0 binaries

however, the error disappears when restarting the worker (still with clean reset)

upon second run, the error doesn't show and the demo works