Testnet node is always ~12 hours and 50 minutes behind

[andreclaro]

Our Blast testnet node is always around ~12 hours and 50 minutes behind the latest block and it is unable to catch up.

Basically, the node is forking from the network and keeps producing blocks...

We tried to recover the node using https://bwarelabs.com/snapshots/aptos snapshot ... it synced until it gets around 12-13 hours behind and it is unable to catch up.

Logs

op-geth

May 26 13:01:13 t-blast-sepolia-01 geth[214131]: INFO [05-26|13:01:13.500] Starting work on payload                 id=0xc1dd33096a3bba4a
May 26 13:01:13 t-blast-sepolia-01 geth[214131]: INFO [05-26|13:01:13.507] Imported new potential chain segment     number=5,997,741 hash=c768f5..a91e02 blocks=1 txs=1  mgas=0.044  elapsed=3.908ms     mgasps=11.225  age=12h51m43s snapdiffs=160.82KiB  triedirty=0.00B
May 26 13:01:13 t-blast-sepolia-01 geth[214131]: INFO [05-26|13:01:13.508] Chain head was updated                   number=5,997,741 hash=c768f5..a91e02 root=ee989b..7690b3 elapsed=1.08981ms   age=12h51m43s
May 26 13:01:13 t-blast-sepolia-01 geth[214131]: INFO [05-26|13:01:13.513] Starting work on payload                 id=0x29b6d8f02d469e66

op-node

May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=warn msg="ignoring batch with mismatching parent hash" batch_index=0 batch_type=SpanBatch batch_timestamp=1,716,682,386 parent_check=0x14b9914b0978090b1e5af03b4209f43f100fff08 origin_check=0xf712ed17dccc2d47056c0972f8adf8e4b158fc04 start_epoch_number=5978140 end_epoch_number=5,978,165 block_count=160 parent_block=0x87bd25972d804dbb1688b0435aa168083dda8b7d1f28af058ef6d3ac223368be
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=warn msg="Dropping batch"                         batch_type=SpanBatch batch_timestamp=1,716,682,386 parent_check=0x14b9914b0978090b1e5af03b4209f43f100fff08 origin_check=0xf712ed17dccc2d47056c0972f8adf8e4b158fc04 start_epoch_number=5978140 end_epoch_number=5,978,165 block_count=160 l2_safe_head=0x87bd25972d804dbb1688b0435aa168083dda8b7d1f28af058ef6d3ac223368be:5997848 l2_safe_head_time=1,716,682,384
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="Generating next batch"                  epoch=0x48ced58945e6858e2d9781a4d465a58799debeebf57f39d93b14d7170bdf72c9:5978146 timestamp=1,716,682,386
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="generated attributes in payload queue"  txs=1   timestamp=1,716,682,386
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="inserted block"                         hash=0x92a87d8ead0fe4ab536780d4a35b1a3b856330ff51a09cb4251e89c10f703f17 number=5,997,849 state_root=0xf42c829509514a16da7eb14ab5d72f1cc414846462d048e1b60ffb684678cfc8 timestamp=1,716,682,386 parent=0x87bd25972d804dbb1688b0435aa168083dda8b7d1f28af058ef6d3ac223368be prev_randao=0x025899a87285786222bd23baeef09805852560c2b180249444370c5d44a71dc8 fee_recipient=0x4200000000000000000000000000000000000011 txs=1   update_safe=true
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="Sync progress"                          reason="processed safe block derived from L1" l2_finalized=0x71f05b6b9f6145d90b43a518fc555fd39eff9a7b49bb2f4481062d816c445942:5997209 l2_safe=0x92a87d8ead0fe4ab536780d4a35b1a3b856330ff51a09cb4251e89c10f703f17:5997849 l2_pending_safe=0x92a87d8ead0fe4ab536780d4a35b1a3b856330ff51a09cb4251e89c10f703f17:5997849 l2_unsafe=0x92a87d8ead0fe4ab536780d4a35b1a3b856330ff51a09cb4251e89c10f703f17:5997849 l2_time=1,716,682,386 l1_derived=0x083f1230cf0324eb4322e65e194520023d5ce1d1c4a4695975ab7e56cdb1f089:5981746
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="Generating next batch"                  epoch=0x48ced58945e6858e2d9781a4d465a58799debeebf57f39d93b14d7170bdf72c9:5978146 timestamp=1,716,682,388
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="generated attributes in payload queue"  txs=1   timestamp=1,716,682,388
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="inserted block"                         hash=0x5d9c9c47981823952983eb7c9694cbc03c0ce3ca7d6ed12274269a39ffc62b7f number=5,997,850 state_root=0xacc563eb5b25d44e801e6a18f8987791c97cbc6bad8916dbf855c16ee3b59cd2 timestamp=1,716,682,388 parent=0x92a87d8ead0fe4ab536780d4a35b1a3b856330ff51a09cb4251e89c10f703f17 prev_randao=0x025899a87285786222bd23baeef09805852560c2b180249444370c5d44a71dc8 fee_recipient=0x4200000000000000000000000000000000000011 txs=1   update_safe=true
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="Sync progress"                          reason="processed safe block derived from L1" l2_finalized=0x71f05b6b9f6145d90b43a518fc555fd39eff9a7b49bb2f4481062d816c445942:5997209 l2_safe=0x5d9c9c47981823952983eb7c9694cbc03c0ce3ca7d6ed12274269a39ffc62b7f:5997850 l2_pending_safe=0x5d9c9c47981823952983eb7c9694cbc03c0ce3ca7d6ed12274269a39ffc62b7f:5997850 l2_unsafe=0x5d9c9c47981823952983eb7c9694cbc03c0ce3ca7d6ed12274269a39ffc62b7f:5997850 l2_time=1,716,682,388 l1_derived=0x083f1230cf0324eb4322e65e194520023d5ce1d1c4a4695975ab7e56cdb1f089:5981746
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="Generating next batch"                  epoch=0x48ced58945e6858e2d9781a4d465a58799debeebf57f39d93b14d7170bdf72c9:5978146 timestamp=1,716,682,390
May 26 13:05:13 t-blast-sepolia-01 node[219483]: t=2024-05-26T13:05:13+0000 lvl=info msg="generated attributes in payload queue"  txs=1   timestamp=1,716,682,390

Configuration

op-geth

    --datadir=/var/lib/blast/data/execution \
    --port=30303 \
    --nat=extip:0.0.0.0 \
    --nodiscover \
    --maxpeers=0 \
    --syncmode=full \
    --gcmode=archive \
    --rollup.sequencerhttp=https://sequencer.s2.testblast.io \
    --rollup.disabletxpoolgossip=true \
    --http \
    --http.addr=0.0.0.0 \
    --http.port=8545 \
    --http.corsdomain="*" \
    --http.vhosts="*" \
    --http.api=eth,web3 \
    --rpc.txfeecap 100 \
    --ws \
    --ws.addr=0.0.0.0 \
    --ws.port=8546 \
    --ws.origins="*" \
    --ws.api=eth,web3 \
    --authrpc.addr=127.0.0.1 \
    --authrpc.vhosts=* \
    --authrpc.port=8551 \
    --authrpc.jwtsecret=/var/lib/blast/jwt.hex \
    --metrics \
    --metrics.addr=0.0.0.0 \
    --metrics.port=6060 \
    --override.canyon=0 \
    --override.ecotone=1713999600 \
    --verbosity=3

op-node

    --l1=ws://IP_ADDRESS:8546 \
    --l1.beacon=http://IP_ADDRESS:5052 \
    --l2=http://127.0.0.1:8551 \
    --l2.jwt-secret=/var/lib/blast/jwt.hex \
    --rpc.addr=127.0.0.1 \
    --rpc.port=9545 \
    --rollup.config=/var/lib/blast/data/consensus/rollup.json \
    --rollup.halt=major \
    --p2p.advertise.ip=IP_ADDRESS \
    --p2p.listen.tcp=30304 \
    --p2p.listen.udp=30304 \
    --p2p.bootnodes=enr:-J-4QM3GLUFfKMSJQuP1UvuKQe8DyovE7Eaiit0l6By4zjTodkR4V8NWXJxNmlg8t8rP-Q-wp3jVmeAOml8cjMj__ROGAYznzb_HgmlkgnY0gmlwhA-cZ_eHb3BzdGFja4X947FQAIlzZWNwMjU2azGhAiuDqvB-AsVSRmnnWr6OHfjgY8YfNclFy9p02flKzXnOg3RjcIJ2YYN1ZHCCdmE,enr:-J-4QDCVpByqQ8nFqCS9aHicqwUfXgzFDslvpEyYz19lvkHLIdtcIGp2d4q5dxHdjRNTO6HXCsnIKxUeuZSPcEbyVQCGAYznzz0RgmlkgnY0gmlwhANiQfuHb3BzdGFja4X947FQAIlzZWNwMjU2azGhAy3AtF2Jh_aPdOohg506Hjmtx-fQ1AKmu71C7PfkWAw9g3RjcIJ2YYN1ZHCCdmE \
    --p2p.priv.path=/var/lib/blast/data/consensus/priv.key \
    --p2p.peerstore.path=/var/lib/blast/data/consensus/peerstore \
    --p2p.discovery.path=/var/lib/blast/data/consensus/discovery \
    --metrics.enabled=true \
    --metrics.addr=0.0.0.0 \
    --metrics.port=6061

Related

Related with the following issues:

• [Node Operator Question] op-geth is consistently 12 hours behind ethereum-optimism/developers#68
• [Node Operator Question] Blast mainnet(OPStack) node sync staying behind for 12 hours ethereum-optimism/developers#275

[andreclaro]

I tried again from the latest snapshot in https://bwarelabs.com/snapshots/blastl2 and the node forked again..

$ curl -s http://localhost:8545/ -X POST -H "Content-Type: application/json" --data '{"method":"eth_getBlockByNumber","params":["latest", false],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":{"baseFeePerGas":"0x105","blobGasUsed":"0x0","difficulty":"0x0","excessBlobGas":"0x0","extraData":"0x","gasLimit":"0x1c9c380","gasUsed":"0x7fcfe","hash":"0x83aebddcd8ea7d65a8655eb9ec0c28254b9b7b4e7588586973f876d1b3b98cd9","logsBloom":"0x000000000000000040000080800000200000000010000000004000220000000080080001000000000002000100000000001000000000000000000200000000000000000000000040000020080000000800008000000040000000000000040000000000000a0000010000000000000800000000000000000000000010000000000000000000000810000800400000000000000000000000000000000000000000000000000000000000400000001000000000000000000000000202000000000000000002000000000001000000000000000000010000001000000040000020000000000000000000000000000001000000000000800000000000000000004100","miner":"0x4200000000000000000000000000000000000011","mixHash":"0x1c23547901b85279613eeb39c15d5aa3c19bcb4cb8d1a5b3adeb8dc500ec9712","nonce":"0x0000000000000000","number":"0x5e42b0","parentBeaconBlockRoot":"0x7a59b34673ce71c12422dfcb8737e8b704188409add82a42892f71088c833c0f","parentHash":"0x3b2339e4e9e03a4dfe1ae654a4929e594c781a6272d9aae202f74bb7ec927f8c","receiptsRoot":"0x2ece4b742701d1d607867dc78fd3f356e91e8fb2ab8497b45bfc39e7bd5a6a1f","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x1876","stateRoot":"0x0e301567f0bb82684d100342ea34911b770d71bc28d2c0106a18a876311e94ef","timestamp":"0x6657f9c0","totalDifficulty":"0x0","transactions":["0x4888677900725a158d383983b945bbf5104a4594bc117fbfe60e971da2ced9d8","0xa7baa841e64db767346f1e5d1eb1d650de6ff6d431da9873f4322b2d8da625f8","0x4ae6272e345b54b0d569089099b1b8ec5371722341e25fbe9d405f381b54089f","0x2067a7b38911c0476060a7e9556cd4c3aea5003d787954f55df3e52148af82a1","0xb7211fb5b8b99357c84afe8323f9cb2135fd9e9697b938dee088e82207badd94","0x1b9e76c281bf94bcf0bbf264651a432e40853f69674346d97f8c9cd560852ecb"],"transactionsRoot":"0xeac309124a120628c80f8d4add92a48262f26a5a563fa855b12ecafc8967d62f","uncles":[],"withdrawals":[],"withdrawalsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"}}

$ curl -s http://localhost:8545/ -X POST -H "Content-Type: application/json" --data '{"method":"eth_getBlockByNumber","params":["latest", false],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":{"baseFeePerGas":"0x106","blobGasUsed":"0x0","difficulty":"0x0","excessBlobGas":"0x0","extraData":"0x","gasLimit":"0x1c9c380","gasUsed":"0x890e7","hash":"0xb631cd788e1425f2dba305f2b52331d6858a4e5a52f830ca52f9fa06339acd67","logsBloom":"0x00000000000000000020000002000000000000000000000000000000000000008100008000000000000000000000000001001004000000800000000000220000000020100000004000000008000820000000200000000000000000000000000000000000020000014000000000000800000000002000100000240010000000000000000100000000020000000000000000000000000000001000000000000100022000000000000008000000000000000000000000000000000000000200000020000002000000004000000000000000000000000000000000008100000820000010000000000000000000000000000000000000880000000000000000000000","miner":"0x4200000000000000000000000000000000000011","mixHash":"0x815cbc78ae73fcbaca7737c79d27a0bec36b316e92219f3354863f55618aad73","nonce":"0x0000000000000000","number":"0x5e42f8","parentBeaconBlockRoot":"0xebef30c9b50528da6c371488ad9864408046404e539758687ca129359a24f04b","parentHash":"0x12db87ca1a81755e08664a1f050bdc4e45f29bd59fbb7606782205c15b2a9241","receiptsRoot":"0xcc1be27776902d0f8d032b8eae7ea09f8779904eb0db012c156919ca30f6cfe5","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x760","stateRoot":"0x8cd50481f73f53fe5566305bf96e542a9f2d092440f5d873caa874668014924a","timestamp":"0x6657fa50","totalDifficulty":"0x0","transactions":["0x6ecae29dbe7e446f39730be70f02ecd6a8328b129e6b431ecc3cc9efdca0bcb7","0xb3eaf49d1b8121b8bf583ad98ed9aa6b4c431b1fba8a6914aa8a25675476a60c","0x83351427cfe2178be1c60a8ffade8b8e4c1c878e9abb0acb3817a9d2c7b063d1","0x33fb400df346bb8343ebb8ef684d956c0b57614526c4fbf78cc535393b7c18a0"],"transactionsRoot":"0x8cfa37724cda4ac8b6a876730f561acdd5126b6bdbc787e9c0eca3d76206d52e","uncles":[],"withdrawals":[],"withdrawalsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"}}

block with "hash":"0xb631cd788e1425f2dba305f2b52331d6858a4e5a52f830ca52f9fa06339acd67" is not a valid blast testnet block

[andreclaro]

tried to use op-wheel to set geth back to the snapshot block,.. however the blast-geth does not support debug_chainConfig method

go run cmd/main.go engine set-forkchoice --unsafe=6177456 --safe=6177456 --finalized=6177456 --engine=http://localhost:8551/ --engine.jwt-secret=/var/lib/blast/jwt.hex
CRIT [06-03|21:33:22.440] Application failed                       message="failed to init Engine version provider: the method debug_chainConfig does not exist/is not available"
exit status 1

[andreclaro]

as per https://safe-violet-16b.notion.site/Blast-Deployment-Docs-b2f2b7b3c9a645fe8d6ce49fb963a467, we need to use sepolia-testnet branch.

"If you’d like to run a sepolia node, you’ll need to use the sepolia-testnet branch of the blast repo."