[Node Operator Question] Blast mainnet(OPStack) node sync staying behind for 12 hours

[ConnorRepeat]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

When I am syncing Blast Mainnet, the sync always stays in a state that is 12 hours behind. op-node is always in Drop Batch. but strangely, I delete all the data, re-sync, and sometimes it can sync fine. Still, this problem is most likely to appear on new servers.

Protocol Description

• op-node and op-geth https://github.com/blast-io/blast It's all from the official repository. (genesis.json and rollup.json are also in it)
• network is the Blast Mainnet
• chainId: 81457
• L1 chain: Ethereum Mainnet(chainId: 1)

Node Logs

op-geth nothing special, it just happens to stay around 12h (which seems to be the default time window for Sequencer?).

INFO [03-07|17:13:37.190] Starting work on payload id=0x632d822325fbed5a
INFO [03-07|17:13:37.192] Imported new potential chain segment number=474,772 hash=4cb917..14b8b6 blocks=1 txs=1 mgas=0.047 elapsed=1.027ms mgasps=45.718 age=12h4m18s snapdiffs=330.97KiB triedirty=0.00B
INFO [03-07|17:13:37.192] Chain head was updated number=474,772 hash=4cb917..14b8b6 root=788631..bc1fbe elapsed="165.758µs" age=12h4m18s
INFO [03-07|17:13:37.214] Starting work on payload id=0xe4f1b16b6398cbcf
INFO [03-07|17:13:37.216] Imported new potential chain segment number=474,773 hash=f90af8..842f7a blocks=1 txs=1 mgas=0.047 elapsed="840.383µs" mgasps=55.876 age=12h4m16s snapdiffs=331.15KiB triedirty=0.00B
INFO [03-07|17:13:37.216] Chain head was updated number=474,773 hash=f90af8..842f7a root=f6f51f..4e3a82 elapsed="199.381µs" age=12h4m16s

op-node the output is as follows, it always seems to Drop Batch (which should be normal)

INFO [03-07|17:11:37.865] Generating next batch epoch=cd4fd3..4122e6:19378714 timestamp=1,709,759,241
INFO [03-07|17:11:37.866] generated attributes in payload queue txs=1 timestamp=1,709,759,241
INFO [03-07|17:11:37.869] inserted block hash=8b11b2..f5e081 number=474,713 state_root=090114..244aaa timestamp=1,709,759,241 parent=cefa29..5ed565 prev_randao=7784b0..cf0326 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 update_safe=true
INFO [03-07|17:11:37.869] Sync progress reason="processed safe block derived from L1" l2_finalized=0e21a1..a585b0:474203 l2_safe=8b11b2..f5e081:474713 l2_unsafe=8b11b2..f5e081:474713 l2_engineSyncTarget=8b11b2..f5e081:474713 l2_time=1,709,759,241 l1_derived=54dd23..73ac47:19382314
WARN [03-07|17:11:37.869] ignoring batch with mismatching parent hash batch_index=0 batch_timestamp=1,709,759,243 parent_hash=c26961..f4eeda batch_epoch=5502c1..97acef:19378708 txs=10 current_safe_head=8b11b2..f5e081
WARN [03-07|17:11:37.869] Dropping batch batch_timestamp=1,709,759,243 parent_hash=c26961..f4eeda batch_epoch=5502c1..97acef:19378708 txs=10 l2_safe_head=8b11b2..f5e081:474713 l2_safe_head_time=1,709,759,241

Additional Information

I'm using local ETH L1 ipc RPC and it's fast with no rate limiting. My startup command doesn't differ from the one described in the docker-compose file in the Blast official repository(https://github.com/blast-io/deployment/blob/master/docker-compose.yml)

Feedback

No response

[hash1go]

Have you contacted the Blast team?
Since Blast is not Superchain, it's unclear what changes have been made to the OP Stack code.

[ConnorRepeat]

I tried but no response yet, strangely enough, my friend at Base chain (also OPStack chainId 8453) has encountered the same issue.

It's not a very consistent error, but a high probability of encountering it. I do the same thing on 8 servers and about 1 will sync successfully.

So it's confusing, I've found issues associated with the same phenomenon in the OP's history issue, but there's no clear fix, just that it needs to be re-synchronized. I've tried it and it occasionally works without changing any commands, which is confusing.

[ConnorRepeat]

In the example I gave, I use the snapshots provided by Base chain to start the synchronization and I don't encounter this issue at all, but my friend does and so far no solution has been found. So I guess it's a common issue with OPStack, so I'll ask the question here

[hash1go]

hmm

Hello @sbvegan
Do you have any suggestions?

[Chomtana]

You likely to have problem with op-node P2P port exposure

[sbvegan]

Looks like its a chain divergence and you'll probably need to clear your blockchain database and resync #68. Or maybe what @Chomtana mentioned and you're not getting unsafe blocks from the sequencer. Sharing your op-node and op-geth configuration would help

[andreclaro]

@ConnorRepeat , how did you solve this issue? We are now getting similar issue in testnet: blast-io/blast#11

[bensacc]

also getting the same issue on mainnet, trying to spin up a new node. using basically the standard config from https://github.com/blast-io/deployment