[Node Operator Question] op-node is not working with read-only ethereum node

[aytunc-tunay]

Issue Description

I'm using kubernetes cluster to host my read-only ethereum node. I want to also host read-only op node. For that I deployed op-node and op-geth to my cluster. I downloaded Bedrock data file (303gb) and started op-geth. Even though, I was able to successfully connect to and query both the op-geth and our own Ethereum node, op-node keep getting error that i add as node logs. You can also find args and command for deployment op-geth and op-node in below.

args:
'--l1=http://our-own-eth-node:8545/'
'--l1.rpckind=any'
'--l2=ws://op-geth:8551'
'--rpc.addr=0.0.0.0'
'--rpc.port=8547'
'--network=mainnet'
'--l2.jwt-secret=/secrets/jwt-secret.txt'
'--metrics.enabled=true'
'--metrics.addr=0.0.0.0'
'--metrics.port=7300'
command:
op-node

args:
'--ws'
'--ws.port=8546'
'--ws.origins='
'--http'
'--http.port=8545'
'--http.vhosts=
'
'--http.corsdomain='
'--authrpc.addr=0.0.0.0'
'--authrpc.port=8551'
'--authrpc.vhosts='
'--verbosity=3'
'--nodiscover'
'--syncmode=full'
'--maxpeers=0'
'--datadir=/root/data'
'--password=/secrets/datadir-secret.txt'
'--http.api=web3,debug,eth,txpool,net,engine'
'--http.addr=0.0.0.0'
'--ws.addr=0.0.0.0'
'--ws.api=debug,eth,txpool,net,engine'
'--authrpc.jwtsecret=/secrets/jwt-secret.txt'
'--rollup.disabletxpoolgossip=true'
'--metrics'
'--metrics.addr=0.0.0.0'
'--metrics.port=7300'
'--rollup.sequencerhttp=https://mainnet-sequencer.optimism.io/'
command:
/usr/local/bin/geth

But after i changed following args, op-node started working.
--l1=https://eth-mainnet.g.alchemy.com/v2/api-key
--l1.rpckind=alchemy

So it seems problem should be related to this read-only node. Can you help me to fix this problem?

Node Logs

Error initializing the rollup node with version v0.0.0-f45fa278-1698865430. Multiple attempts to fetch runtime config data failed with the following errors:

Failed to fetch unsafe block signing address from system config. Attempt to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x791453e0e6436d1cf0d33a0ed89650cac5725aeea00155fc06bc58a968b7a6f7 resulted in "Account not found".

Subsequent attempt at block 0x702d1c65b9445e9477c385d4e80f1bf82f71918c1e1a1493e546e45e9adcd499 also resulted in "Account not found".

Final attempt at block 0xa0e96b1b7c0de62107324a973e4a3a4901922cfc210036c42747ebac02a72e31 resulted in "Account not found".

The operation failed permanently after 5 attempts. The rollup node could not be initialized due to repeated failures in loading the runtime configuration.

[sbvegan]

I spoke to some client engineers and they said this issue is likely from your L1 node in your cluster. Can you confirm that your L1 node is fully synced? It would be helpful to know:

• What type of L1 node you're running
• If there are any notable logs in the L1 node

[sbvegan]

You might be able to get away with --l1.trustrpc, but it likely won't fix it. You can also update your --l1.rpckind to be node specific, this will reduce noisy logs from receipt fetching. The basic option will be a better default option because the receipt fetching there will be supported on all L1 nodes.

[aytunc-tunay]

I'm running Besu as an execution client and Teku as a consensus client on Ethereum Mainnet. All of the our other system parts, seems working without any problem with this ethereum node. Also when i curl like following

curl -X POST
--data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
-H "Content-Type: application/json"
http://our-own-node:8545

I got latest block number which matched with etherscan.

[aytunc-tunay]

It should be accessible and able to fetch data, as other pods in our system can use it without any issues. However, it would be helpful if you could provide me with the specific commands you want to see the results for. I can curl these commands, then copy and paste the responses here for you.

[ajsutton]

Can you make sure you're using the latest version of besu. You may be running into hyperledger/besu#4114 though it was fixed a while back. It may be something else or may have regressed but worth being sure you're up to date as a starting point.

[sbvegan]

Thanks Adrian, I just spoke to proto and he was thinking the same thing. It looks like the receipt format that is being sent from the Besu client is different than what our rollup node is expecting.

[ajsutton]

Just dug a bit deeper and it's a variant of that bug - the fix for it was only applied to eth_getLogs but the problem still occurs with eth_getReceipt which is what op-node is using. I've logged hyperledger/besu#6204

I did add support for debug_getRawReceipts to besu a little while back so you could try using the debug_geth rpc kind as a work around (and it will be faster). The debug rpms are disabled by default in besu though I think so you'd have to enable them with beau's --rpc-http-apis or --rpc-ws-apis option depending on if you're using http or ws to connect.

[sbvegan]

@aytunc-tunay I'm going to mark this as the answer. Please let us know if you'd like to reopen.

[github-actions[bot]]

Hey @aytunc-tunay, We hope your recent question was resolved to your satisfaction. Your feedback is invaluable and helps us improve our support services. Could you spare a moment to fill out a short feedback survey. Thank you for helping us improve our developer community.