[OP Stack Deployment] Running two sequencers to solve the single-point failure problem

[luleigreat]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

I have scaned the responses in discord and disccusions about how to avoid single point failure.

I know one of the resolution is the blue green deployment strategy.

And the team have an op-conductor service that can help to manage the blue green deployment workflow.(I don't know how to run the service though.)

My question here is:

1. How to keep two alive(or one dead and one alive?) sequencers in sync when there are transactions.
   I tried to run two sequencers , they can keep in sync with no user transactions. But when I send a transaction to the first sequencer, the blocks of the two sequencers forked, the second sequencer does not contain the tx sent to the first.
2. Both run with a batcher and a proposer ? If both sequencers are alive, will the batcher / proposer send duplicate transactions ?

My running command:
op-node A:

./op-node \
        --l2=ws://localhost:8551 \
        --l2.jwt-secret=./jwt.txt \
        --sequencer.enabled \
        --sequencer.l1-confs=5 \
        --verifier.l1-confs=4 \
        --rollup.config=./rollup.json \
        --rpc.addr=0.0.0.0 \
        --rpc.port=8547 \
        --rpc.enable-admin \
        --p2p.sequencer.key=$GS_SEQUENCER_PRIVATE_KEY \
        --p2p.listen.ip=0.0.0.0 \
        --p2p.listen.tcp=9003 \
        --p2p.listen.udp=9003 \
        --l1=$L1_RPC_URL \
        --l1.beacon=$L1_BECON_URL \
        --l1.rpckind=$L1_RPC_KIND \
        --log.level=info

op-node B:

./op-node \
        --l2=ws://localhost:8551 \
        --l2.jwt-secret=./jwt.txt \
        --sequencer.enabled \
        --sequencer.l1-confs=5 \
        --verifier.l1-confs=4 \
        --rollup.config=./rollup.json \
        --rpc.addr=0.0.0.0 \
        --rpc.port=8547 \
        --p2p.static=/ip4/10.168.3.72/tcp/9003/p2p/16Uiu2HAmCxWv7LqbXQyfJM98aH3m4Atoy7keakTSe7p5YNVtuzoi \
        --p2p.listen.ip=0.0.0.0 \
        --p2p.listen.tcp=9003 \
        --p2p.listen.udp=9003 \
        --p2p.sequencer.key=$GS_SEQUENCER_PRIVATE_KEY \
        --l1=$L1_RPC_URL \
        --l1.beacon=$L1_BECON_URL \
        --l1.rpckind=$L1_RPC_KIND

Logs

No response

Additional Information

No response

Feedback

No response

[sbvegan]

Sharing Transaction Pool

You're going to want to peer your op-geth instances so they can share their transactions.

Notes

You don't want both of these nodes to have the sequencer-enabled at the same time and op-node B's sequencer key has a typo in the env variable.

Documentation:

• Here's some information on configuring peer-to-peer networking : https://docs.optimism.io/builders/node-operators/management/configuration#configuring-peer-to-peer-networking
• go-ethereum networking documentation (it will be the same as downstream op-geth): https://geth.ethereum.org/docs/fundamentals/peer-to-peer#checking-connectivity

[luleigreat]

Thanks for your reply. I have corrected the typo.

But I don't think it's just the p2p problem, I have a replica node that can keep in sync with my sequencer node , and it functions well when I send transactions to it.

I think you didn't look closely at my question.
What I want to know is the Zero Downtime Deployment implementation. When I upgrade my op-node/op-geth of the sequencer, the whole chain will not work for a while. How can I upgrade the sequencer node without a Downtime ?

Option A:
If I run two sequencers with sequencer-enabled at the same time, they will fork when transactions are sent to one of them.

Option B:
If I run a sequencer node and a replica node, when I upgrade the sequencer, there is no sequencer running, the chain is down for a period.

Option C:
Using the op-conductor service，but how to configure，the two sequencers all run with sequencer-enabled ? I know the op-conductor uses the raft CFT consensus,It will re-select the leader when origion leader is off the network.

So my question focused on the following points:

1. Should I run two sequencers at the same time with sequencer-enabled ?
2. If not two sequencers run at the same time，how to implement the Zero Downtime Deployment?
3. Should I run a load-balance for the two sequencers,and the op-batcher,op-proposer, replica op-geth all connect to the load-balance?

[joohhnnn]

For the Zero Downtime Deployment implementation, have you considered using a fixed block height as a boundary? This means you would implement the upgrade in the nodes first, but the execution of the code logic depends on whether the block height has been reached. This allows old and new logic to coexist and makes the transition smoother. If you're running two sequencer nodes, make sure that transactions in both sequencers come from the same gateway. This means you'll need a proxy forwarding service. In other words, users (normal nodes) only need to focus on the sequencer node address without worrying about how many nodes are actually running behind it.

Question

1. You need to separate the concepts of sequencer start and sequencer enable. 'Enable' is a configuration, while 'start' is when it actually begins running. Whether they run simultaneously depends on your specific implementation. If you want to run multiple sequencer instances to enhance robustness, make sure all related logic is compatible. For example, ensure that all sequencer-transferred transactions come from a single wrap service and that only one node is publishing unsafe blocks to the gossip network, etc.

2. As above, or use op-conductor.

3. Unsure. However, op-conductor currently only serves the sequencer.

[kustrun]

To enable multiple sequencers set sequencer.enabled = true on all instances. Then select the sequencer that will be active. Once selected keep its configuration as is, on all other sequencers set sequencer.stopped = true. (https://docs.optimism.io/builders/node-operators/configuration/consensus-config#sequencerenabled). This will enable sequencers on all nodes but not start them.

After you can play around with admin_startSequencer and admin_stopSequencer to select active sequencer (https://docs.optimism.io/builders/node-operators/json-rpc#admin_startsequencer).