Struggling to sync - prysm blames geth while geth blames prysm

[lil0uuu]

Describe the bug

Every day my eth node struggles to sync, stuck in some kind of deadlock, where the beacon chain reports:

[2024-03-27 01:55:16] WARN execution: Execution client is not syncing
[2024-03-27 01:55:30] WARN execution: Execution client is not syncing

While execution says:

WARN [03-27|01:51:45.835] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!

After some times (like ten minutes), it sometimes automatically resolve. Other times, I have to restart prysm, geth, or both. When restarting, it sometimes keeps happening.

geth suggests prysm is the problem ethereum/go-ethereum#28899

What's the issue?

geth: v1.13.14
beacon prysm: v5.0.1

Has this worked before in a previous version?

No response

🔬 Minimal Reproduction

No response

Error

No response

Platform(s)

No response

What version of Prysm are you running? (Which release)

No response

Anything else relevant (validator index / public key)?

No response

[nisdas]

Do you have logs @lil0uuu ?

[nonsense]

Same issue, started happening after upgrading Prysm to 5.0.2. Geth is 1.13.14-stable

[Chaz27]

Same issue here, using nethermind. Was happening on 5.0.0, just updated to 5.0.2, hasn't changed.

Prysm: WARN execution: Execution client is not syncing
Nethermind: No incoming messages from the consensus client that is required for sync.

Edit: finally started syncing. Something is very slow when rebooting the client.

[git-ljm]

Same issue
prysm 5.0.2
geth 1.13.14

[nisdas]

This should be fixed in the next release, in the mean time you can try --block-batch-limit=16 for those who are facing long restarts.

[lil0uuu]

Here is a sample of logs I'm seeing on the prysm side, where nothing seems processed for a while, then at 01:55:09 things are moving again

[2024-04-01 01:51:04] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:05] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:05] INFO blockchain: Synced new block block=0xccf367a7... epoch=273783 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761086
[2024-04-01 01:51:05] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0x321f3155984b slot=8761086
syncBitsCount=511 txCount=146 voluntaryExits=4
[2024-04-01 01:51:05] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:06] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:06] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:07] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:07] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:07] WARN execution: Execution client is not syncing
[2024-04-01 01:51:08] INFO blockchain: Synced new block block=0xf78f13a1... epoch=273783 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761087
[2024-04-01 01:51:08] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0x700a83c6477b slot=8761087
syncBitsCount=511 txCount=185 voluntaryExits=5
[2024-04-01 01:51:08] INFO initial-sync: Processing block 0x03c48115... 8761088/8761153 - estimated time remaining 4m20s blocksPerSecond=0.2 peers=16
[2024-04-01 01:51:08] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:08] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:10] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:10] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761153
[2024-04-01 01:51:14] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:17] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:17] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:19] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:21] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:21] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761154
[2024-04-01 01:51:21] WARN execution: Execution client is not syncing
[2024-04-01 01:51:29] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761030 currentSlot=8761155
[2024-04-01 01:51:30] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761029 currentSlot=8761155
[2024-04-01 01:51:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761155
[2024-04-01 01:51:37] WARN execution: Execution client is not syncing
[2024-04-01 01:51:37] INFO p2p: Peer summary activePeers=16 inbound=0 outbound=14
[2024-04-01 01:51:37] WARN sync: Goodbye message from rate-limited peer. error=rate limited
[2024-04-01 01:51:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:38] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:39] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:40] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:41] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:41] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:41] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:42] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:42] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761039 currentSlot=8761156
[2024-04-01 01:51:42] WARN initial-sync: Block processing failure error=could not notify the engine of the new payload: timeout from http.Client: receiv
ed an undefined execution engine error firstSlot=8761074 firstUnprocessed=8761076 lastSlot=8761088 root=0x03c481153e77e9457b690e026af48d4a21bb3dec7313e51
9f7e5aa8ee5cc3c0d
...
...
...
[2024-04-01 01:53:24] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:24] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:24] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:25] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:26] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:27] WARN execution: Execution client is not syncing
[2024-04-01 01:53:27] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:29] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:30] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:31] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:31] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:35] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:35] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761166
[2024-04-01 01:53:37] INFO p2p: Peer summary activePeers=14 inbound=0 outbound=14
[2024-04-01 01:53:41] WARN execution: Execution client is not syncing
[2024-04-01 01:53:55] WARN execution: Execution client is not syncing
[2024-04-01 01:54:09] WARN execution: Execution client is not syncing
[2024-04-01 01:54:23] WARN execution: Execution client is not syncing
[2024-04-01 01:53:24] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:24] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:25] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:26] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:27] WARN execution: Execution client is not syncing
[2024-04-01 01:53:27] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:28] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:29] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:30] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:31] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:31] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761165
[2024-04-01 01:53:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:32] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:33] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761165
[2024-04-01 01:53:35] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:35] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:36] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761036 currentSlot=8761166
[2024-04-01 01:53:37] WARN p2p: Attestation is too old to broadcast, discarding it attestationSlot=8761037 currentSlot=8761166
[2024-04-01 01:53:37] INFO p2p: Peer summary activePeers=14 inbound=0 outbound=14
[2024-04-01 01:53:41] WARN execution: Execution client is not syncing
[2024-04-01 01:53:55] WARN execution: Execution client is not syncing
[2024-04-01 01:54:09] WARN execution: Execution client is not syncing
[2024-04-01 01:54:23] WARN execution: Execution client is not syncing
[2024-04-01 01:54:37] WARN execution: Execution client is not syncing
[2024-04-01 01:54:37] INFO p2p: Peer summary activePeers=15 inbound=0 outbound=15
[2024-04-01 01:54:51] WARN execution: Execution client is not syncing
[2024-04-01 01:55:05] WARN execution: Execution client is not syncing
[2024-04-01 01:55:09] INFO initial-sync: Processing block 0x03c48115... 8761088/8761173 - estimated time remaining 28m20s blocksPerSecond=0.1 peers=16
[2024-04-01 01:55:11] INFO blockchain: Synced new block block=0x03c48115... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761088
[2024-04-01 01:55:11] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0xb5b554791093 slot=8761088 syncBitsCount=505 txCount=134 voluntaryExits=3
[2024-04-01 01:55:17] INFO blockchain: Synced new block block=0x7cf16cfc... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761089
[2024-04-01 01:55:17] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0xa37bca4f6bf3 slot=8761089 syncBitsCount=346 txCount=137 voluntaryExits=3
[2024-04-01 01:55:19] WARN execution: Execution client is not syncing
[2024-04-01 01:55:22] INFO blockchain: Synced new block block=0x9905d4f8... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761090
[2024-04-01 01:55:22] INFO blockchain: Finished applying state transition attestations=128 payloadHash=0x2d52ff078f22 slot=8761090 syncBitsCount=479 txCount=137 voluntaryExits=4
[2024-04-01 01:55:27] INFO blockchain: Synced new block block=0x16ec5658... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761091
[2024-04-01 01:55:27] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=4 payloadHash=0x36ddf41e7c3b slot=8761091 syncBitsCount=507 txCount=193 voluntaryExits=3
[2024-04-01 01:55:29] INFO blockchain: Synced new block block=0x56d89355... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761092
[2024-04-01 01:55:29] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0x365e229cf739 slot=8761092 syncBitsCount=500 txCount=133 voluntaryExits=5
[2024-04-01 01:55:33] INFO blockchain: Synced new block block=0x95b183cb... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761093
[2024-04-01 01:55:33] INFO blockchain: Finished applying state transition attestations=128 payloadHash=0x15461e423300 slot=8761093 syncBitsCount=505 txCount=150 voluntaryExits=4
[2024-04-01 01:55:33] WARN execution: Execution client is not syncing
[2024-04-01 01:55:34] INFO blockchain: Synced new block block=0x398980e7... epoch=273784 finalizedEpoch=273782 finalizedRoot=0x38ecc378... slot=8761094
[2024-04-01 01:55:34] INFO blockchain: Finished applying state transition attestations=128 kzgCommitmentCount=6 payloadHash=0x7e4b01db4163 slot=8761094 syncBitsCount=511 txCount=154 voluntaryExits=4

[Fantasim]

Same problem.. Is there a quick fix ?

[Sreeram1993]

Facing same issue ? Any updates yet ?
Versions
prysm - v5.0.0
geth - v1.13.13

[nisdas]

This should be fixed in v5.0.3. Closing this issue unless someone observes that they are still running into this

[enk37]

With 5.0.3 I keep seeing the same problem after upgrade. geth 1.14.3, windows node. It looks like node running for about 1d normally and then beacon node start flooding message repeatedly:

Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577
Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577
Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577
Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577

logs full of these messages, slot number and current slot varies.

Once node restarted, it keeps running for one more day and then story repeats.

[prestonvanloon]

With 5.0.3 I keep seeing the same problem after upgrade. geth 1.14.3, windows node. It looks like node running for about 1d normally and then beacon node start flooding message repeatedly:

Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577 Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577 Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577 Attestation is too old to broadcast, discarding it attestationSlot=9113540 currentSlot=9113577

logs full of these messages, slot number and current slot varies.

Once node restarted, it keeps running for one more day and then story repeats.

These logs can be safely ignored. I am going to propose a PR to remove the log message or push it to debug or trace levels.

[Yolokos]

The same problem with prysm, I'm always getting a 4 WARN log messages with cycle "Execution client is not syncing" waiting 3-5 minutes and after syncing to new blocks. I don't understand how to fix it, but I didn't get last blocks, I have missed about a 10-20 blocks, sometimes I have current block, but not always. Version of my prysm 5.0.4 if you have ideas, solutions it would be excelent

[Brindrajsinh-Chauhan]

Facing similar issue with geth version 1.14.7 and prysm 5.0.3 Any fixes/suggestions/configs needed to fix this?