" Invalid - block" reported for all blocks starting around 1500 EST on Jan 21

[wga22]

Description
Execution client stopped being able to process blocks (perhaps across all Nethermind client users)

Steps to Reproduce
Run nethermind versions 1.25, 1.25.1 or 1.26

Actual behavior

last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:11 mobius.lan nethermind[8828]: 21 Jan 15:51:11 | Received ForkChoice: Head: 19056963 (0xe7bc99...78e25f), Safe: 19056884 (0xcf8b3e...5255a7),
Finalized: 19056852 (0xeb564e...dce829)
Jan 21 15:51:12 mobius.lan nethermind[8828]: 21 Jan 15:51:12 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:12 mobius.lan nethermind[8828]: 21 Jan 15:51:12 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:15 mobius.lan nethermind[8828]: 21 Jan 15:51:15 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:15 mobius.lan nethermind[8828]: 21 Jan 15:51:15 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:22 mobius.lan nethermind[8828]: 21 Jan 15:51:22 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:22 mobius.lan nethermind[8828]: 21 Jan 15:51:22 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:26 mobius.lan nethermind[8828]: 21 Jan 15:51:26 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:26 mobius.lan nethermind[8828]: 21 Jan 15:51:26 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:30 mobius.lan nethermind[8828]: 21 Jan 15:51:30 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:30 mobius.lan nethermind[8828]: 21 Jan 15:51:30 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:41 mobius.lan nethermind[8828]: 21 Jan 15:51:41 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:41 mobius.lan nethermind[8828]: 21 Jan 15:51:41 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46
Jan 21 15:51:43 mobius.lan nethermind[8828]: 21 Jan 15:51:43 | Removing 1059 out of 11059 peer candidates (candidates cleanup).
Jan 21 15:51:47 mobius.lan nethermind[8828]: 21 Jan 15:51:47 | Received ForkChoice: Head: 19056963 (0xe7bc99...78e25f), Safe: 19056884 (0xcf8b3e...5255a7),
Finalized: 19056852 (0xeb564e...dce829)
Jan 21 15:51:55 mobius.lan nethermind[8828]: 21 Jan 15:51:55 | Received new block: 19056922 (0x76eca7...81d489)
Jan 21 15:51:55 mobius.lan nethermind[8828]: 21 Jan 15:51:55 | Invalid - block 19056922 (0x76eca7...81d489) is known to be a part of an invalid chain. The
last valid is 0x932123bf49f6ffce68aac29820bda6028d3bf7aebbebd5fdc758dac9d1c81c46

Expected behavior
process blocks

[tianshiz]

can confirm, was on 1.25.1, went back to 1.25, same issue

[MarekM25]

@wga22 @tianshiz update your nodes to 1.25.2
https://github.com/NethermindEth/nethermind/releases/tag/1.25.2

[CJValdez1]

Resync database now

[MarekM25]

No resync required :)