Invalid Transaction Nonce on Validators

[shufps]

After restarting a single validators, all other crashed with following error:

2023-07-07T05:55:58.263156398Z  DEBUG   Chains.0xfd477686.SM.gpa        Input consensus decided state <c8eae0dfe5c7e802686aa952b60e202526097984;0xc3b9c41f53110701e008fd708a822fca6f56906b> handled
2023-07-07T05:55:58.274741113Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664  Input *cons.inputMempoolRequests: {cons.inputMempoolRequests:  {request, id=0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000} {request, id=0x912bee848b504a376eb0cb3351e4
ceb362f3760ce44d1f44f084cac5adcc8e100000} {request, id=0x9dfa180b5423324254cd0cd436fae773802696ab3cccf0c7e5d2ebe4bb4ff6300000} {request, id=0x796c651939fe98e9fe7cae2be3a95ebe1d9c8a8cf4b3be411673f0b6fe8607de0000}}
2023-07-07T05:55:58.274842602Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664  Input *cons.inputStateMgrDecidedVirtualState: {cons.inputStateMgrDecidedVirtualState: blockIndex=19749, trieRoot=c8eae0dfe5c7e802686aa952b60e202526097984}
2023-07-07T05:55:58.275611492Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       callFromRequest: 0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000
2023-07-07T05:55:58.28773727Z   DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       recovered panic from contract call: invalid transaction nonce: got 5, want 4
2023-07-07T05:55:58.287904392Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       goroutine 13295528 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).callTheContract.func1.1()
        /scratch/packages/vm/vmcontext/runreq.go:220 +0xe7
panic({0x2d0af60, 0xc0031c4ac0})
        /usr/local/go/src/runtime/panic.go:884 +0x213
panic: error running the VM:  panic when running request #0 ID:0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000, requestbytes:0x02fd477686e642855f2ead5931a8ee152fa9e8c52882bb291a3c02a7738ab74647ef08f9046c0585e8d4a510008299d894107400000000000000000000000000000000000080b90404b7a53f5300000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000120000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000003400000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002100bc4fa46f8d9e1d702fc293e69690eb79fdd67ae4309a56da838dd67d1882820f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003b9a25f000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000820883a06be465588f8442d45a8311de7ac3c10c67a81f39970ee1b755082b67029e5a7ea049612ed15fdddf8dd5666a82cdb3f2cc41d59a3872ce8e1874e2c2819905e041 err:nil receipt
github.com/iotaledger/wasp/packages/vm/core/evm/evmimpl.applyTransaction({0x35a3080, 0xc001963a88})
goroutine 13295528 [running]:
        /scratch/packages/vm/core/evm/evmimpl/impl.go:148 +0x539
github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).run(0xc001e08100, 0xc002abc9a0, 0xc002ab4400?)
github.com/iotaledger/wasp/packages/vm/core/evm/evmimpl.restricted.func1({0x35a3080?, 0xc001963a88})
        /scratch/packages/chain/cons/cons_gr/vm_async.go:46 +0x1f4
        /scratch/packages/vm/core/evm/evmimpl/restricted.go:21 +0x5c
github.com/iotaledger/wasp/packages/isc/coreutil.(*EntryPointHandler).Call(0xc0002588a0, {0x301dca0?, 0xc001963a88})
        /scratch/packages/isc/coreutil/contract.go:108 +0x42
created by github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).ConsensusRunTask
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).callProgram(0xc003bb2c60, 0x0?, 0xe30cd9fa, 0x0?, 0xc003404168?, {0x0, 0x0, 0x0})
        /scratch/packages/vm/vmcontext/call.go:41 +0x1ff
        /scratch/packages/chain/cons/cons_gr/vm_async.go:34 +0xaa
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).callFromRequest(0xc003bb2c60)
        /scratch/packages/vm/vmcontext/runreq.go:286 +0x24e
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).callTheContract.func1(0xc003bb2c60, 0x3592290?, 0xc0032176e0)
        /scratch/packages/vm/vmcontext/runreq.go:226 +0x76
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).callTheContract(0xc003bb2c60)
        /scratch/packages/vm/vmcontext/runreq.go:229 +0x6b
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).RunTheRequest.func3()
        /scratch/packages/vm/vmcontext/runreq.go:79 +0x59
github.com/iotaledger/wasp/packages/util/panicutil.CatchPanic.func1(0x2f95680?, 0x0?)
        /scratch/packages/util/panicutil/panicutil.go:79 +0x53
github.com/iotaledger/wasp/packages/util/panicutil.CatchPanic(0xc000278a50?)
        /scratch/packages/util/panicutil/panicutil.go:80 +0x27
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).catchRequestPanic(0xc003bb2c60, 0xc000391088?)
        /scratch/packages/vm/vmcontext/runreq.go:158 +0x36
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).RunTheRequest(0xc003bb2c60, {0x3592290?, 0xc003d66b40}, 0x0)
        /scratch/packages/vm/vmcontext/runreq.go:72 +0x513
github.com/iotaledger/wasp/packages/vm/runvm.runRequests(0xc002abc9a0?, {0xc002aaf780?, 0x4, 0x4?}, 0x4?, 0x9a00ddac80?)
        /scratch/packages/vm/runvm/runtask.go:44 +0xf9
github.com/iotaledger/wasp/packages/vm/runvm.runTask(0xc002abc9a0)
        /scratch/packages/vm/runvm/runtask.go:75 +0xb2
github.com/iotaledger/wasp/packages/vm/runvm.VMRunner.Run.func1()
        /scratch/packages/vm/runvm/runtask.go:17 +0x26
github.com/iotaledger/wasp/packages/util/panicutil.CatchAllButDBError.func1(0x3568cc0?, 0xc0018a5e20?, {0x0?, 0x2d38b00?}, 0x35930c0?)
        /scratch/packages/util/panicutil/panicutil.go:59 +0x77
github.com/iotaledger/wasp/packages/util/panicutil.CatchAllButDBError(0x165d9e0?, 0x7fd79c0b6300?, {0x0?, 0x1708cd1?, 0x7fd79c0b6300?})
        /scratch/packages/util/panicutil/panicutil.go:60 +0x49
github.com/iotaledger/wasp/packages/vm/runvm.VMRunner.Run({}, 0xc002abc9a0)
        /scratch/packages/vm/runvm/runtask.go:16 +0x8b
github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).run(0xc001e08100, 0xc002abc9a0, 0xc002ab4400?)
        /scratch/packages/chain/cons/cons_gr/vm_async.go:41 +0x83
created by github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).ConsensusRunTask
        /scratch/packages/chain/cons/cons_gr/vm_async.go:34 +0xaa
2023-07-07T05:55:58.289655792Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       writeReceiptToBlockLog - reqID:0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000 err: invalid transaction nonce: got 5, want 4
2023-07-07T05:55:58.289933764Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       panic when running request #0 ID:0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000, requestbytes:0x02fd477686e642855f2ead5931a8ee152fa9e8c52882bb291a3c02a7738ab74647ef08f9046c0585e8d4a510008299d894107400000000000000000000000000000000000080b90404b7a53f5300000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000120000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000003400000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002100bc4fa46f8d9e1d702fc293e69690eb79fdd67ae4309a56da838dd67d1882820f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003b9a25f000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000820883a06be465588f8442d45a8311de7ac3c10c67a81f39970ee1b755082b67029e5a7ea049612ed15fdddf8dd5666a82cdb3f2cc41d59a3872ce8e1874e2c2819905e041 err:nil receipt
2023-07-07T05:55:58.29000172Z   DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664.VM       goroutine 13295528 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/iotaledger/wasp/packages/util/panicutil.CatchAllButDBError.func1.1()
        /scratch/packages/util/panicutil/panicutil.go:57 +0x270
panic({0x2d620e0, 0xc0031e4c00})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).catchRequestPanic(0xc003bb2c60, 0xc000391088?)
        /scratch/packages/vm/vmcontext/runreq.go:170 +0x353
github.com/iotaledger/wasp/packages/vm/vmcontext.(*VMContext).RunTheRequest(0xc003bb2c60, {0x3592290?, 0xc003d66b40}, 0x0)
        /scratch/packages/vm/vmcontext/runreq.go:72 +0x513
github.com/iotaledger/wasp/packages/vm/runvm.runRequests(0xc002abc9a0?, {0xc002aaf780?, 0x4, 0x4?}, 0x4?, 0x9a00ddac80?)
        /scratch/packages/vm/runvm/runtask.go:44 +0xf9
github.com/iotaledger/wasp/packages/vm/runvm.runTask(0xc002abc9a0)
        /scratch/packages/vm/runvm/runtask.go:75 +0xb2
github.com/iotaledger/wasp/packages/vm/runvm.VMRunner.Run.func1()
        /scratch/packages/vm/runvm/runtask.go:17 +0x26
github.com/iotaledger/wasp/packages/util/panicutil.CatchAllButDBError.func1(0x3568cc0?, 0xc0018a5e20?, {0x0?, 0x2d38b00?}, 0x35930c0?)
        /scratch/packages/util/panicutil/panicutil.go:59 +0x77
github.com/iotaledger/wasp/packages/util/panicutil.CatchAllButDBError(0x165d9e0?, 0x7fd79c0b6300?, {0x0?, 0x1708cd1?, 0x7fd79c0b6300?})
        /scratch/packages/util/panicutil/panicutil.go:60 +0x49
github.com/iotaledger/wasp/packages/vm/runvm.VMRunner.Run({}, 0xc002abc9a0)
        /scratch/packages/vm/runvm/runtask.go:16 +0x8b
github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).run(0xc001e08100, 0xc002abc9a0, 0xc002ab4400?)
        /scratch/packages/chain/cons/cons_gr/vm_async.go:41 +0x83
created by github.com/iotaledger/wasp/packages/chain/cons/cons_gr.(*vmAsync).ConsensusRunTask
        /scratch/packages/chain/cons/cons_gr/vm_async.go:34 +0xaa
2023-07-07T05:55:58.290010617Z  WARN    Chains.0xfd477686.C-0x981def2d.LI-5664.VM       GENERAL VM EXCEPTION: the task has been abandoned due to: panic when running request #0 ID:0x1f5e255090f4b17b3bcc2472dc1a81ca083f86c5b4e2dba3d05bb575750886420000, requestbytes:0x02fd477686e642855f2ead5931a8ee152fa9e8c52882bb291a3c02a7738ab74647ef08f9046c0585e8d4a510008299d894107400000000000000000000000000000000000080b90404b7a53f5300000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000120000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000003400000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002100bc4fa46f8d9e1d702fc293e69690eb79fdd67ae4309a56da838dd67d1882820f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003b9a25f000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000820883a06be465588f8442d45a8311de7ac3c10c67a81f39970ee1b755082b67029e5a7ea049612ed15fdddf8dd5666a82cdb3f2cc41d59a3872ce8e1874e2c2819905e041 err:nil receipt
2023-07-07T05:55:58.290034332Z  DEBUG   Chains.0xfd477686.C-0x981def2d.LI-5664  VM processed 4 requests in 15.071264ms