Could not query posdao epoch at parent

[SurfingNerd]

Example on hbbft9:

2022-03-24 17:27:09  Worker Hbbft0 TRACE engine  replay_cached_messages: Could not query posdao epoch at parent block#142878, re-trying later. Probably due to the block not being imported yet. CallFailed("Transaction execution error (Couldn\'t find the transaction block\'s state in the chain).")

The Node was active, up to date, but could not write It's Part.

It is clearly lacking behind with the import of blocks:

022-03-24 17:24:11  Worker Client0 INFO import  Informant: trying to get lock.
2022-03-24 17:24:11  Worker Client0 INFO import  Informant: got lock.
2022-03-24 17:24:11  Worker Client0 INFO import  Imported #142864 0x2468…b6f4 (0 txs, 0.00 Mgas, 579 ms, 0.56 KiB)
2022-03-24 17:24:22  Worker Client1 INFO import   35/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:24:28  Worker Client1 INFO import  Informant: trying to get lock.
2022-03-24 17:24:28  Worker Client1 INFO import  Informant: got lock.
2022-03-24 17:24:28  Worker Client1 INFO import  Imported #142866 0x9f5f…9e0b (1 txs, 6.68 Mgas, 90 ms, 10.80 KiB)
2022-03-24 17:24:57  Worker Client0 INFO import   35/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:25:10  Worker Client0 INFO import  Informant: trying to get lock.
2022-03-24 17:25:10  Worker Client0 INFO import  Informant: got lock.
2022-03-24 17:25:10  Worker Client0 INFO import  Imported #142870 0x48d0…a6fc (0 txs, 0.00 Mgas, 598 ms, 0.56 KiB)
2022-03-24 17:25:32  Worker Client3 INFO import   35/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:25:54  Worker Hbbft1 TRACE engine  replay_cached_messages: Could not query posdao epoch at parent block#142873, re-trying later. Probably due to the block not being imported yet. CallFailed("Transaction execution error (Couldn\'t find the transaction block\'s state in the chain).")
2022-03-24 17:25:54  Worker Client0 INFO import  Informant: trying to get lock.
2022-03-24 17:25:54  Worker Client0 INFO import  Informant: got lock.
2022-03-24 17:25:54  Worker Client0 INFO import  Imported #142873 0x9ca2…a461 (1 txs, 0.21 Mgas, 531 ms, 0.73 KiB)
2022-03-24 17:26:02  Worker Client1 INFO import   36/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:26:32  Worker Client0 INFO import   36/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:26:38  Worker Hbbft0 TRACE engine  replay_cached_messages: Could not query posdao epoch at parent block#142876, re-trying later. Probably due to the block not being imported yet. CallFailed("Transaction execution error (Couldn\'t find the transaction block\'s state in the chain).")
2022-03-24 17:26:39  Worker Client3 INFO import  Informant: trying to get lock.
2022-03-24 17:26:39  Worker Client3 INFO import  Informant: got lock.
2022-03-24 17:26:39  Worker Client3 INFO import  Imported #142876 0xf113…b330 (0 txs, 0.00 Mgas, 572 ms, 0.56 KiB)
2022-03-24 17:27:02  Worker Client0 INFO import   36/50 peers   5 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,   24 µs
2022-03-24 17:27:09  Worker Hbbft0 TRACE engine  replay_cached_messages: Could not query posdao epoch at parent block#142878, re-trying later. Probably due to the block not being imported yet. CallFailed("Transaction execution error (Couldn\'t find the transaction block\'s state in the chain).")
2022-03-24 17:27:09  Worker Client2 INFO import  Informant: trying to get lock.
2022-03-24 17:27:09  Worker Client2 INFO import  Informant: got lock.

Hint:
this filter igves good results analysing that problem:

tail parity.log -n 1000000 | grep import

[SurfingNerd]

The node managed to send it's Parts, but those could not get included.
real issue might be Key Gen Transactions not broadcasted ?

log section:

2022-03-24 17:24:27  Verifier #3 DEBUG txqueue  Service tx 0xbfa78838a308f04b56659201751880eae4f54db8874948ac65e7ac2ef3816dda below minimal gas price accepted
2022-03-24 17:24:27  Verifier #3 DEBUG txqueue  [0xbfa78838a308f04b56659201751880eae4f54db8874948ac65e7ac2ef3816dda] Added to the pool.
2022-03-24 17:24:27  Verifier #3 DEBUG txqueue  [0xbfa78838a308f04b56659201751880eae4f54db8874948ac65e7ac2ef3816dda] Sender: 0xc7dd…6d59, nonce: 6727, gasPrice: 0, gas: 8277600, value: 0, dataLen: 10372))
2022-03-24 17:25:38  Verifier #3 DEBUG txqueue  [0xbfa78838a308f04b56659201751880eae4f54db8874948ac65e7ac2ef3816dda] Dropped. Replaced by [0x0ffd81d37cb1651f79ffbf6f01beba37dafd04b7630817f5c42799f49a1e68a1]

[SurfingNerd]

It has been analysed to be most likely a timing/threading issue, where the client is about to finish a block, but the block is not fully commited on the blockchain, so it can't be read.
In a bad luck situation, this happens every time when a node tries to write it's part & acks during key generation phase - so it becomes unavailable after severel errors