feat: Retry tm connection when orchestrator hangs

[rach-id]

Overview

This PR allows the orchestrator to retry connection when it disconnects from tendermint node.
Closes #181

The relayer part is covered in #242

Checklist

• New and updated code has appropriate documentation
• New and updated code has new and/or updated testing
• Required CI checks are passing
• Visual proof for any user facing features like CLI or documentation updates
• Linked issues closed with keywords

[codecov-commenter]

Codecov Report

Merging #243 (8b1ead1) into main (e2d0f17) will decrease coverage by 0.17%.
The diff coverage is 22.60%.

@@            Coverage Diff             @@
##             main     #243      +/-   ##
==========================================
- Coverage   33.02%   32.86%   -0.17%     
==========================================
  Files          22       22              
  Lines        2065     2115      +50     
==========================================
+ Hits          682      695      +13     
- Misses       1308     1344      +36     
- Partials       75       76       +1     

Impacted Files	Coverage Δ
e2e/qgb_network.go	0.00% <0.00%> (ø)
orchestrator/orchestrator.go	11.86% <0.00%> (-1.25%)	⬇️
rpc/tm_querier.go	28.57% <13.33%> (-28.58%)	⬇️
rpc/app_querier.go	50.81% <88.46%> (+1.71%)	⬆️
p2p/dht.go	70.11% <100.00%> (+1.44%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.