Check for ibc-go version after a chain upgrade

[miladz68]

Summary

The Hermes relayer should be able to detect version changes in upgrades and continue to relay without requiring a restart.

Problem Definition

The Hermes relayer needs to be restarted after a chain upgrade which includes updating ibc-go from v4 to v7, otherwise it cannot correctly relay messages. It would be nice if hermes could detect the upgrade and check ibc version changes, so the restart would not be needed.
This is the error log produced right after the upgrade.

2023-08-31T08:49:28.832869Z ERROR ThreadId(29) event_source.rpc{chain.id=coreum-devnet-1}: failed to collect events: RPC error: serde parse error: invalid utf-8 sequence of 1 bytes from index 0 at line 1 column 181 height=69
2023-08-31T08:49:29.794413Z  WARN ThreadId(25) spawn:chain{chain=coreum-devnet-1}:wallet{chain=coreum-devnet-1}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
2023-08-31T08:49:29.833589Z ERROR ThreadId(29) event_source.rpc{chain.id=coreum-devnet-1}: failed to collect events: RPC error: serde parse error: invalid utf-8 sequence of 1 bytes from index 0 at line 1 column 181 height=69
2023-08-31T08:49:30.834989Z ERROR ThreadId(29) event_source.rpc{chain.id=coreum-devnet-1}: failed to collect events: RPC error: serde parse error: invalid utf-8 sequence of 1 bytes from index 0 at line 1 column 181 height=69
2023-08-31T08:49:31.835460Z ERROR ThreadId(29) event_source.rpc{chain.id=coreum-devnet-1}: failed to collect events: RPC error: serde parse error: invalid utf-8 sequence of 1 bytes from index 0 at line 1 column 181 height=69
2023-08-31T08:49:32.835802Z ERROR ThreadId(29) event_source.rpc{chain.id=coreum-devnet-1}: failed to collect events: RPC error: serde parse error: invalid utf-8 sequence of 1 bytes from index 0 at line 1 column 181 height=69

Proposal

As stated, Hermes should check and update ibc-go version after an upgrade, without requiring a restart.

Acceptance Criteria

Do an upgrade which contains update of ibc-go from v4 to v7, and the relayer should continue to relay correctly.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[chatton]

We recently ran into some issues with our upgrade tests upgrading from ibc-go v6 to v7 and are seeing verify similar behaviour

We noticed that if we restart hermes after the chain upgrade, everything works as expected.

Note: this wasn't an issue with upgrading from v7 to v7.1, and from v7 to v8, everything worked as expected.

By adding a restart here in our upgrade test it passed.

Full hermes logs here

�������M2023-09-14T08:49:59.192277Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
�������M2023-09-14T08:49:59.252876Z  INFO ThreadId(01) telemetry: telemetry disabled
�������J2023-09-14T08:49:59.255906Z  INFO ThreadId(01) rest: REST server disabled
��������2023-09-14T08:49:59.381537Z  WARN ThreadId(13) health_check{chain=chain-a}: Chain 'chain-a' has no minimum gas price value configured for denomination 'atoma'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
��������2023-09-14T08:49:59.455717Z  INFO ThreadId(13) health_check{chain=chain-a}: Updated `max_block_time` using /genesis endpoint. Old value: `30s`, new value: `30s`
�������]2023-09-14T08:49:59.457564Z  INFO ThreadId(01) health_check{chain=chain-a}: chain is healthy
��������2023-09-14T08:49:59.482553Z  WARN ThreadId(14) health_check{chain=chain-b}: Chain 'chain-b' has no minimum gas price value configured for denomination 'atomb'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
��������2023-09-14T08:49:59.521402Z  INFO ThreadId(14) health_check{chain=chain-b}: Updated `max_block_time` using /genesis endpoint. Old value: `30s`, new value: `30s`
�������]2023-09-14T08:49:59.521578Z  INFO ThreadId(01) health_check{chain=chain-b}: chain is healthy
�������\2023-09-14T08:49:59.522920Z  INFO ThreadId(01) scan.chain{chain=chain-a}: scanning chain...
��������2023-09-14T08:49:59.523872Z  INFO ThreadId(01) scan.chain{chain=chain-a}: scanning chain for all clients, connections and channels
�������b2023-09-14T08:49:59.524010Z  INFO ThreadId(01) scan.chain{chain=chain-a}: scanning all clients...
�������;2023-09-14T08:49:59.538636Z  WARN ThreadId(13) scan.chain{chain=chain-a}: failed to parse client state IdentifiedClientState { client_id: 06-solomachine-2, client_state: Any { type_url: /ibc.lightclients.solomachine.v2.ClientState } }. Error: unknown client state type: /ibc.lightclients.solomachine.v2.ClientState
��������2023-09-14T08:49:59.542427Z  INFO ThreadId(01) scan.chain{chain=chain-a}:scan.client{client=07-tendermint-0}: scanning client...
��������2023-09-14T08:49:59.568459Z  INFO ThreadId(01) scan.chain{chain=chain-a}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
��������2023-09-14T08:49:59.612846Z  INFO ThreadId(01) scan.chain{chain=chain-a}:scan.client{client=07-tendermint-1}: scanning client...
�������\2023-09-14T08:49:59.620691Z  INFO ThreadId(01) scan.chain{chain=chain-b}: scanning chain...
��������2023-09-14T08:49:59.621359Z  INFO ThreadId(01) scan.chain{chain=chain-b}: scanning chain for all clients, connections and channels
�������b2023-09-14T08:49:59.621455Z  INFO ThreadId(01) scan.chain{chain=chain-b}: scanning all clients...
��������2023-09-14T08:49:59.633443Z  INFO ThreadId(01) scan.chain{chain=chain-b}:scan.client{client=07-tendermint-0}: scanning client...
��������2023-09-14T08:49:59.641076Z  INFO ThreadId(01) scan.chain{chain=chain-b}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
��������2023-09-14T08:49:59.658006Z  INFO ThreadId(01) scan.chain{chain=chain-b}:scan.client{client=07-tendermint-1}: scanning client...
�������?2023-09-14T08:49:59.665344Z  INFO ThreadId(01) scanned chains:
�������@2023-09-14T08:49:59.665455Z  INFO ThreadId(01) # Chain: chain-a
��������  - Client: 07-tendermint-0
��������    * Connection: connection-0
��������      | State: OPEN
�������!      | Counterparty state: OPEN
��������      + Channel: channel-0
��������        | Port: transfer
��������        | State: OPEN
�������"        | Counterparty: channel-0
��������  - Client: 07-tendermint-1
��������# Chain: chain-b
��������  - Client: 07-tendermint-0
��������    * Connection: connection-0
��������      | State: OPEN
�������!      | Counterparty state: OPEN
��������      + Channel: channel-0
��������        | Port: transfer
��������        | State: OPEN
�������"        | Counterparty: channel-0
��������  - Client: 07-tendermint-1
��������
��������2023-09-14T08:49:59.667247Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=chain-a connection=connection-0 counterparty_chain=chain-b
��������2023-09-14T08:49:59.667441Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=chain-a connection=connection-0
��������2023-09-14T08:49:59.667888Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=chain-a connection=connection-0
��������2023-09-14T08:49:59.668276Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=chain-a counterparty_chain=chain-b channel=channel-0
��������2023-09-14T08:49:59.713692Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::chain-b->chain-a:07-tendermint-0
��������2023-09-14T08:49:59.751409Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned packet worker: packet::channel-0/transfer:chain-a->chain-b
��������2023-09-14T08:49:59.751963Z  INFO ThreadId(01) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=chain-a channel=channel-0
�������s2023-09-14T08:49:59.755293Z  INFO ThreadId(01) spawn:chain{chain=chain-a}: spawning Wallet worker: wallet::chain-a
��������2023-09-14T08:49:59.755859Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=chain-b connection=connection-0 counterparty_chain=chain-a
��������2023-09-14T08:49:59.755924Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=chain-b connection=connection-0
��������2023-09-14T08:49:59.755965Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=chain-b connection=connection-0
��������2023-09-14T08:49:59.756017Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=chain-b counterparty_chain=chain-a channel=channel-0
��������2023-09-14T08:49:59.795887Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::chain-a->chain-b:07-tendermint-0
��������2023-09-14T08:49:59.829320Z  INFO ThreadId(01) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=chain-b channel=channel-0
�������s2023-09-14T08:49:59.830045Z  INFO ThreadId(01) spawn:chain{chain=chain-b}: spawning Wallet worker: wallet::chain-b
�������B2023-09-14T08:49:59.868024Z  INFO ThreadId(01) Hermes has started
��������2023-09-14T08:50:00.274078Z  INFO ThreadId(14) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-b->chain-a:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-b->chain-a:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-b
��������2023-09-14T08:50:00.340402Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-a
��������2023-09-14T08:50:00.641838Z  INFO ThreadId(14) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-b->chain-a:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-b->chain-a:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-b
��������2023-09-14T08:50:00.712892Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-a
��������2023-09-14T08:50:01.001192Z  INFO ThreadId(14) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-b->chain-a:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-b->chain-a:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-b
�������*2023-09-14T08:50:01.021207Z  INFO ThreadId(17) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.packet.cmd{src_chain=chain-a src_port=transfer src_channel=channel-0 dst_chain=chain-b}:schedule_packet_clearing{height=Some(Height { revision: 0, height: 55 })}:relay_pending_packets{height=Some(Height { revision: 0, height: 54 })}:schedule_recv_packet_and_timeout_msgs{query_height=0-54}: pulled packet data for 1 events out of 1 sequences: 1..=1; events.total=1 events.left=0
��������2023-09-14T08:50:01.073645Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-a
�������a2023-09-14T08:50:01.226949Z  INFO ThreadId(17) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.packet.cmd{src_chain=chain-a src_port=transfer src_channel=channel-0 dst_chain=chain-b}:relay{odata=cleared/482d00ea ->Destination @0-54; len=1}: assembled batch of 2 message(s)
��������2023-09-14T08:50:01.306148Z  INFO ThreadId(16) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-b->chain-a:07-tendermint-0}: client is valid
��������2023-09-14T08:50:01.309828Z  INFO ThreadId(17) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.packet.cmd{src_chain=chain-a src_port=transfer src_channel=channel-0 dst_chain=chain-b}:relay{odata=cleared/482d00ea ->Destination @0-54; len=1}: response(s): 1; Ok:6085DF1B9D59FA1146105E59BC9814CA1CA5632F45763282DA1531F0E1E05C59 target_chain=chain-b
�������K2023-09-14T08:50:01.310187Z  INFO ThreadId(17) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.packet.cmd{src_chain=chain-a src_port=transfer src_channel=channel-0 dst_chain=chain-b}:relay{odata=cleared/482d00ea ->Destination @0-54; len=1}: submitted
��������2023-09-14T08:50:01.437997Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-a
��������2023-09-14T08:50:01.799645Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=None}: No evidence of misbehavior detected for chain chain-a
��������2023-09-14T08:50:02.104142Z  INFO ThreadId(21) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=chain-a->chain-b:07-tendermint-0}: client is valid
�������
2023-09-14T08:50:03.138037Z  INFO ThreadId(154) worker.batch{chain=chain-b}:supervisor.handle_batch{chain=chain-b}:supervisor.process_batch{chain=chain-b}:worker.packet.cmd{src_chain=chain-b src_port=transfer src_channel=channel-0 dst_chain=chain-a}:schedule_packet_clearing{height=Some(Height { revision: 0, height: 56 })}:relay_pending_packets{height=Some(Height { revision: 0, height: 55 })}:build_packet_ack_msgs{query_height=0-55}: pulled packet data for 0 events out of 1 sequences: 1..=1; events.total=1 events.left=0
��������2023-09-14T08:50:03.560658Z  INFO ThreadId(13) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 55 } }, header: Some( Header {...}) }) client=chain-a->chain-b:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-a->chain-b:07-tendermint-0 update_height=Some(Height { revision: 0, height: 55 })}: No evidence of misbehavior detected for chain chain-a
�������>2023-09-14T08:50:03.666379Z  INFO ThreadId(21) spawn:chain{chain=chain-b}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-a dst_chain=chain-b}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 55 } }, header: Some( Header {...}) }) client=chain-a->chain-b:07-tendermint-0}: client is valid
�������L2023-09-14T08:50:04.725175Z  INFO ThreadId(154) worker.batch{chain=chain-b}:supervisor.handle_batch{chain=chain-b}:supervisor.process_batch{chain=chain-b}:worker.packet.cmd{src_chain=chain-b src_port=transfer src_channel=channel-0 dst_chain=chain-a}:relay{odata=d56f592e ->Destination @0-56; len=1}: assembled batch of 2 message(s)
��������2023-09-14T08:50:04.773104Z  INFO ThreadId(154) worker.batch{chain=chain-b}:supervisor.handle_batch{chain=chain-b}:supervisor.process_batch{chain=chain-b}:worker.packet.cmd{src_chain=chain-b src_port=transfer src_channel=channel-0 dst_chain=chain-a}:relay{odata=d56f592e ->Destination @0-56; len=1}: response(s): 1; Ok:7246837CAF74EE822B051269F8B3061893B17D7329A7F9A1491E788985505BDF target_chain=chain-a
�������62023-09-14T08:50:04.773316Z  INFO ThreadId(154) worker.batch{chain=chain-b}:supervisor.handle_batch{chain=chain-b}:supervisor.process_batch{chain=chain-b}:worker.packet.cmd{src_chain=chain-b src_port=transfer src_channel=channel-0 dst_chain=chain-a}:relay{odata=d56f592e ->Destination @0-56; len=1}: submitted
��������2023-09-14T08:50:08.168316Z  INFO ThreadId(14) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 57 } }, header: Some( Header {...}) }) client=chain-b->chain-a:07-tendermint-0}:foreign_client.detect_misbehaviour{client=chain-b->chain-a:07-tendermint-0 update_height=Some(Height { revision: 0, height: 57 })}: No evidence of misbehavior detected for chain chain-b
�������>2023-09-14T08:50:08.268779Z  INFO ThreadId(16) spawn:chain{chain=chain-a}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=chain-b dst_chain=chain-a}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 57 } }, header: Some( Header {...}) }) client=chain-b->chain-a:07-tendermint-0}: client is valid
��������2023-09-14T08:53:10.105463Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:53:10.106763Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:10.107639Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 1s...
��������2023-09-14T08:53:11.110213Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: tcp connect error: Connection refused (os error 111)
�������g2023-09-14T08:53:11.110322Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 1.5s...
��������2023-09-14T08:53:12.619972Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:12.620368Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 2s...
��������2023-09-14T08:53:14.638336Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������g2023-09-14T08:53:14.638553Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 2.5s...
��������2023-09-14T08:53:17.150966Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:17.151059Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 3s...
��������2023-09-14T08:53:20.164249Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������g2023-09-14T08:53:20.164336Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 3.5s...
��������2023-09-14T08:53:23.684733Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:23.684819Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 4s...
��������2023-09-14T08:53:27.697163Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������g2023-09-14T08:53:27.697263Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 4.5s...
�������:2023-09-14T08:53:28.169353Z  WARN ThreadId(154) worker.batch{chain=chain-b}:supervisor.handle_batch{chain=chain-b}:supervisor.process_batch{chain=chain-b}:worker.packet.cmd{src_chain=chain-b src_port=transfer src_channel=channel-0 dst_chain=chain-a}: packet worker has been idle for more than 100 blocks, aborting
��������2023-09-14T08:53:32.210085Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:32.210313Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:53:37.224041Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:37.224185Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:53:42.234160Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:42.234280Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:53:47.088546Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:53:47.251140Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:47.251220Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:53:52.098451Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:53:52.266751Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:52.266875Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:53:57.113124Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:53:57.276989Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:53:57.277074Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:02.125418Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:02.287035Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:02.287121Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:07.133940Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:07.300543Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:07.300681Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:12.149101Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:12.309727Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:12.309878Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:17.163327Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:17.325164Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:17.325249Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:22.178123Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:22.336580Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:22.336679Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:27.193957Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:27.349594Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:27.349695Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:32.204255Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:32.359524Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:32.359611Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:37.215299Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:37.372712Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:37.372846Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:42.228158Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:42.387994Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:42.388065Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:47.247118Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:47.399023Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:47.399149Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:52.258694Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:52.407795Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:52.407876Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:54:57.288007Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:54:57.417615Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:54:57.417796Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:55:02.301009Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:55:02.426903Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:55:02.427012Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:55:07.312995Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:55:07.435653Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:55:07.435922Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...
��������2023-09-14T08:55:12.323948Z  WARN ThreadId(19) spawn:chain{chain=chain-a}:wallet{chain=chain-a}: task encountered ignorable error: failed to query balance for the account: error in underlying transport when making gRPC call: transport error
��������2023-09-14T08:55:12.448589Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: event source encountered an error: RPC error: HTTP error: error trying to connect: dns error: failed to lookup address information: Name or service not known
�������e2023-09-14T08:55:12.448693Z ERROR ThreadId(23) event_source.rpc{chain.id=chain-a}: retrying in 5s...

[kstoykov]

I also confirms that upgrading from ibc-go v2.2 to v7.2 also requires an restart