Metric oldest_timestamp gets updated without oldest_sequence as more packets are pending

[AlianBenabdallah]

Summary of Bug

When a packet is pending, oldest_timestamp and oldest_sequence should both be fixed. However, when a new packet arrives, oldest_timestamp takes temporarily the value of the youngest pending packet.

Version

hermes 1.0.0-rc.0+6c23af53

Steps to Reproduce

You should run this on branch ali/fix_oldest_sequence (or master after merge) because the process used to generate pending packets has a high probability of failure and, if it fails, will generate a timeout. This branch handles correctly the oldest_* metrics when a timeout occurs. If you are not using this branch, a timeout will freeze the metrics.

• Set up 2 chains and create channels between them
• Run hermes start
• Create a pending packet : hermes tx ft-transfer --dst-chain ibc-0 --src-chain ibc-1 --src-port transfer --src-channel channel-0 --amount 1000 --timeout-seconds 10 should work most of the times.
• Wait 30s.
• Ensure that a packet is indeed pending with hermes query packet pending --chain ibc-1 --port transfer --channel channel-0

You should see an output similar to this :

Success: Summary {
    src: PendingPackets {
        unreceived_packets: [
            Sequence(
                21,
            ),
        ],
        unreceived_acks: [],
    },
    dst: PendingPackets {
        unreceived_packets: [],
        unreceived_acks: [],
    },
}

• Ensure that the oldest_* are frozen at a non zero value. If it's not the case then no packet is pending or the code is not running on the correct branch.
  You should see non zero metrics for these two fields :

# HELP oldest_sequence Sequence number of the oldest pending SendPacket event.
# TYPE oldest_sequence gauge
oldest_sequence{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 21
# HELP oldest_timestamp Local timestamp of the oldest pending SendPacket event
# TYPE oldest_timestamp gauge
oldest_timestamp{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} **1658850385**

• Now send more packets with hermes tx ft-transfer --dst-chain ibc-0 --src-chain ibc-1 --src-port transfer --src-channel channel-0 --amount 1000 --timeout-seconds 1000.
• You should see these packets pending with hermes query packet pending --chain ibc-1 --port transfer --channel channel-0.
• You should see the metrics being updated. In my case :

# HELP oldest_sequence Sequence number of the oldest pending SendPacket event.
# TYPE oldest_sequence gauge
oldest_sequence{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 21
# HELP oldest_timestamp Local timestamp of the oldest pending SendPacket event
# TYPE oldest_timestamp gauge
oldest_timestamp{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} **1658850476**

• Notice that oldest_timestamp was updated.
• If you wait 30 more seconds and refresh, the old value should be back.

Acceptance Criteria

oldest_timestamp is correctly updated.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned