Metric updates when timeout occurs

[AlianBenabdallah]

This PR fixes the following:

• Telemetry: introduce backlog_* metrics, plus minor fixes #2451
• Incorrect handling of the metrics when a packet is relayed by another relayer #2467
• Metric oldest_timestamp gets updated without oldest_sequence as more packets are pending #2469

Pushed to follow-up work:

• "There is an inconsistency between send packets count and acknowledgements count in the displayed metrics."
  • This is Anca's comment from Telemetry: introduce backlog_* metrics, plus minor fixes #2451, need to investigate further
• Discovery phase as described in Telemetry: introduce backlog_* metrics, plus minor fixes #2451

Both captured in #2479

Description

1. Even after a channel is cleared, oldest_* metrics can remain to the same value (i.e., not reset to 0).

3. I think we should clarify what the oldest_timestamp is, it seems this field is a local timestamp to the Hermes process, not an on-chain packet timestamp (when the packet was created), which is not clear from the telemetry help message, specifically:

4. There is an inconsistency between send packets count and acknowledgements count in the displayed metrics.

Previously, oldest_* metrics were not updated when a timeout would occur. The oldest_* metrics take their values from the sequences_histories map which was updated only when an ACK was received.

This PR handles the case of a timeout by calling record_ack_history when a timeout occurs.

The description of the oldest_* metric is also updated to incorporate the fact that the oldest_* is not only updated when an ACK occurs.

---

PR author checklist:

• Added changelog entry, using unclog.
• Added tests: integration (for Hermes) or unit/mock tests (for modules).
• Linked to GitHub issue.
• Updated code comments and documentation (e.g., docs/).

Reviewer checklist:

• Reviewed Files changed in the GitHub PR explorer.
• Manually tested (in case integration/unit/mock tests are absent).

[seanchen1991]

Other than Romain's comments, this looks good to me! 👍

[adizere]

Amazing work Ali!

In additional to the bugs attached to the PR, we did some work refining the metrics and try to make them more useful, specifically:

• renamed oldest_sequence -> backlog_oldest_sequence
• renamed oldest_timestamp -> backlog_oldest_timestamp
• introduced backlog_size to complement the other backlog_* data, as a metric reporting how many packets are pending on a channel

What's left to do:

• "There is an inconsistency between send packets count and acknowledgements count in the displayed metrics."
  • This is Anca's comment from Telemetry: introduce backlog_* metrics, plus minor fixes #2451, need to investigate further
• Discovery phase as described in Telemetry: introduce backlog_* metrics, plus minor fixes #2451

I think both of these should be done in a follow-up PR. The present PR is linked to 3 issues and will be difficult to track what change is fixing what issue.

Discovery phase

Not sure how we can implement the discovery phase, because Hermes does not start packet workers for any channel until there's packets to relay on that channel, so not sure how we can populate the channel-specific metrics. For the other (non-channel-specific) metrics, we can probably initialize them with zero.

Inconsistency

For the "inconsistency between send packets count and acknowledgements count" we need some more digging. We actually have a lot of complexity around these metrics, there's four of them:

1. send_packet_count
2. acknowledgement_count
3. cleared_send_packet_count
4. cleared_acknowledgment_count

and its not clear how operators can use them (aside from inconsistencies). At the very least, we should document what is the purpose and insights they offer. Do they indicate activity? Lack of activity? Problems relaying? I'm not sure I know what they do.

[AlianBenabdallah]

I tested again and the bug is correctly fixed.