Add Prometheus metric for gRPC txs simulations errors

[freak12techno]

Summary

It'd be really awesome if Prometheus would expose a metric that displays the number of errors of encountered gRPC transaction similation errors, to build alerts upon it.

Problem Definition

I am currently getting a lot of errors like this one in my logs:

Feb 10 15:00:54 monitoring-logging hermes[3533022]: 2024-02-10T14:00:54.862596Z ERROR ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=0b01290a ->Destination @4-19097544; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=0b01290a}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=984}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "spendable balance 2950583uosmo is smaller than 3000001uosmo: insufficient funds: insufficient funds [osmosis-labs/osmosis/v22/x/txfees/keeper/feedecorator.go:294] With gas wanted: '300000000' and gas used: '913739' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "13725151"} }

Not sure yet what's the reason, that's another discussion topic out of the scope of this feature request, but it would be nice if there would be a way to 1) see this happening via a Grafana dashboard that takes data from Prometheus set up to scrape Hermes metrics, and 2) build alerts from this metrics to know about such things in advance.

Proposal

Add a Prometheus metric, as a counter, displaying how many errors happened when simulating a transaction. The labels can be a chain, and probably the error type/status/whether the error is recoverable or not.

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[freak12techno]

(I can add it myself it it's possible, always wanted to play with Rust a bit lol)

[romac]

Great idea! We have our hands a bit full with other core development tasks at the moment but we'd happily accept and shepard a PR, as you kindly suggest.