Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add telemetry and Prometheus endpoint #985

Merged
merged 32 commits into from
Jun 1, 2021
Merged

Add telemetry and Prometheus endpoint #985

merged 32 commits into from
Jun 1, 2021

Conversation

brapse
Copy link
Contributor

@brapse brapse commented May 25, 2021

Closes: #868

Description

Adds telemetry to Hermes and exposes an Prometheus endpoint.

Testing

  • Configure telemetry exporter in config.toml by adding a [telemetry] section:
[telemetry]
enabled = true
port = 3001
  1. Start dev environment, for example with 3 chains in a config.toml:
./scripts/dev-env ~/.hermes/config.toml ibc-0 ibc-1
  1. Run Hermes:
$ cargo run -- start

In a separate terminal, watch the telemetry (please ensure the port matches the one specified in the telemetry.port config parameter

$ watch -n 1 curl -s localhost:3001/metrics

Expect the output to look like

# HELP hermes_chains_num Number of chains the relay is connecting to
# TYPE hermes_chains_num counter
hermes_chains_num{hermes="all"} 2
# HELP hermes_channels_num Number of channels the relay is connecting to
# TYPE hermes_channels_num counter
hermes_channels_num{hermes="all"} 2
# HELP hermes_tx_msg_ibc_acknowledge_packet Total number of IBC packets acknowledged
# TYPE hermes_tx_msg_ibc_acknowledge_packet counter
hermes_tx_msg_ibc_acknowledge_packet{hermes="all"} 2

Run the e2e test trigger other metrics....


For contributor use:

  • Updated the Unreleased section of CHANGELOG.md with the issue.
  • If applicable: Unit tests written, added test to CI.
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Updated relevant documentation (docs/) and code comments.
  • Re-reviewed Files changed in the Github PR explorer.

config.toml Outdated Show resolved Hide resolved
@andynog andynog requested review from romac and ancazamfir May 26, 2021 13:08
@romac romac marked this pull request as ready for review June 1, 2021 08:36
@romac romac removed the in progress label Jun 1, 2021
@romac romac merged commit a1e95e1 into master Jun 1, 2021
@romac romac deleted the andy/telemetry branch June 1, 2021 09:05
@adizere adizere mentioned this pull request Jun 1, 2021
9 tasks
hu55a1n1 pushed a commit to hu55a1n1/hermes that referenced this pull request Sep 13, 2022
* Initial telemetry support implementation (informalsystems#868)

* Refactored code for state and service. Replaced hyper with rouille (informalsystems#868)

* Initial logic to include the telemetry in the Supervisor (informalsystems#868)

* Refactored logic into server and service. Server working (informalsystems#868)

* Added new methods for state and server (informalsystems#868)

* Telemetry service logic working, recording a metric (informalsystems#868)

* Added more metrics (informalsystems#868)

* Added logic to disable/enable telemetry service and server (informalsystems#868)

* Added more metrics to service. Hookup the packet timeout metric (informalsystems#868)

* Move telemetry service into `ibc-telemetry` crate

* Move `metric!` macro into its own module

* Move telemetry config under `[telemetry]` section

* Disable telemetry by default, fix port to 3001

* Try to fix libm.so error

* Wrap telemetry state in Arc and simplify server a little

* Simplify server a bit more

* Fix glibc version mismatch between CI and Docker image

* Push telemetry handle down into workers

* Implement `workers`, `ibc_client_misbehaviours` and `receive_packets` metrics

* Add `ibc_client_update` metric

* Remove need for telemetry service by passing around the telemetry state

* Add ack and timeout metrics

* Fix compilation when telemetry feature is not included

* FMT

* Rename metric! macro to telemetry!

* Add `clippy --no-default-features` to CI

Co-authored-by: Andy Nogueira <me@andynogueira.dev>
Co-authored-by: Romain Ruetschi <romain@informal.systems>
Co-authored-by: Anca Zamfir <zamfiranca@gmail.com>
Co-authored-by: Adi Seredinschi <adi@informal.systems>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add telemetry to hermes - bootstrap
5 participants