Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics to operator #20

Merged
merged 10 commits into from
Jul 23, 2024
Merged

Add metrics to operator #20

merged 10 commits into from
Jul 23, 2024

Conversation

v9n
Copy link
Member

@v9n v9n commented Jul 16, 2024

Fixed ENG-844

Update the endpoint that generate the prometheus metric.

Convert all the metrics to vector to supports:

  • operator address: This will allow us to later scrape and group by operator address.
  • version: the version. Useful to find operator that has not update. Also being used on our telemetry page

(Version is just a number for demo purpose on this screenshot to show how it's appear)
CleanShot 2024-07-23 at 04 59 40@2x

Metrics generator:

  • ping check duration|total, worker looper count
  • Add "task received" metrics. Right now this isn't connected with real tasks yet, but the mechanism allow us to generate metrics, and therefor can update our dashboard.
  • Add retrying in the sync loop so if the grpc stream is broken, we can re-connect to fetch new tasks

Change to connection pool:

  • Record versions and metric port. We need the metrics port to later on connect the prometheus endpoint to our server to scrape all operator metrics, group by operator address.

CleanShot 2024-07-23 at 05 04 00@2x

@v9n v9n marked this pull request as draft July 16, 2024 12:09
@v9n v9n force-pushed the add-metrics-to-operator branch from f2acc51 to ddb2223 Compare July 16, 2024 21:36
@v9n v9n marked this pull request as ready for review July 23, 2024 11:58
@v9n v9n requested a review from chrisli30 July 23, 2024 12:05
Copy link
Member

@chrisli30 chrisli30 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@v9n v9n merged commit 786da8a into main Jul 23, 2024
2 checks passed
@v9n v9n deleted the add-metrics-to-operator branch July 23, 2024 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants