Grafana dashboard updates #5255

Tom-Newton · 2024-04-19T18:47:36Z

Tracking issue

Why are the changes needed?

There are a few issues with the dashboards as is. This includes some bugs and some outdated metric names.

What changes were proposed in this pull request?

This is a series of incremental changes I ended up making when trying to understand flyte propeller performance and debug a problem on our network.
- Add a few extra graphs/shuffle some around a bit.
- Fix some aggregation bugs e.g. missing rate funciton while the title says rate
- Fix some axis labels.
- Correct some metrics names based on flyte-core 1.11
- Add a few descriptions.
Probably this is still a long way from perfect but I think its probably still worth contributing.

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly. Not applicable
All new and existing tests passed. Not applicable
All commits are signed-off.

Related PRs

Docs link

Tom-Newton · 2024-04-19T19:50:01Z

It looks like the CI failure is because matplotlib's website is down and breaking a docs build.

davidmirror-ops

Thank you!

davidmirror-ops · 2024-04-23T19:31:22Z

@neverett any idea why CI fails with a doc that's not changed by this PR?

Tom-Newton · 2024-04-23T19:43:07Z

I think the doc failure was caused by matplotlib's website going down. Is it still failing?

Tom-Newton · 2024-04-23T19:44:34Z

Actually that looks like a different error now

neverett · 2024-04-23T20:23:27Z

@Tom-Newton @davidmirror-ops I think you may need to merge master in to pick up changes from #5254 since that introduced some significant updates to the docs build

davidmirror-ops · 2024-05-07T18:10:54Z

@Tom-Newton could you merge master and try again?

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Tom-Newton · 2024-05-07T18:46:46Z

Sorry for being so slow. I just rebased on master 🤞

codecov · 2024-05-07T19:16:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.09%. Comparing base (8db9901) to head (5af4a10).
Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5255       +/-   ##
===========================================
- Coverage   79.30%   61.09%   -18.21%     
===========================================
  Files          18      794      +776     
  Lines        1295    51213    +49918     
===========================================
+ Hits         1027    31289    +30262     
- Misses        204    17043    +16839     
- Partials       64     2881     +2817

Flag	Coverage Δ
unittests-datacatalog	`69.31% <ø> (?)`
unittests-flyteadmin	`58.86% <ø> (?)`
unittests-flytecopilot	`17.79% <ø> (?)`
unittests-flytectl	`68.30% <ø> (?)`
unittests-flyteidl	`79.30% <ø> (ø)`
unittests-flyteplugins	`61.94% <ø> (?)`
unittests-flytepropeller	`57.32% <ø> (?)`
unittests-flytestdlib	`65.75% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* Add enqueued workflows graph Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Update queue metrics Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Better aggregations Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add enqueued workflows graph Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix trailing comma Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add transition latency, etc Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Check-in latest dashboard Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fixes Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add informer stats to dashboard Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add more stats on workflow store Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * More round metrics Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix Y axis labels Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Check-in latest dashboard Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add garbage collection stats Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix scale on cache hit rate Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Working flytepropeller grpc histograms Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Change to a single accumulated graph Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add GRPC histogram on flyteadmin too Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix admin metrics name Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Update workqueue metric names for flyte 1.11.0 Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add other queues to queue graphs Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add round success counter, remove workqueue latencies Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Split enqueues by type and add missing rate for skip rate metric Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Don't use rate for round latency graphs Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix metric name for plugin failures and make it a rate Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix duplicate refIds Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix typo Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Update lots of title and units Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add round errors graph Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add total round rate Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Group metrics onto the same graphs were relevant Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Small updates Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix streaks graph Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * rename legend Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Rename Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add descriptions Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add a missing title Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Remove unused imports Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Adjust a description Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * More minor updates Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> --------- Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Tom-Newton force-pushed the tomnewton/dashboard_updates branch 2 times, most recently from 402ba45 to 598bd8c Compare April 19, 2024 19:32

Tom-Newton force-pushed the tomnewton/dashboard_updates branch from fb208a1 to c35d80a Compare April 19, 2024 20:09

Tom-Newton marked this pull request as ready for review April 19, 2024 20:09

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels Apr 19, 2024

eapolinario assigned davidmirror-ops Apr 19, 2024

davidmirror-ops approved these changes Apr 23, 2024

View reviewed changes

Tom-Newton added 16 commits May 7, 2024 19:45

Add enqueued workflows graph

644d65c

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Update queue metrics

622d163

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Better aggregations

94e307a

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add enqueued workflows graph

60bea36

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix trailing comma

8edd38e

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add transition latency, etc

c490077

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix

d1e5da2

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Check-in latest dashboard

828beb2

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fixes

06baf7b

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add informer stats to dashboard

17153bf

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add more stats on workflow store

8ae2c2e

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

More round metrics

e93b7e5

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix Y axis labels

78a92e2

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Check-in latest dashboard

071ca91

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add garbage collection stats

103e053

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix scale on cache hit rate

5180e65

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Tom-Newton added 25 commits May 7, 2024 19:45

Working flytepropeller grpc histograms

5f0aa76

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Change to a single accumulated graph

47b4c73

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add GRPC histogram on flyteadmin too

547057d

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix admin metrics name

0efd55a

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Update workqueue metric names for flyte 1.11.0

34a6d35

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add other queues to queue graphs

5d02c9f

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add round success counter, remove workqueue latencies

65db5fc

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Split enqueues by type and add missing rate for skip rate metric

f7610ef

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Don't use rate for round latency graphs

52e3a36

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix metric name for plugin failures and make it a rate

a44e788

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix duplicate refIds

7116a3f

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix typo

6222ad9

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Update lots of title and units

1ce7f3c

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add round errors graph

57ff180

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add total round rate

2766041

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Group metrics onto the same graphs were relevant

53dce4f

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Small updates

22a5914

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Fix streaks graph

5965cf7

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

rename legend

73e2230

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Rename

802b7ba

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add descriptions

2003d7c

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Add a missing title

da613c9

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Remove unused imports

6d1f00d

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Adjust a description

d9076ab

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

More minor updates

5af4a10

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

Tom-Newton force-pushed the tomnewton/dashboard_updates branch from c35d80a to 5af4a10 Compare May 7, 2024 18:46

Tom-Newton merged commit 2f38d65 into flyteorg:master May 12, 2024
51 of 52 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grafana dashboard updates #5255

Grafana dashboard updates #5255

Tom-Newton commented Apr 19, 2024 •

edited

Loading

Tom-Newton commented Apr 19, 2024

davidmirror-ops left a comment

davidmirror-ops commented Apr 23, 2024

Tom-Newton commented Apr 23, 2024

Tom-Newton commented Apr 23, 2024

neverett commented Apr 23, 2024

davidmirror-ops commented May 7, 2024

Tom-Newton commented May 7, 2024

codecov bot commented May 7, 2024 •

edited

Loading

Grafana dashboard updates #5255

Grafana dashboard updates #5255

Conversation

Tom-Newton commented Apr 19, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Tom-Newton commented Apr 19, 2024

davidmirror-ops left a comment

Choose a reason for hiding this comment

davidmirror-ops commented Apr 23, 2024

Tom-Newton commented Apr 23, 2024

Tom-Newton commented Apr 23, 2024

neverett commented Apr 23, 2024

davidmirror-ops commented May 7, 2024

Tom-Newton commented May 7, 2024

codecov bot commented May 7, 2024 • edited Loading

Codecov Report

Tom-Newton commented Apr 19, 2024 •

edited

Loading

codecov bot commented May 7, 2024 •

edited

Loading