Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docdb][metrics] Add metrics for number of running flushes/compactions #1805

Closed
bmatican opened this issue Jul 15, 2019 · 3 comments
Closed
Assignees
Labels
area/docdb YugabyteDB core features

Comments

@bmatican
Copy link
Contributor

Now that we have the new PriorityPool for flushes and compactions, we should add specific metrics, split by type (flushes, small/large compactions):

  • number of running tasks
  • number of paused tasks
  • number of queued tasks
@bmatican bmatican added the area/docdb YugabyteDB core features label Jul 15, 2019
@bmatican bmatican assigned rajukumaryb and unassigned spolitov Nov 4, 2019
@bmatican
Copy link
Contributor Author

bmatican commented Nov 4, 2019

We currently graph these in YW: managed/src/main/resources/metrics.yml

  • read/write bytes
  • files per compaction
  • time taken

With the additional pause/resume mechanic, there's a couple of interesting things to consider:

  • the sheer counts mentioned above (running vs paused vs queued up)
  • how many files are currently picked by some of these compactions (which might just be the same as the "files per compaction" metric we're already using?)
  • similarly, how many bytes are currently picked?

Passing this to @rajukumaryb as he's already looking at some rocksdb work.

@bmatican
Copy link
Contributor Author

bmatican commented Nov 2, 2021

This one is a bit less well described and we should figure out which metrics make sense. So far, I noted 3 categories above, which in turn should be split by compactions that are active vs paused (ie: pre-empted)

  • number of tasks
  • number of files
  • data size

The priority thread pool implementation is src/yb/util/priority_thread_pool.cc

For metrics usage, you can search for examples using METRIC_DEFINE_gauge. An example of another pretty static set of gauges that we setup in the generic thread utility: src/yb/util/thread.cc

Good people references: @spolitov and @jmeehan16

@bmatican
Copy link
Contributor Author

bmatican commented Dec 8, 2021

@arybochkin I put this on the backburner for now, to switch into some tablet splitting work

parikhdev0 added a commit that referenced this issue Feb 15, 2022
…tes split by active, paused, and queued.

Summary: Added 9 new metrics for compaction tasks, files and bytes split by active, paused, and queued.

Test Plan: Used curl -q http://127.0.0.1:9000/prometheus-metrics to observe the metrics updating during compactions, and reverting back to 0 afterwards.

Reviewers: timur, rsami, jmeehan

Reviewed By: jmeehan

Subscribers: jenkins-bot, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D15005
jayant07-yb pushed a commit to jayant07-yb/yugabyte-db that referenced this issue Mar 8, 2022
…, and bytes split by active, paused, and queued.

Summary: Added 9 new metrics for compaction tasks, files and bytes split by active, paused, and queued.

Test Plan: Used curl -q http://127.0.0.1:9000/prometheus-metrics to observe the metrics updating during compactions, and reverting back to 0 afterwards.

Reviewers: timur, rsami, jmeehan

Reviewed By: jmeehan

Subscribers: jenkins-bot, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D15005
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features
Projects
None yet
Development

No branches or pull requests

5 participants