Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

excessive metrics sent to statsdaemon, leads to high statsdaemon usage #335

Closed
Dieterbe opened this issue Oct 3, 2016 · 3 comments
Closed
Assignees

Comments

@Dieterbe
Copy link
Contributor

Dieterbe commented Oct 3, 2016

(this is using the carbon input plugin for ingestion)
https://snapshot.raintank.io/dashboard/snapshot/aqlU18OPIdtyaZ6vtMOYZZGnNERZfx6B
=> large amounts of timers to statsdaemon. those cause the vast majority of cpu since they require most computation.

with ngrep -d any -W byline port 8125 | grep 'ms' | cut -d ':' -f1 I see almost all timers are either:

  1. carbon.metrics_per_message (can be removed since it's always 1)
  2. idx.memory.get_duration
  3. idx.{memory,cassandra}.add_duration

solving this should significantly reduce statsdaemon cpu usage.
we can also look at some other metrics, like some counts, though i suspect those will make less of a difference.
anyway, can easily be done in raintank-docker.

@Dieterbe
Copy link
Contributor Author

Dieterbe commented Oct 4, 2016

this requires adding rate args to the statsd functions and their invocations. i'm going to wait for #320 since that is such a huge changeset, otherwise we'll have a nasty conflict

@Dieterbe
Copy link
Contributor Author

now that #400 is stabilizing, started working on this. will just get rid of statsd[aemon]. early WIP in #384

@Dieterbe Dieterbe self-assigned this Dec 15, 2016
@Dieterbe Dieterbe added this to the hosted-metrics-alpha milestone Dec 15, 2016
@Dieterbe
Copy link
Contributor Author

Dieterbe commented Jan 3, 2017

fixed via #384

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant