Skip to content
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.

Add metric handler for grafana #1400

Closed
jrconlin opened this issue Jun 17, 2020 · 4 comments · Fixed by #1406 or #1407
Closed

Add metric handler for grafana #1400

jrconlin opened this issue Jun 17, 2020 · 4 comments · Fixed by #1406 or #1407
Assignees
@pjenvey
Labels
3 Estimate - m - This is a small change, but there's some uncertainty.

Comments

@jrconlin
Copy link
Member

Currently tags are getting dropped after moving from Datadog. We are now defaulting to TwistedMetrics, which ignores tags.

Grafana tags should be specified as key=val
@jrconlin jrconlin self-assigned this Jun 17, 2020
@pjenvey
Copy link
Member

pjenvey commented Jun 17, 2020

The grafana/influxdb metrics collector (telegraf) prefers key=val but our key:val format seems to work with it, given the example of autopush-rs emitting key:val via cadence.

Problem here is the underlying statsd client of TwistedMetrics (txtstatsd) does not support metrics tags.

@tublitzed tublitzed added the 3 Estimate - m - This is a small change, but there's some uncertainty. label Jun 22, 2020
@sciurus
Copy link

sciurus commented Jun 22, 2020

Telegraf supports dogstatsd protocol, so you should be able to revert back to however your statsd client was configured before.

https://github.com/mozilla-services/cloudops-docs/blob/master/Infrastructure/Monitoring/Monitoring%20Guidance.md#collect-high-volume-application-metrics are the current recommendations

jrconlin added a commit that referenced this issue Jun 22, 2020
@jrconlin
feat: Add tags back to metrics.
1f927bc 
per ops request, this switches metrics to using markus. This also
changes the DatadogMetrics library to TaggedMetrics, and becomes
the default metrics library if the config value `statsd_host` is set.

Closes #1400
@jrconlin jrconlin mentioned this issue Jun 22, 2020
Closed
@jrconlin
Copy link
Member Author

The code uses a dedicated DatadogMetrics library, so guessing that it's more than just tweaking a config to use a crap API key. Instead I just wired in the markus library.

jrconlin added a commit that referenced this issue Jun 26, 2020
@jrconlin
feat: Add tags back to metrics.
be216ea 
While this package does not use the officially sanctioned "markus"
metrics package, it does use the same Datadog library.

Closes #1400
@jrconlin jrconlin mentioned this issue Jun 26, 2020
Merged
pjenvey pushed a commit that referenced this issue Jun 26, 2020
@jrconlin
feat: Add tags back to metrics. (#1406)
7ac68d6 
* feat: Add tags back to metrics.

While this package does not use the officially sanctioned "markus"
metrics package, it does use the same Datadog library.

Closes #1400
@pjenvey pjenvey reopened this Jun 27, 2020
@pjenvey pjenvey assigned pjenvey and unassigned jrconlin Jun 27, 2020
@pjenvey
Copy link
Member

pjenvey commented Jun 27, 2020

Oops: I thought the existing dogstatsd backend we use would work w/ statsd, apparently it doesn't.

pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metric + tags
09b284c 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain Datadog(that's statsd compatible) backend

+ upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Closes #1400
pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metric + tags
1ef3d26 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain Datadog(that's statsd compatible) backend

+ upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Closes #1400
@pjenvey pjenvey mentioned this issue Jun 27, 2020
Merged
pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metric + tags
ef737f0 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain Datadog(that's statsd compatible) backend

+ upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Closes #1400
pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metrics + tags
6a4b5ea 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain Datadog(that's statsd compatible) backend

+ upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Closes #1400
pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metrics + tags
5d3ce80 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain Datadog(that's statsd compatible) backend

+ upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Closes #1400
pjenvey added a commit that referenced this issue Jun 27, 2020
@pjenvey
feat: fixup metrics + tags
2eb6791 
remove datadog's ThreadStats (as it only speaks datadog's http protocol)
w/ markus backed by the plain statsd compatible Datadog backend

upgrade datadog to avoid its own telemetry metrics producing garbage
tags

Co-authored-by: JR Conlin <jrconlin@gmail.com>

Closes #1400
@pjenvey pjenvey mentioned this issue Jul 9, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@pjenvey pjenvey

Labels
3 Estimate - m - This is a small change, but there's some uncertainty.
Projects
None yet
Milestone
No milestone
4 participants
@pjenvey @jrconlin @sciurus @tublitzed