From 3efc8cac9b77ca42d2a8f99830be2b51144b99e1 Mon Sep 17 00:00:00 2001 From: Koenraad Verheyden Date: Wed, 30 Jun 2021 14:41:17 +0200 Subject: [PATCH] docs: document Tempo instrumentation (#797) * docs: document Tempo instrumentation * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update monitoring.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> --- docs/tempo/website/operations/monitoring.md | 39 +++++++++++++++++++-- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/tempo/website/operations/monitoring.md b/docs/tempo/website/operations/monitoring.md index 7d1ec0a619e..fc22e5db0db 100644 --- a/docs/tempo/website/operations/monitoring.md +++ b/docs/tempo/website/operations/monitoring.md @@ -5,12 +5,45 @@ weight: 4 # Monitoring Tempo -The Tempo repository has a [mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) that includes a -set of dashboards, rules and alerts. Together, these can be used to monitor Tempo in production. +Tempo is instrumented to expose metrics, logs and traces. +Additionally, the Tempo repository has a [mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) that includes a +set of dashboards, rules and alerts. +Together, these can be used to monitor Tempo in production. + +## Instrumentation + +Tempo is already instrumented with metrics, logs and traces. +These can be collected to observe Tempo. + +#### Metrics + +Tempo is instrumented with [Prometheus metrics](https://prometheus.io/). +It emits RED metrics for most services and backends. +The [Tempo mixin](#dashboards) provides several dashboards using these metrics. + +#### Logs + +Tempo emits logs in the `key=value` ([logfmt](https://brandur.org/logfmt)) format. + +#### Traces + +Tempo uses the [Jaeger Golang SDK](https://github.com/jaegertracing/jaeger-client-go) for tracing instrumentation. +As of this writing, the complete read path and some parts of the write of Tempo are instrumented for tracing. + +The tracer can be configured [using environment variables](https://github.com/jaegertracing/jaeger-client-go#environment-variables). +To enable tracing, set one of the following: `JAEGER_AGENT_HOST` and `JAEGER_AGENT_PORT`, or `JAEGER_ENDPOINT`. + +The Jaeger client uses remote sampling by default, if the management server is not available no traces will be sent. +To always send traces (no sampling), set the following environment variables: + +``` +JAEGER_SAMPLER_TYPE=const +JAEGER_SAMPLER_PARAM=1 +``` ## Dashboards -The Tempo mixin has four Grafana dashboards in the `yamls` folder that can be downloaded and imported into your Grafana UI. +The [Tempo mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) has four Grafana dashboards in the `yamls` folder that you can download and import into your Grafana UI. Note that at the moment, these work well when Tempo is run in a k8s environment and metrics scraped have the `cluster` and `namespace` labels!