docs: document Tempo instrumentation (#797)

* docs: document Tempo instrumentation

* Apply suggestions from code review

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update monitoring.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

docs/tempo/website/operations/monitoring.md

@@ -5,12 +5,45 @@ weight: 4
 
 # Monitoring Tempo
 
-The Tempo repository has a [mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) that includes a
-set of dashboards, rules and alerts. Together, these can be used to monitor Tempo in production.
+Tempo is instrumented to expose metrics, logs and traces.
+Additionally, the Tempo repository has a [mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) that includes a
+set of dashboards, rules and alerts.
+Together, these can be used to monitor Tempo in production.
+
+## Instrumentation
+
+Tempo is already instrumented with metrics, logs and traces.
+These can be collected to observe Tempo.
+
+#### Metrics
+
+Tempo is instrumented with [Prometheus metrics](https://prometheus.io/).
+It emits RED metrics for most services and backends.
+The [Tempo mixin](#dashboards) provides several dashboards using these metrics.
+
+#### Logs
+
+Tempo emits logs in the `key=value` ([logfmt](https://brandur.org/logfmt)) format.
+
+#### Traces
+
+Tempo uses the [Jaeger Golang SDK](https://github.com/jaegertracing/jaeger-client-go) for tracing instrumentation.
+As of this writing, the complete read path and some parts of the write of Tempo are instrumented for tracing.
+
+The tracer can be configured [using environment variables](https://github.com/jaegertracing/jaeger-client-go#environment-variables).
+To enable tracing, set one of the following: `JAEGER_AGENT_HOST` and `JAEGER_AGENT_PORT`, or `JAEGER_ENDPOINT`.
+
+The Jaeger client uses remote sampling by default, if the management server is not available no traces will be sent.
+To always send traces (no sampling), set the following environment variables:
+
+```
+JAEGER_SAMPLER_TYPE=const
+JAEGER_SAMPLER_PARAM=1
+```
 
 ## Dashboards
 
-The Tempo mixin has four Grafana dashboards in the `yamls` folder that can be downloaded and imported into your Grafana UI.
+The [Tempo mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) has four Grafana dashboards in the `yamls` folder that you can download and import into your Grafana UI.
 Note that at the moment, these work well when Tempo is run in a k8s environment and metrics scraped have the
 `cluster` and `namespace` labels!