Cloud Native Rejekts Europe - Paris - 2024

Building Resilient Observability Pipelines using OpenTelemetry Collector.

Prerequisites

This tutorial requires a docker and Kubernetes cluster, refer to Kind for a local Kubernetes cluster installations and Telemetrygen tool.

Starting Kind Cluster

 kind create cluster -n rejekts-2024 --config kind.yaml

Installing Telemetrygen

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

Deploy observability backend

This tutorial uses Prometheus, Grafana, Loki and Tempo as observability backend to store metrics, logs and traces.

Deploy the backend systems:

kubectl apply -f backend.yaml

Install Prometheus via Helm Chart:

helm install prometheus prometheus-community/kube-prometheus-stack --namespace prometheus --create-namespace

For visualisation port forward Grafana:

kubectl port-forward -n backend svc/grafana 3000:3000

OpenTelemetry

Installing CertManager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.yaml

Installing the OpenTelemetry Operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Enable the Prometheus Feature Gates

- --feature-gates=+operator.autoinstrumentation.go,+operator.autoinstrumentation.nginx,+operator.observability.prometheus

Deploy the OpenTelemetry Collector Local

kubectl apply -f otel-local.yaml

Deploy the OpenTelemetry Collector as Backend

kubectl apply -f otel-backend.yaml

Deploy the ServiceMonitors to send metrics to Promethus

kubectl apply -f servicemonitor.yaml

Run the Telemetrygen to generate Logs

Forward a connection to OpenTelemetry Local

kubectl port-forward svc/local-collector 4317:4317

telemetrygen logs --otlp-insecure --logs 1_000_000

Check the Logs being transmited on Grafana Dashboard

http://localhost:3000/grafana/d/fb8648c6-5488-4966-bda0-298c12512f2e/rejekts-2024?orgId=1&from=now-15m&to=now

Stop the Backend Instance to simulate an Outage

kubectl delete -f otel-backend.yaml

Check the Queue size incresing every second

http://localhost:3000/grafana/d/fb8648c6-5488-4966-bda0-298c12512f2e/rejekts-2024?orgId=1&from=now-15m&to=now

Redeploy the OTel Collector Backend

kubectl apply -f otel-backend.yaml

Check the Queue getting back to Zero.

http://localhost:3000/grafana/d/fb8648c6-5488-4966-bda0-298c12512f2e/rejekts-2024?orgId=1&from=now-15m&to=now