debug

Remote debugging

How to use remote debugging with Tempo. This example builds on the local example, for questions about the general setup, please refer to its readme file.

Although it's possible to debug Tempo with dlv debug, this approach also has disadvantages in scenarios where it is desirable to run Tempo inside a container. This example demonstrates how to debug Tempo running in docker compose.

The make target docker-tempo-debug compiles tempo without optimizations and creates a docker image that runs Tempo using dlv exec.

1. Build the Tempo debug image in the root directory of the project:

make docker-tempo-debug

To distinguish the debug image from the conventional Tempo image, it's tagged with grafana/tempo-debug. To check if the image is present:

docker images | grep grafana/tempo-debug

grafana/tempo-debug                            latest                         3d6789d20dc3   2 days ago      112MB

2. Take a look at tempo service in the docker-compose.yaml. The environment variable DEBUG_BLOCK controls whether delve halts the execution of Tempo until a debugger is connected. Setting this option to 1 is helpful to debug errors during the start-up phase.

3. Now, start up the stack from this directory.

docker compose up -d

At this point, the following containers should be running:

docker compose ps

       Name                     Command               State                            Ports
---------------------------------------------------------------------------------------------------------------------
debug_grafana_1      /run.sh                          Up      0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
debug_k6-tracing_1   /k6-tracing run /example-s ...   Up
debug_prometheus_1   /bin/prometheus --config.f ...   Up      0.0.0.0:9090->9090/tcp,:::9090->9090/tcp
debug_tempo_1        /entrypoint-debug.sh -conf ...   Up      0.0.0.0:14268->14268/tcp,:::14268->14268/tcp,
                                                              0.0.0.0:2345->2345/tcp,:::2345->2345/tcp,
                                                              0.0.0.0:3200->3200/tcp,:::3200->3200/tcp,
                                                              0.0.0.0:4317->4317/tcp,:::4317->4317/tcp,
                                                              0.0.0.0:4318->4318/tcp,:::4318->4318/tcp,
                                                              0.0.0.0:9411->9411/tcp,:::9411->9411/tcp

4. The Tempo container exposes delves debugging server at port 2345 and it's now possible to connect to it and debug the program. If you prefer to operate delve from the command line, you can connect to it via:

dlv connect localhost:2345

Golang users can connect with the Go Remote run configuration:

5. To stop the setup, use:

docker compose down -v