OpenTelemetry with Datadog Configuration [not getting any traces sent to Agent]

[kencieszykowski]

Hey everyone,

I've been working with Datadog support to troubleshoot this issue specifically with Triton and was curious if anybody has a working configuration that they use to ingest OpenTelemetry traces from triton-inference-server running in Kubernetes.

For some details; we are running the following in EKS:
Triton Server r23.04
Datadog Agent 7.44.1
Kubernetes 1.25

The Datadog Agent's HTTP OTLP endpoint is enabled and available on every node at port 4318; and the receiving configuration is set up according to their documentation.

In Kubernetes here are the startup commands for Triton Server:

      containers:
      - args:
        - tritonserver
        - --model-repository=gs://[path-to-models]
        - --disable-auto-complete-config
        - --log-verbose=0
        - --allow-gpu-metrics=true
        - --metrics-interval-ms=1
        - --trace-config
        - mode=opentelemetry
        - --trace-config
        - level=TIMESTAMPS
        - --trace-config
        - rate=1
        - --trace-config
        - opentelemetry,url=http://$(DATADOG_TRACE_AGENT_HOSTNAME):4318

DATADOG_TRACE_AGENT_HOSTNAME is an env var that's pulling the node IP, and is ingesting properly.

We can run our models just fine, but unfortunately it does not seem like anything is being sent over to the Datadog Agent's receiving endpoint. I'm not entirely sure how to troubleshoot that Triton is actually sending traces; but if I hit the trace API endpoint I can verify that the trace configuration is getting populated as expected.

Does anybody have this successfully implemented in their deployment? I'm almost positive this is just a configuration issue somewhere along the line, but at my wits' end trying to track down where to go from here.

Any help or suggestions is appreciated.
Thanks!

[MatthieuToulemont]

I would be very interested in this too, having clear traces in Triton would help us tremendously

[koval]

hey @kencieszykowski, I was able to successfully get Triton traces in DataDog. One more tritonserver option you might want to add is --trace-config level=TIMESTAMPS. The default value for level is OFF which I think causes triton to not collect and send traces.

[MatthieuToulemont]

Amazing, did you have to modify your Datadog config to make it work ?

[koval]

@MatthieuToulemont Yes, the OLTP ingestion endpoint also needs to be enabled in the Datadog agent according to this guide https://docs.datadoghq.com/opentelemetry/otlp_ingest_in_the_agent/?tab=host

[MatthieuToulemont]

Sorry to revive this, but I somehow can't make this work and keep getting the error:

[Error] File: /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/opentelemetry-cpp/src/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:213 [OTLP HTTP Client] Session state: connection failed.Couldn't connect to server

my docker-compose is as follows:

tritonserver:
    runtime: nvidia
    image: ${TRITONSERVER_IMAGE_NAME}
    command: 
      - tritonserver
      - --model-control-mode=explicit
      - --trace-config
      - mode=opentelemetry
      - --trace-config
      - opentelemetry,url=http://dogstatsd:4318/v1/traces
      - --trace-config
      - level=TIMESTAMPS
      - --trace-config
      - rate=1
      - --model-repository=/backend/model_repository
      - --log-verbose=1
      - --log-warning=1
      - --log-error=1



dogstatsd:
    build:
      context: .
      dockerfile: ./datadog/Dockerfile
    environment:
     - ... 
     - DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=http://localhost:4318/v1/traces
    ports:
     - 4318:4318

The datadog doker file only contains the necessary code to make Datadog pull Triton metrics.

Did you ever run into this error ?

[koval]

In dogstatsd container DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT should be 0.0.0.0:4318 (assuming you are starting OLTP ingest DataDog component in your container). And then in tritonserver container you should have opentelemetry,url=http://dogstatsd:4318.

[MatthieuToulemont]

The dogstatsd container is datadog/agent:7.48.0-rc4 which I assume contains the OLTP ingest Datadog component you are mentioning.

I have set both:

• opentelemetry,url=http://dogstatsd:4318 in triton
• DD_OLTP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318

But now I get a 404 :

File: /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/opentelemetry-cpp/src/opentelemetry-cpp/expo,Body:404 page not foundptions: nosniff GMTtf-8TP Client] \

Export failed, Status:404, Header:    Content-Length: 19

But that is already better than before. I got it working by setting for the tritonserver:

``opentelemetry,url=http://dogstatsd:4318/v1/traces`

This issue helped me crack it .

Thank you @koval for the help !!