How to enable optional tags from the telemetry sent by datadog-agent source and datadog sink?

[dcebola-mollie]

Hi team,
Currently trying to enhance the monitoring for our vector setup. We run vector as an aggregator between our datadog-agent source and datadog-metrics sink. One of the limitations we are facing is on the optional tags being sent with the telemetry data of the datadog-agent source and datadog-metrics sink, namely the podname tag.

In the documentation (e.g. source.component_received_events_count or sink.component_received_events_count) some metrics have this optional tag available but when observing it in datadog, it lacks the tag. Currently the reported metric has the component_id, component_type, component_kind, env, version, host and service tags. Do we have to configure it? If so what would be steps necessary to enable it?

[jszwedko]

Hi @dcebola-mollie !

I believe that pod_name tag is added to component_received_events_count but only for the kubernetes_logs source. I'm not sure any sinks actually add that tag. I think that might be a bug in the docs. In your case, would you want tho pod_name tag to be the name of the Vector pod?

Admittedly the docs here are a bit confusing because only some components add some of the optional tags.

[dcebola-mollie]

Hi @jszwedko,

Thank you for the answer, there might be a bug in the docs indeed. We were hopeful that the pod_name and container_name would increase our ability to to understand which services might be impacting utilisation the most. By comparing the number of request received at the sources, checking the number of dropped events and then evaluating the actual received events by the sinks, we can get a general understanding of the system. The pod_name and container_name tags, from where the data is originated, would increase the granularity of the information and help us pinpoint potential issues from specific origins.

Because we host the vector resources in a centralised cluster and receive telemetry data via a load balancer, the host level granularity does not give any insight regarding the origin of the data.

[jszwedko]

Ah, gotcha, yeah. I think you'd be interested in this feature request: #7403.

As a workaround you may be able to generate the metrics you want from your events using the log_to_metric transform.

[dcebola-mollie]

Thank you for the answers! I'll look into that.