Identification of data in multiple telegraf

[jyotiverma03]

Feature Request

Opening a feature request kicks off a discussion.

Proposal: Identification of multiple telelgraf in output plugin

Current behavior: Not able to identify which data is coming from which telegraf

Desired behavior: User should be able to filter data from a given tag name

Use case: I have a use case where I am integrating multiple telegrafs (child telegraf) data into one (Parent telegraf) and sending parent telegraf's data to prometheus.

So the problem we are facing is that we want to identify from which child telegraf data is coming or we can say we want to filter metrics data on the basis of child telegraf's tag or given name. So it becomes necessary for us to give tag to input plugins. But when child input's plugin's data being feed to parent telegraf then there is no way to find out from which child telegraf data is coming.

[danielnelson]

You should use the host tag to identify the host Telegraf is running on, this is the most common way to identify the Telegraf instance, and it should almost never be needed to run more than one Telegraf per host.

However, if for some reason you are running multiple Telegraf's, you can use the global_tags configuration to set a tag that will be added to all metrics.

[jyotiverma03]

@danielnelson I am using different telegraf for different hosts but when I give input of these telegrafs to one telegraf and then send those metrics to prometheus. So at this point of time, I am not able to identify the origin of metrics.

[danielnelson]

Can you describe the full path of the metrics from creation to prometheus, including which plugins are used?

[jyotiverma03]

Hello @danielnelson ,
Currently, I am using two telegraf as inputs of one telegraf(that is acting as an aggregator).
NF-1:
NF-1 docker-compose.yml

version: '3'
services:
telegraf:
image: telegraf
container_name: NF-1
restart: always
environment:
HOST_PROC: /rootfs/proc
HOST_SYS: /rootfs/sys
HOST_ETC: /rootfs/etc
volumes:
- ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys:/rootfs/sys:ro
- /proc:/rootfs/proc:ro
- /etc:/rootfs/etc:ro
- ./tmp/:/opt/tele

NF-1 config file (telegraf.conf)

[agent]
hostname = "NF-1"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
[inputs.cpu.tags]
#tag1 = "NF-1"
[[outputs.file]]
files = ["stdout", "/opt/tele/metrics.out"]
data_format = "json"
[[outputs.http]]
url = "http://URL:8123/telegraf"
method = "POST"
data_format = "json"

NF-2 docker-compose.yml

version: '3'
services:
telegraf:
image: telegraf
container_name: NF-2
restart: always
environment:
HOST_PROC: /rootfs/proc
HOST_SYS: /rootfs/sys
HOST_ETC: /rootfs/etc
volumes:
- ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys:/rootfs/sys:ro
- /proc:/rootfs/proc:ro
- /etc:/rootfs/etc:ro
- ./tmp/:/opt/tele

NF-2 Config file

[agent]
hostname = "NF-2"
################### Inputs ################
[[inputs.http]]
urls = [
"http://URL:32758/metrics"
]
method = "GET"
headers = {"Accept" = "application/json"}
data_format = "json"
[[inputs.cpu]]
name_override = "cpu_agg"
percpu = false
totalcpu = true
name_suffix = "_NF-2"
collect_cpu_time = true
report_active = false
################### Outputs ################
[[outputs.file]]
files = ["stdout", "/opt/tele/metrics.out"]
data_format = "json"
[[outputs.http]]
url = "http://URL:8123/telegraf"
method = "POST"
data_format = "json"

Aggregator telegraf's docker-compose

version: '3'
volumes:
grafana_volume: {}
services:
prometheus:
build: './prometheus'
container_name: 'prometheus'
volumes:
- ./prometheus/:/etc/prometheus/
expose:
- 9090
ports:
- "9090:9090"
telegraf:
image: telegraf
container_name: NF-Aggregator
restart: always
environment:
HOST_PROC: /rootfs/proc
HOST_SYS: /rootfs/sys
HOST_ETC: /rootfs/etc
volumes:
- ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys:/rootfs/sys:ro
- /proc:/rootfs/proc:ro
- /etc:/rootfs/etc:ro
- ./tmp/:/opt/tele
ports:
- "9273:9273"
- "8123:8123"
grafana:
image: grafana/grafana:6.2.0
env_file: ./configuration.env
container_name: grafana.6.2.0
volumes:
- ./grafana_volume:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
command:
- COPY ./grafana/config.ini /etc/grafana/config.ini
restart: unless-stopped
ports:
- 3000:3000

Telegraf aggregator's config:

[agent]
hostname = "telegraf-aggrigator"
[[inputs.http_listener_v2]]
service_address = ":8123"
path = "/telegraf"
data_format = "json"
methods = ["POST"]
[[outputs.file]]
files = ["stdout", "/opt/tele/metrics.out"]
data_format = "json"
[[outputs.prometheus_client]]
listen = ":9273"

So when data comes at prometheus then I am not able to identify which data comes from which metrics. Let me know if you get something from this.

[danielnelson]

I notice you are using data_format = "json" to relay the metrics from the edge Telegraf to the aggregating/batching Telegraf, this will be problematic because this data format does not round trip without modification (#2252). This results in metrics not being received in the same form as they are originally created and also the loss of tags.

What I recommend is to switch to data_format = "influx" for transfer between Telegraf instances, which will preserve the original host tag. Then you can use this tag to determine which host created the original metric.

[jyotiverma03]

@danielnelson Thanks, I will try this and let you know whether it works or not.

[jyotiverma03]

@danielnelson Thanks, This solution worked!!!