internal gather does not work

[rlex]

Relevant telegraf.conf:

N/A, internal metric

System info:

Debian 10, Telegraf 1.13.2 (git: HEAD 6dad859)

Steps to reproduce:

Break any plugin (ie run systemd_units on system w/o systemd)

Expected behavior:

SELECT mean("errors") FROM "internal_gather" WHERE ("input" = 'systemd_units') AND $timeFilter GROUP BY time($__interval) fill(null) should return value > 0

Actual behavior:

SELECT mean("errors") FROM "internal_gather" WHERE ("input" = 'systemd_units') AND $timeFilter GROUP BY time($__interval) fill(null) always return 0

Additional info:

internal_agent gather_errors works and shows gather errors, but per-plugin internal_gather do not.

[ssoroka]

Thanks for the bug report!

gather_errors is a global metric that's collected across all inputs.

I've tested locally with this config and I do get input-specific errors from internal_gather. Here's an example:

internal_gather,host=me.local,input=procstat,version=unknown errors=20i,metrics_gathered=20i,gather_time_ns=96471i 1582646190000000000

I'd need to see your specific config for systemd_units to know if the input name in your query is correct. Let me know what you think.

Thanks

[rlex]

Here is my full config:

### MANAGED BY telegraf ANSIBLE ROLE ###

[global_tags]


# Configuration for telegraf agent
[agent]
    interval = "10s"
    debug = false
    hostname = "asd.asd.asd.asd"
    round_interval = true
    flush_interval = "10s"
    flush_jitter = "0s"
    collection_jitter = "0s"
    metric_batch_size = 1000
    metric_buffer_limit = 10000
    quiet = false
    logfile = ""
    omit_hostname = false

###############################################################################
#                                  OUTPUTS                                    #
###############################################################################

[[outputs.influxdb]]
    urls = ["https://asd"]
    database = "asd"
    username = "asd"
    password = "asd"
    retention_policy = ""
    write_consistency = "any"
    timeout = "5s"
[[outputs.prometheus_client]]
    listen = ":9755"
    collectors_exclude = ["gocollector", "process"]
    metric_version = 2
    ip_range = ["10.55.55.0/24"]
    basic_username = "asd"
    basic_password = "asd"

###############################################################################
#                                  INPUTS                                     #
###############################################################################

[[inputs.cpu]]
    percpu = true
    totalcpu = true
    fielddrop = ["time_*"]
[[inputs.disk]]
    ignore_fs = ["tmpfs", "devtmpfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.system]]
[[inputs.swap]]
[[inputs.internal]]
[[inputs.kernel]]
[[inputs.processes]]
[[inputs.interrupts]]
[[inputs.linux_sysctl_fs]]
[[inputs.kernel_vmstat]]
[[inputs.systemd_units]]
[[inputs.net]]
[[inputs.netstat]]
[[inputs.nstat]]
    dump_zeros = true
[[inputs.conntrack]]
    files = ["ip_conntrack_count","ip_conntrack_max","nf_conntrack_count","nf_conntrack_max"]
    dirs = ["/proc/sys/net/ipv4/netfilter","/proc/sys/net/netfilter"]
[[inputs.filecount]]
    directories = ["/tmp", "/var/tmp"]


###############################################################################
#                                  PROCESSORS                                    #
###############################################################################

Running on rather old system without systemd and with old redis version
I see errors in logs:

2020-02-25T17:44:20Z E! [inputs.systemd_units] Error in plugin: exec: "systemctl": executable file not found in $PATH
2020-02-25T17:44:20Z E! [inputs.redis] Error in plugin: ERR wrong number of arguments for 'info' command
2020-02-25T17:44:30Z E! [inputs.systemd_units] Error in plugin: exec: "systemctl": executable file not found in $PATH
2020-02-25T17:44:30Z E! [inputs.redis] Error in plugin: ERR wrong number of arguments for 'info' command

However, if i check it in influxdb with

SELECT mean("errors") FROM "internal_gather" WHERE ("input" = 'redis' AND "host" = 'asd.asd.asd.asd') AND $timeFilter GROUP BY time($__interval) fill(null)

It will always be 0.

Same goes for prometheus client.

[ssoroka]

Thanks! We've managed to reproduce the problem locally and I've committed a fix for this (#7077).

It will be in the 1.13.4 release.

Thanks!