Enable debug logging on a per-plugin basis

[lhanson]

Feature Request

I'd like to be able to see debug logging for specific plugins I'm using rather than simply for the top-level agent.

Proposal:

Allow the [agent] level "debug" configuration value to be applied on a per-plugin basis.

Current behavior:

Enabling debug = true at the [agent] level only emits high-level debug output.

Desired behavior:

Allow debug = true to be set for individual plugins I've defined, which then log specific actions they are or are not taking as they run.

Use case:

I can't figure out why I can't seem to use [[processors.rename.replace]] to replace fields based on a specific tag; all subsequent series are apparently affected regardless of tagpass values I'm trying to use. It would be helpful to have verbose logging to ascertain how inputs are being processed by my current configuration so I can determine what I'm doing wrong.

[glinton]

Enabling debug = true at the [agent] level only emits high-level debug output.

Setting debug = true in the `agent section actually turns on debugging for everything. The problem is that many plugins don't implement any debug logging.

Since this sounds like a config issue, maybe posting your config and asking over at the InfluxData Community site will yield some results.

[lhanson]

@glinton That's helpful, thanks. My immediate issue has been mostly resolved through a combination of trial-and-error and code spelunking (you're correct, it was a config issue; the TOML bracket syntax caught me out and I was using [[processors.rename.tagpass]] instead of [processors.rename.tagpass]).

I think that the lack of debugging output I was hoping to see (but didn't) is, as you say, the result of plugins not printing any debug information. Would PRs adding additional debugging output to plugins be welcome if I find them useful to fix any future issues?

Thinking more broadly, would it be worth adding a note in CONTRIBUTING.md explaining how the debug setting applies to plugins and perhaps encouraging authors to add relevant messages?

[glinton]

Would PRs adding additional debugging output to plugins be welcome if I find them useful to fix any future issues?

It would be helpful to add the least amount of debug logs required to debug an issue, but I'm not sure what our "official" stance is, @danielnelson can say better when he's back in.

[danielnelson]

We would have to evaluate logging on a case by case basis, its one of those somewhat subjective things where you don't want to little or too much. Logging every modification by the plugin would be too much though. It might also be helpful to use the printer processor to help with debugging the metric filtering.

On the particular error you ran into, it needs to be fixed in the configuration loading, as it's not a plugin level error, and it is #6474.

Back on the subject of per plugin logging level, I would like to add this. The way I'm thinking to do it would be to deprecate the debug and quiet options from the agent and replace them with a loglevel = "info" option to the agent and allow it to be overriden per plugin, similar to how the interval option works:

[agent]
  # default log level
  loglevel = "info"

[[inputs.cpu]]
  # overrides the log level for this plugin
  loglevel = "debug"

[AlexHeylin]

danielnelson - I agree, that's a good way to go. On Windows debug should get sent to eventlog as info, because currently if you're using eventlog debug level messages are dropped.

I agree with lhanson that guidelines in CONTRIBUTING.md would be helpful, because right now using inputs.win_perf_counters and inputs.exec they're both completely silent and all I see is outputs.influxdb writing data. The problem I have is missing data, without any debug from the inputs I can't find out why sometimes it goes missing on every 7th poll (70 seconds). The output appears to write all the data it has.

[wfjm]

I'd like to second the original feature request.
debug = true seems to show only [agent] level information.
More tools to debug plugin level issues would be helpful, see post for another example.

[n1nj4888]

Hi @danielnelson. Is this per-plugin debug logging (rather than global) still planned/on the roadmap?

This would be really useful in cases where say there is a ping plugin and snmp plugin for a specific host - In normal circumstances, It would only make sense to enable the debug on the ping plugin to indicate in the logs that the host was down and suppress all the (potentially extensive) errors from the snmp plugin that would inevitably occur if the host was down anyway.

Thanks!