feat(inputs.directory_monitor): Traverse sub-directories

[sspaink]

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #11718
resolves https://github.com/influxdata/feature-requests/issues/441

This resolves a feature request to have the input plugin directory_monitor also process sub-directories. This does change the default behavior, I suppose this could be a breaking change? Not sure if this would be a problem for users? I could add a configuration flag but the best idea I have for this is a boolean flag called "recurse=true/false", so if you think a config option should be required be happy to hear suggestions.

edit: This feature has been put behind the boolean configuration option "recursive" to make this behavior optional

[Hipska]

I think this should be a user configurable setting to not change behaviour for current users?
For example, me and possibly other users drop the finished files into a "done" directory in the same main directory. With this new feature, that would mean these files will be read again and again after updating and not changing any configurations..

Edit: Your suggested "recurse=true/false" sounds good to me. (I would have uses "recursive", not sure what is better)

PS: This feature would be useful for me as well!

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me. Thanks for looking into it @sspaink!

[Hipska]

@sspaink one question, do the options files_to_monitor and files_to_ignore now work with paths? It doesn't seem so, and now it's not possible to include/exclude certain files..

[sspaink]

Your right those options don't work with paths, but just the names of the files. So it should work for a file in a sub-directory if you just provide its name.

[Hipska]

Yeah, but the names are the same, so only based on name it isn't possible to distinguish

[sspaink]

@Hipska Good point, I've created a pull request to support paths in the ignore and monitor config options: #11784 could you take a look and maybe try it out to make sure it works for you?