Module should allow to override the pipeline id

[ph]

Modules accepts an input option in the YAML configuration, this allows users to overrides the input options directly (close_inactive, clean_inactive, etc...). One option you cannot override is the pipeline option.

beats/filebeat/fileset/fileset.go

Lines 331 to 332 in 26cef51

	// force our pipeline ID
	err = cfg.SetString("pipeline", -1, fs.pipelineID)

Lets say that you have fixed an issue with a pipeline or you want to do more processing on the collected data, you cannot override the pipeline. You either have to update the ES pipeline or create a custom input block.

[hjmwijnen]

Hi team, as I already mentioned here (https://discuss.elastic.co/t/filebeat-cannot-change-pipeline-in-nginx-module/219360), we have some customized NGINX access logs, which cannot be processed via the default filebeat nginx module pipeline. Instead of changing the current pipeline I would like to configure another pipeline on the module (and not on beat level, since it also has other modules running). I think that also prevents issues with upgrading the Beat in the future. Therefore I'd like to ask you to consider this request.

Thanks.

[seweissman]

Just wanted to add here that I was hoping to be able to do this to change the nginx error logs pipeline so I could customize the date processor to add a timezone. It took me a long time to find this issue and realize that this can't be done currently.

[ycombinator]

@hjmwijnen @seweissman Thanks for your comments. I've put up a PR now to add this enhancement: #16561. It is currently targeting the 7.7.0 release.