Ability to use path.logs under each pipeline

[jguay]

• Version: Logstash 6.6.0
• Current feature:
  In logstash.yml it is possible to specify path.logs so logs related to all pipelines will be directed to the same log folder
• Requested feature
  Ability to have a different log file for each pipeline
  For example if pipeline.yml would support something like this :

- pipeline.id: pipeline1 
path.config: "/etc/logstash/conf.d/pipeline1.conf" 
path.logs: "/var/log/logstash/logstash-pipeline1.log" 
- pipeline.id: pipeline2 
path.config: "/etc/logstash/conf.d/pipeline2.conf" 
path.logs: "/var/log/logstash/logstash-pipeline2.log" 

[cskowronnek]

+1
We need that, too!

[treknado]

+1

[jsvd]

To all voting +1 here, there's a PR in progress to write one log file per pipeline. Would this solve the needs y'all have, or is a configurable path.log necessary? If so, can you explain your use case?

If all are good with having separate logs per pipeline in the same path.logs, then I'll rename this issue accordingly.

[cskowronnek]

IMHO the ideal behavior should be like this:

path.logs set in logstash.yml: all logs go into one logfile
path.logs.pipelines = true : all logs go to path.logs but with a seperate filename (=pipline.id)

path.logs set in pipelines.yml: specific logfile in a configurable path

We are running logstash as a shared environment for different departments. Each one has an own folder with the pipeline config. We would like to have the logs for each individual pipeline under the user's home directory.

[andsel]

@cskowronnek path.logs already define a path to logs directory where log files is created. Instead of defining path.logs in two separate locations (logstash.yml and pipelines.yml) which could drive to confusion, the proposed PR (#11108) uses the flag pipeline.separate_logs to enable and disable the creation of separate log files per pipeline. The name of single pipeline log file is already named by pipeline's id. So your use case can be easily solved by configuring the path.logs in each pipeline config and then automatically have the the creation of separate log per pipeline files.

[jsvd]

#11108 as been merged and will ship in 7.5.0 (disabled by default, enabled by pipeline.separate_logs: true.
Awesome stuff @andsel, thanks for putting this together :)

[cskowronnek]

Hi,
I just installed 7.5.2.
I set "pipeline.separate_logs: true in logstash.yml
I then tried to set a path.logs in pipeline.yml and I get the error:

[2020-01-28T15:30:38,433][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>ArgumentError, :message=>"Only pipeline related settings are expected. Received \"path.logs\". Allowed settings: [\"config.debug\", \"config.support_escapes\", \"config.reload.automatic\", \"config.reload.interval\", \"config.string\", \"dead_letter_queue.enable\", \"dead_letter_queue.max_bytes\", \"metric.collect\", \"pipeline.java_execution\", \"pipeline.plugin_classloaders\", \"path.config\", \"path.dead_letter_queue\", \"path.queue\", \"pipeline.batch.delay\", \"pipeline.batch.size\", \"pipeline.id\", \"pipeline.reloadable\", \"pipeline.system\", \"pipeline.workers\", \"queue.checkpoint.acks\", \"queue.checkpoint.interval\", \"queue.checkpoint.writes\", \"queue.checkpoint.retry\", \"queue.drain\", \"queue.max_bytes\", \"queue.max_events\", \"queue.page_capacity\", \"queue.type\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:128:in `block in merge_pipeline_settings'", "org/jruby/RubyHash.java:1417:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:126:in `merge_pipeline_settings'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:20:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:18:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2579:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:149:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:114:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:103:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:369:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

I was under the impression I could set individual log paths for each pipeline. Now I've got individual files in /var/log/logstash/

[andsel]

@cskowronnek could you share here how you valued the path.logs into the logstash.yml please?
FYI path.logs has the meaning "path where to store the log files". pipeline.separate_logs: true simply make to create a different log file (named with the name/id of) for each pipeline and place them in the path path.logs

[cskowronnek]

@andsel
logstash.yml: path.logs: /var/log/logstash
pipelines.yml:

- pipeline.id: qa
  path.config: "/home/chroot/home/qa/logstash/*.conf"
  queue.type: persisted
  path.logs: "/home/chroot/home/qa/logs"

[andsel]

@cskowronnek the config path.logs put in pipelines.yml is not true, the ``path.logsshould be specified only inlogstash.yml` with the semantic described the comment above

[cskowronnek]

@andsel So the short answer is: I can't specify different logfile targets in pipelines? Is there any chance that could be implemented? After this request here and #11076 I was hoping I could set different log paths.
I would assume that the setting in logstash.yml is leading and default and whatever you specify in pipelines.yml would overwrite the value for that pipeline. Only if pipelines.separate_logs is set to true.

[andsel]

Yes you are right, at the moment you can only specify where you want to put all your log file, but you can' define a specific path for each one.We thought that using the pipeline name as log file name was the most straightforward and simply settings for users. The discussion to provide a separate file definition for each pipeline could be open if it gains interest.

[Grunticus03]

I'd be interested in being able to set per-pipeline log levels. Being able to set a troublesome pipeline to debug without having all the other logs flooded with unnecessary debug information would be great.

[andsel]

@wwalker0307 your suggestion is tracked in #12473