add support for multiple output connectors

[ekneg54]

• Add support for multiple output connectors
• changes rule language of selective_extractor, pseudonymizer, pre_detector to support multiple outputs

TODO:

• validation of output_names in selective_extractor, pseudonymizer, pre_detector against defined outputs
• add storing extra_data with multiple outputs
• add documentation for callback behavior if two default outputs are defined

ToDo but NotNow:

• add optional overwrite of outputs in pseudonymizer and pre_detector -> add optional overwrite of outputs in pseudonymizer and pre_detector #321
• add optional default outputs to selective_extractor -> add default outputs to selective_extractor #322
• refactor and harmonize code of pseudonymizer, pre_detector and selective_extractor -> refactor code of pseudonymizer, pre_detector and selective_extractor #323

[codecov-commenter]

Codecov Report

Base: 91.25% // Head: 91.41% // Increases project coverage by +0.15% 🎉

Coverage data is based on head (3f5cac1) compared to base (a224468).
Patch coverage: 98.72% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #306      +/-   ##
==========================================
+ Coverage   91.25%   91.41%   +0.15%     
==========================================
  Files         120      120              
  Lines        8235     8293      +58     
==========================================
+ Hits         7515     7581      +66     
+ Misses        720      712       -8     

Impacted Files	Coverage Δ
logprep/util/auto_rule_tester/auto_rule_tester.py	86.65% <50.00%> (-0.24%)	⬇️
logprep/processor/pre_detector/processor.py	94.00% <85.71%> (ø)
logprep/abc/output.py	100.00% <100.00%> (ø)
logprep/abc/processor.py	100.00% <100.00%> (ø)
logprep/framework/pipeline.py	95.04% <100.00%> (+0.25%)	⬆️
logprep/processor/clusterer/processor.py	93.54% <100.00%> (ø)
logprep/processor/pseudonymizer/processor.py	98.19% <100.00%> (ø)
logprep/processor/selective_extractor/processor.py	100.00% <100.00%> (ø)
logprep/processor/selective_extractor/rule.py	100.00% <100.00%> (ø)
logprep/util/configuration.py	97.65% <100.00%> (+0.38%)	⬆️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ppcad]

This is a very nice feature, but would it be possible to have multiple outputs for extra data as well?
Furthermore, it is not verified if the output name for custom data is valid. Logprep will therefore run and throw a KeyError as soon as custom data is written.

[ekneg54]

Sure... I will give it a try.👍😁

[ppcad]

Thank you!

[ekneg54]

For the selective_extractor storing extra_output to multiple outputs should yet be possible by defining different rules for different outputs or topics.

we could make the same in the pseudonymizer and the pre_detector, that you are able to overwrite the the output and topic defined in the processor configuration on rule level.

this would be the most useful solution in my opinion.

[ppcad]

If the goal is to write extra data for the same event into multiple outputs, then having multiple rules with different outputs would have the problem that events need to be processed by each rule variant. This would multiply the rule count by the number of outputs, which would multiply the processing time of that processor and make it hard to maintain the rules.
Depending on the processor it might not work at all, since it could also change the event by a random value (hash/uuid) or it can't overwrite etc.

[ekneg54]

ok... good point... then I have to dig a lot deeper keeping my fingers crossed to not get to a really ugly solution :-D

[ppcad]

Thank you very much!

[herrfeder]

I tested locally with individiual selective_extractor configs on my host and the FileInputConnector.

Incompability with console_output

The output type console_output doesn't work with the new implementation at all as it will raise the following error:

2023-02-16 09:11:08,030 Pipeline ERROR   : A critical error occurred for processor SelectiveExtractor (selective_extractor) when processing an event, processing was aborted: (AttributeError: module 'sys' has no attribute 'first_topic') (59 in ~29.0 sek)

Apparently this output connector doesn't like the topic attribute but ommiting target_topic isn't possible which is also confusing when using file outputs.

Confusion about multi output selective extraction

According to my review comments I don't understand the actual intended test behaviour as I would assume with individual selective_extractor the affected outputs should be different.

Suggestion for optimized dealing with multiple outputs through consecutive pipeline processors

And I guess it touches somehow @ppcad comments I don't see the possibility to create individual streams when using multi-outputs which leads to the described behaviour that for each additional output all rules will be executed again.
An easy but not perfect possibility to deal with it in my opinion would be to introduce (just like in Logstash) Meta-Fields that can be filtered. When addressing multiple or even new outputs in a multi-output aware processor a meta-field could be added {"output":"output_stream_123"} which can be filtered in the following processors in the pipeline. This would still touch the filter of each following processor but doesn't execute the full processor behaviour.

[ppcad]

I tested locally with individiual selective_extractor configs on my host and the FileInputConnector.

Incompability with console_output

The output type console_output doesn't work with the new implementation at all as it will raise the following error:

2023-02-16 09:11:08,030 Pipeline ERROR   : A critical error occurred for processor SelectiveExtractor (selective_extractor) when processing an event, processing was aborted: (AttributeError: module 'sys' has no attribute 'first_topic') (59 in ~29.0 sek)

Apparently this output connector doesn't like the topic attribute but ommiting target_topic isn't possible which is also confusing when using file outputs.

Yes, I have also noticed that. It works if target_topic is set to stdout.

Confusion about multi output selective extraction

According to my review comments I don't understand the actual intended test behaviour as I would assume with individual selective_extractor the affected outputs should be different.

Suggestion for optimized dealing with multiple outputs through consecutive pipeline processors

And I guess it touches somehow @ppcad comments I don't see the possibility to create individual streams when using multi-outputs which leads to the described behaviour that for each additional output all rules will be executed again. An easy but not perfect possibility to deal with it in my opinion would be to introduce (just like in Logstash) Meta-Fields that can be filtered. When addressing multiple or even new outputs in a multi-output aware processor a meta-field could be added {"output":"output_stream_123"} which can be filtered in the following processors in the pipeline. This would still touch the filter of each following processor but doesn't execute the full processor behaviour.

Extra data doesn't get passed to the next processor, so it needs to be generated again for each output if the goal is to send the same extra data to different outputs. To generate it the whole event needs to be processed again.

[ekneg54]

It is a draft... I am not ready

[ekneg54]

@ppcad @herrfeder ... I'm ready now... please have a look

[herrfeder]

As we finished the review with at least 6 eyes gracefully, I can finish this review with minor request changes.

[ppcad]

Thank you for the changes!

[ekneg54]

thank you for your approve. I will merge this, if we are ready. Because it will break our configuration.