v12.0.0

Breaking

• pseudonymizer change rule config field pseudonyms to mapping
• clusterer change rule config field target to source_fields
• generic_resolver change rule config field append_to_list to extend_target_list
• hyperscan_resolver change rule config field append_to_list to extend_target_list
• calculator now adds the error tag _calculator_missing_field_warning to the events tag field instead of _calculator_failure in case of missing field in events
• domain_label_extractor now writes _domain_label_extractor_missing_field_warning tag to event tags in case of missing fields
• geoip_enricher now writes _geoip_enricher_missing_field_warning tag to event tags in case of missing fields
• grokker now writes _grokker_missing_field_warning tag to event tags instead of _grokker_failure in case of missing fields
• requester now writes _requester_missing_field_warning tag to event tags instead of _requester_failure in case of missing fields
• timestamp_differ now writes _timestamp_differ_missing_field_warning tag to event tags instead of _timestamp_differ_failure in case of missing fields
• timestamper now writes _timestamper_missing_field_warning tag to event tags instead of _timestamper_failure in case of missing fields
• rename --thread_count parameter to --thread-count in http generator
• removed --report parameter and feature from http generator
• when using extend_target_list in the field managerthe ordering of the given source fields is now preserved
• logprep now exits with a negative exit code if pipeline restart fails 5 times
  • this was implemented because further restart behavior should be configured on level of a system init service or container orchestrating service like k8s
  • the restart_count parameter is configurable. If you want the old behavior back, you can set this parameter to a negative number
• logprep now exits with a exit code of 2 on configuration errors

Features

• add UCL into the quickstart setup
• add logprep http output connector
• add pseudonymization tools to logprep -> see: logprep pseudo --help
• add restart_count parameter to configuration
• add option mode to pseudonymizer processor and to pseudonymization tools to chose the AES Mode for encryption and decryption
• add retry mechanism to opensearch parallel bulk, if opensearch returns 429 rejected_execution_exception

Improvements

• remove logger from Components and Factory signatures
• align processor architecture to use methods like write_to_target, add_field_to and get_dotted_field_value when reading and writing from and to events
  • required substantial refactoring of the hyperscan_resolver, generic_resolver and template_replacer
• change pseudonymizer, pre_detector, selective_extractor processors and pipeline to handle extra_data the same way
• refactor clusterer, pre_detector and pseudonymizer processors and change rule_tree so that the processor do not require process override
  • required substantial refactoring of the clusterer
• handle missing fields in processors via _handle_missing_fields from the field_manager
• add LogprepMPQueueListener to outsource logging to a separate process
• add a single Queuehandler to root logger to ensure all logs were handled by LogprepMPQueueListener
• refactor http_generator to use a logprep http output connector
• ensure all cached_properties are populated during setup time

Details

• remove logger from signatures by @ekneg54 in #589
• add queuelistener for logging by @ekneg54 in #590
• remove pex build by @ekneg54 in #587
• Align processor architecture by @ppcad in #583
• integrate UCL into quickstart setup by @dtrai2 in #582
• remove arrow dependency by @ekneg54 in #595
• refactor http generator output to http_output connector by @ekneg54 in #591
• make extend_target_list preserve ordering by @ekneg54 in #593
• add depseudo tools by @ekneg54 in #599
• add calling setup during configuration verify by @ekneg54 in #601
• fix quickstart compose setup for WSL by @ekneg54 in #604
• revise restart behavior by @ekneg54 in #602
• add option to pseudonymizer to use different encryption modes by @ekneg54 in #600
• add retry mechanism to write_backlog by @dtrai2 in #605
• Prepare Release v12 by @dtrai2 in #607

Full Changelog: v11.3.0...v12.0.0