v10.0.0

v10.0.0

Breaking

• reimplement the logprep CLI, see logprep --help for more information.
• remove feature to reload configuration by sending signal SIGUSR1
• remove feature to validate rules because it is already included in logprep test config

Features

• add a number_of_successful_writes metric to the s3 connector, which counts how many events were successfully written to s3
• make the s3 connector work with the new _write_backlog method introduced by the confluent_kafka commit bugfix in v9.0.0
• add option to Opensearch Output Connector to use parallel bulk implementation (default is True)
• add feature to logprep to load config from multiple sources (files or uris)
• add feature to logprep to print the resulting configruation with logprep print json|yaml <Path to config> in json or yaml
• add an event generator that can send records to Kafka using data from a file or from Kafka
• add an event generator that can send records to a HTTP endpoint using data from local dataset

Improvements

• a do nothing option do dummy output to ensure dummy does not fill memory
• make the s3 connector raise FatalOutputError instead of warnings
• make the s3 connector blocking by removing threading
• revert the change from v9.0.0 to always check the existence of a field for negated key-value based lucene filter expressions
• make store_custom in s3, opensearch and elasticsearch connector not call batch_finished_callback to prevent data loss that could be caused by partially processed events
• remove the schema_and_rule_checker module
• rewrite Logprep Configuration object see documentation for more details
• rewrite Runner
• delete MultiProcessingPipeline class to simplify multiprocesing
• add FDA to the quickstart setup
• bump versions for fastapi and aiohttp to address CVEs

Bugfix

• make the s3 connector actually use the max_retries parameter
• fixed a bug which leads to a FatalOutputError on handling CriticalInputError in pipeline

Details

• add architecture overview by @djkhl in #478
• make parallel_bulk optional by @ekneg54 in #504
• remove tox setup by @ekneg54 in #500
• Adapt s3 connector for kafka fix by @ppcad in #499
• add do nothing option to dummy output by @ekneg54 in #503
• Revise CLI by @dtrai2 in #513
• Fix handling of CriticalInputError exceptions by @clumsy9 in #514
• Revert not exists changes from #480, but keep refactorings in filter_expression by @ppcad in #515
• Set no offsets on store custom by @ppcad in #518
• config from multiple sources by @ekneg54 in #507
• Add load-tester by @ppcad in #487
• Add http connector to quickstart setup by @djkhl in #509
• Add FDA to quickstart setup by @dtrai2 in #477
• Prepare release 10.0.0 by @ekneg54 in #521

New Contributors

• @djkhl made their first contribution in #478 🎆

Full Changelog: v9.0.3...v10.0.0