Merge PR #11 from beta branch to main #24
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n and HDFS storage (WIP). Skipping implementing idempotent consumer pattern to the Kafka consumer groups.
…-serialization in Kafka consumer processing.
…n of records to specific sized files.
…ad of definite values as definite requires continuous use of flush() after appending records to AVRO.
…n files with flush() after every record append. Improved test cases.
…plementation. (WIP)
…erformance. Cleaned up namings in in DurationStatistics. Debugging faulty consumed kafka topic partition ordering.
… gets its own exclusive AVRO-file queueNamePrefix which is used to generate the AVRO-files where the records of the partition are stored.
…ded test configuration files and their usage. Fixes to Config.java resource locations.
…tion.properties. Updated example configuration file.
…dition to ingestion tests.
Implement idempotent Kafka consumer
Fix to rpm packaging configuration
* Fix for issue #32 by enabling automatic TGT renewal. * Replaced hadoop Configuration class usage with the child class HdfsConfiguration. * Added simplified handling for consumed kafka records with null content. * Added additional null content records to MockKafkaConsumerFactory for tests. * Added exception handling for non-parseable records. Modified test data and tests accordingly. * Moved NULL record handling from KafkaReader to DatabaseOutput. Changed KafkaConsumerTest accordingly. * Added configuration flags for skipping empty and non RFC5424 records. Improved logging to give metadata for records causing the exceptions. * Added configuration flags for automatic kerberos TGT renewal. * Added separate record processing tests. Added cleanup of temporary files that weren't written to HDFS because of exception during processing. * Added checkTGTAndReloginFromKeytab() functionality. --------- Co-authored-by: Mikko Kortelainen <kortemik@users.noreply.github.com>
|
see #35 (review) |
* Added encryption flags for configuration * Added missing flags to example/test configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is still work in progress and should not be reviewed at this point
Original comments and PR #11