Implement Kafka Consumer group and record processing #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Seems like the history is not linear, can you rebase and solve the merge conflicts? Also pom.xml contains internal references |
…n and HDFS storage (WIP). Skipping implementing idempotent consumer pattern to the Kafka consumer groups.
…-serialization in Kafka consumer processing.
…n of records to specific sized files.
…ad of definite values as definite requires continuous use of flush() after appending records to AVRO.
…n files with flush() after every record append. Improved test cases.
…plementation. (WIP)
…erformance. Cleaned up namings in in DurationStatistics. Debugging faulty consumed kafka topic partition ordering.
… gets its own exclusive AVRO-file queueNamePrefix which is used to generate the AVRO-files where the records of the partition are stored.
…ation files to test folder.
|
Overall seems to be missing licenses on the files |
|
Closed as newest PR contain the same commits. |
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial PR to partially fix issue #2 and #3.
Still HDFS access/write/pruning and query handling left to implement and test for issue #2.