CCXDEV-12772: S3UploadEngine compatible with KafkaConsumer

[joselsegura]

Description

Refactor of the previous PR for adapting the engine to be used with several consumers.
Added unit tests for the IDPConsumer class, that didn´t have any

Fixes #CCXDEV'12772

Type of change

• Refactor (refactoring code, removing useless files)
• Unit tests (no changes in the code)

Testing steps

Please describe how the change was tested locally. If, for some reason, the testing was not done or not done fully, please describe what are the testing steps.

Checklist

• pre-commit run -a passes
• updated documentation wherever necessary
• added or modified tests if necessary
• updated schemas and validators in insights-data-schemas in case of input/output change

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.23%. Comparing base (c14ccbd) to head (23a3a0d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #207      +/-   ##
==========================================
+ Coverage   85.03%   88.23%   +3.19%     
==========================================
  Files          34       34              
  Lines        1116     1139      +23     
==========================================
+ Hits          949     1005      +56     
+ Misses        167      134      -33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.