CCXDEV-12772: S3UploadEngine compatible with KafkaConsumer

[joselsegura]

Description

Modifying the way of interaction between consumers and S3Uploadengine in order to make it compatible with any Consumer that wants to use it.

In addition, adding compatibility for both Internal Data Pipeline Consumer and KafkaConsumer.

Fixes #CCXDEV-12772

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Refactor (refactoring code, removing useless files)

Testing steps

Tested only with unit tests.

Checklist

• pre-commit run -a passes
• updated documentation wherever necessary
• added or modified tests if necessary
• updated schemas and validators in insights-data-schemas in case of input/output change

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 61.76471% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 84.32%. Comparing base (3c46b28) to head (78a4d88).

Files	Patch %	Lines
ccx_messaging/consumers/idp_kafka_consumer.py	0.00%	13 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #205      +/-   ##
==========================================
- Coverage   85.03%   84.32%   -0.72%     
==========================================
  Files          34       34              
  Lines        1116     1129      +13     
==========================================
+ Hits          949      952       +3     
- Misses        167      177      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.