auto.offset.reset:latest not honored?

[flolas]

Description:
When setting optional
auto.offset.reset:latest
enable.auto.commit:false
parameters in kafka-consumer the expected behaviour is reading from the latest offset if the consumer group has not commited yet, but the behaviour is getting all messages from the beggining Using a new group id does not solve the problem.

Affected Product Version:
siddhi-io-kafka-4.1.13

OS, DB, other environment details and versions:
Centos, Kafka 0.10.0.1

[suhothayan]

@pcnfernando can you check on this

[pcnfernando]

To keep track of which messages that have already been processed, Siddhi-io-kafka source commits the offsets of the messages that were processed. This happens either by automatic means periodically (when enable.auto.commit = true) or manually in asynchronous manner (when enable.auto.commit = false).
In other words, with the current implementation, the offset will definitely will be committed.
This is the cause for auto.offset.reset:latest not behaving as expected.

In order to resolve this, we need to improve Siddhi-IO-Kafka extension to disable committing the offset if user configures.
Will update the progress of this improvement in the issue.