Policy evaluation can take too long and cause the consumer thread to be kicked out of the consumer group

[nscuro]

We're seeing Kafka Streams consumer threads occasionally getting kicked out of the consumer group due to policy evaluation taking too long:

consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll with max.poll.records.

Followed by:

Detected that the thread is being fenced. This implies that this thread missed a rebalance and dropped out of the consumer group. Will close out all assigned tasks and rejoin the consumer group.

Prior to this happening, we can see the average thread process latency skyrocketing to 15-30s, which is entirely too much. It should be <= 10ms.

[mehab]

Might be able to address this with #405

[nscuro]

Related PRs already merged:

• Optimize initialization of policy engine hyades-apiserver#145
• Feature/improve policy evaluation hyades-apiserver#147

[mehab]

@nscuro will be creating a separate issue for policy evaluation since it is still a time consuming task