unexpected tcp i/o timeout

[sbhardwaj-mt]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: v1.22.0
Kafka Version: 2.0.0
Go Version: 1.11.4

Configuration

config for sarama :

kc.cfg.ConsumerConfig.Timeout = 120 
kc.cfg.HealthCheckInterval = 20
        config := sarama.NewConfig()
	config.ClientID = kc.cfg.ClientPrefix
	config.Version = sarama.V2_0_0_0
	config.Consumer.Group.Session.Timeout = kc.cfg.ConsumerConfig.Timeout * time.Second
	config.Consumer.Group.Heartbeat.Interval = kc.cfg.HealthCheckInterval * time.Second
	config.Consumer.Offsets.Initial = offset

Logs

SARAMA: 09:30:06 client/metadata fetching metadata for [****] from broker 10.0.X.XX:9092
SARAMA: 09:30:06 client/coordinator requesting coordinator for consumergroup baton_dependency_processor from 10.0.X.XX:9092
SARAMA: 09:30:06 client/coordinator coordinator for consumergroup baton_dependency_processor is #3 (10.0.Y.YY:9092)
SARAMA: 09:30:06 Connected to broker at 10.0.Y.YY:9092 (registered as #3)
SARAMA: 09:31:05 Closed connection to broker 10.0.Y.YY:9092
client.Consume returned
{"level":"error","ts":1562405465.306836,"msg":"consumer returned with error = read tcp 100.119.0.4:38146->10.0.Y.YY:9092: i/o timeout, exiting","host":"","ReqId":""}

On kafka broker logs were 
[2019-07-06 09:32:17,202] INFO [GroupCoordinator 3]: Stabilized group baton_dependency_processor generation 382 (__consumer_offsets-19) (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:32:20,252] INFO [GroupCoordinator 3]: Preparing to rebalance group baton_dependency_processor with old generation 382 (__consumer_offsets-19) (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:32:35,297] INFO [GroupCoordinator 3]: Member dependency-resolver-b08b9190-156d-4ee7-baa3-d880a37a8420 in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:32:35,298] INFO [GroupCoordinator 3]: Member dependency-resolver-fd2d6f79-016b-44a7-aaf0-2bb7d35a23aa in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:32:35,298] INFO [GroupCoordinator 3]: Member dependency-resolver-4f5fc8ef-c562-4ef0-ac08-552789433f71 in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:33:17,202] INFO [GroupCoordinator 3]: Member dependency-resolver-cce93661-9ee6-4da3-8ef0-0af6e7e53f95 in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:33:17,202] INFO [GroupCoordinator 3]: Member dependency-resolver-b85ad75f-c0c2-490e-b8a6-006f133ae226 in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-07-06 09:33:17,202] INFO [GroupCoordinator 3]: Member dependency-resolver-79da7135-c752-4c2c-91b0-a18d19575968 in group baton_dependency_processor has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)

Problem Description

can you let me know what could be the reason for this i/o timeout?, we are trying to run this group consumer from docker image, on kubernetes, and every time this happens pod restarts.
this is happening continously, and in kafka broker logs, we see Member xxx has failed, removing it from the group.

[2308524]

need change c.Net.ReadTimeout (default 30 * time.Second) to > config.Consumer.Group.Session.Timeout(120)

[ghost]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.