Revert "consumer-group: do not cancel sessions context during rebalance"

[niamster]

Reverts #2193

@dnwe @pavius I must revert this PR.
I did more testing and for some reason consumer is regularly stuck after restart on initial join request.
It seems like the communication with the broker is somehow broken.

I think it's also worth reverting #2110 (to avoid confusion).

I'm not entirely sure what is the root cause of the problem, I think there's a race condition in the new session creation and rebalance event.

With current main brunch I get following stacktraces while the consumer is stuck (eventually it unblocks but it takes ~30s - default network timeout):

6 @ 0x456692 0x466ad2 0x1b6d3b3 0x1b69f1e 0x1b90a45 0x1b8eb4b 0x1bf33c7 0x488321
#       0x1b6d3b2       github.com/Shopify/sarama.(*Broker).sendAndReceive+0x212                /go/src/github.com/DataDog/dd-go/sarama/broker.go:975
#       0x1b69f1d       github.com/Shopify/sarama.(*Broker).Fetch+0x7d                          /go/src/github.com/DataDog/dd-go/sarama/broker.go:455
#       0x1b90a44       github.com/Shopify/sarama.(*brokerConsumer).fetchNewMessages+0x6c4      /go/src/github.com/DataDog/dd-go/sarama/consumer.go:1076
#       0x1b8eb4a       github.com/Shopify/sarama.(*brokerConsumer).subscriptionConsumer+0x16a  /go/src/github.com/DataDog/dd-go/sarama/consumer.go:929
#       0x1bf33c6       github.com/Shopify/sarama.withRecover+0x46                              /go/src/github.com/DataDog/dd-go/sarama/utils.go:43

....


1 @ 0x456692 0x466ad2 0x1b6d3b3 0x1b6a23e 0x1b93f55 0x1b92825 0x1b92008 0x1b9339a 0x1b9173e 0x20d4435 0x488321
#       0x1b6d3b2       github.com/Shopify/sarama.(*Broker).sendAndReceive+0x212                                        /go/src/github.com/DataDog/dd-go/sarama/broker.go:975
#       0x1b6a23d       github.com/Shopify/sarama.(*Broker).JoinGroup+0x7d                                              /go/src/github.com/DataDog/dd-go/sarama/broker.go:492
#       0x1b93f54       github.com/Shopify/sarama.(*consumerGroup).joinGroupRequest+0x434                               /go/src/github.com/DataDog/dd-go/sarama/consumer_group.go:412
#       0x1b92824       github.com/Shopify/sarama.(*consumerGroup).newSession+0x6e4                                     /go/src/github.com/DataDog/dd-go/sarama/consumer_group.go:274
#       0x1b92007       github.com/Shopify/sarama.(*consumerGroup).retryNewSession+0x307                                /go/src/github.com/DataDog/dd-go/sarama/consumer_group.go:245
#       0x1b93399       github.com/Shopify/sarama.(*consumerGroup).newSession+0x1259                                    /go/src/github.com/DataDog/dd-go/sarama/consumer_group.go:360
#       0x1b9173d       github.com/Shopify/sarama.(*consumerGroup).Consume+0x2dd                                        /go/src/github.com/DataDog/dd-go/sarama/consumer_group.go:192

Not very useful logs, but better than nothing:

[Sarama] 2022/03/31 02:29:23 Initializing new client
[Sarama] 2022/03/31 02:29:23 Successfully initialized new client
[Sarama] 2022/03/31 02:29:23 client/metadata fetching metadata for [my-topic] from broker my-kafka:9092
[Sarama] 2022/03/31 02:29:23 client/coordinator requesting coordinator for consumergroup my-consumer-group from my-kafka:9092
[Sarama] 2022/03/31 02:29:23 client/coordinator coordinator for consumergroup my-consumer-group is #30011 (1.1.1.1:9092)
[Sarama] 2022/03/31 02:29:23 Connected to broker at 1.1.1.1:9092 (registered as #30011)
[Sarama] 2022/03/31 02:29:53 Closed connection to broker 1.1.1.1:9092
[Sarama] 2022/03/31 02:29:53 client/metadata fetching metadata for all topics from broker 1.1.1.2:9092
[Sarama] 2022/03/31 02:29:53 Connected to broker at 1.1.1.2:9092 (unregistered)

[dnwe]

Thanks — I wonder if this may explain the flakiness of some of the functional tests recently as well

[niamster]

Thanks — I wonder if this may explain the flakiness of some of the functional tests recently as well

Thanks @dnwe ! Yes, it's possible.