Proper shutdown of kafka consumer impl and fix test

[taylanisikdemir]

What changed?
While trying to run unit tests in github actions I noticed kafka consumer tests are consistently failing with below error:

panic: Log in goroutine after TestNewConsumer has completed: 2024-03-04T23:30:17.785Z ERROR Error from consumer: %v {"error": "kafka: client has run out of available brokers to talk to: dial tcp 127.0.0.1:46365: connect: connection refused", "logging-call-at": "consumer_impl.go:120"}

The log comes from a goroutine which wasn't waited on during consumer.Stop(). After debugging I found that TestNewConsumer was missing a mock response for FindCoordinatorRequest and sarama consumer library was stuck and never initiated consume loop. It was irresponsive to context cancel and the consumerImpl's goroutine was hanging. Fixed that mock setup and also added WaitGroup so that consumerImpl.Stop() actually waits for its dependencies to stop.

Why?
Fix broken test setup and also ensure Stop doesn't return before its dependencies are stopped/cleaned up.

How did you test it?
unit test

[coveralls]

Pull Request Test Coverage Report for Build 018e0f7c-64bc-4026-bdf0-6573ff2528c4

Details

• 9 of 9 (100.0%) changed or added relevant lines in 1 file are covered.
• 69 unchanged lines in 13 files lost coverage.
• Overall coverage increased (+0.04%) to 63.137%

---

Files with Coverage Reduction	New Missed Lines	%
service/history/execution/mutable_state_task_refresher.go	1	66.77%
common/persistence/executionManager.go	2	78.59%
common/persistence/statsComputer.go	2	95.71%
service/history/task/transfer_active_task_executor.go	2	72.38%
service/matching/matcher.go	2	90.72%
service/history/queue/timer_queue_processor_base.go	3	77.82%
common/persistence/nosql/nosqlplugin/cassandra/tasks.go	4	76.09%
service/matching/taskListManager.go	4	79.7%
common/persistence/nosql/nosql_task_store.go	5	61.95%
common/persistence/nosql/nosql_execution_store.go	9	62.72%

Totals
Change from base Build 018e0f0f-35f3-44b0-8512-458527ee339a:	0.04%
Covered Lines:	92603
Relevant Lines:	146671

---

💛 - Coveralls