Kafka unable to connect

[kojuka]

@cjimti Saw your comments on kafka but thought it'd be best to start a new thread.

For some reason I can't seem to connect to kafka using kubefwd. Did you use helm? If so which chart. I'm using https://github.com/helm/charts/tree/master/incubator/kafka

➜  helm git:(master) ✗ sudo kubefwd svc -l "app in (kafka)"
Password:
2019/07/02 00:24:54  _          _           __             _
2019/07/02 00:24:54 | | ___   _| |__   ___ / _|_      ____| |
2019/07/02 00:24:54 | |/ / | | | '_ \ / _ \ |_\ \ /\ / / _  |
2019/07/02 00:24:54 |   <| |_| | |_) |  __/  _|\ V  V / (_| |
2019/07/02 00:24:54 |_|\_\\__,_|_.__/ \___|_|   \_/\_/ \__,_|
2019/07/02 00:24:54
2019/07/02 00:24:54 Version 1.8.0
2019/07/02 00:24:54 https://github.com/txn2/kubefwd
2019/07/02 00:24:54
2019/07/02 00:24:54 Press [Ctrl-C] to stop forwarding.
2019/07/02 00:24:54 'cat /etc/hosts' to see all host entries.
2019/07/02 00:24:54 Loaded hosts file /etc/hosts
2019/07/02 00:24:54 Hostfile management: Original hosts backup already exists at /Users/ken/hosts.original
2019/07/02 00:24:54 Forwarding: kafka:9092 to pod kafka-0:9092
2019/07/02 00:24:54 Forwarding: kafka-headless:9092 to pod kafka-0:9092

When i try to connect using this script in nodejs, it fails.

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka:9092']
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group1' })

const run = async () => {
  // Producing
  await producer.connect()

    await producer.send({
      topic: 'test-topic',
      messages: [
        { value: `Hello KafkaJS user!` },
      ],
    })

  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      })
    },
  })
}

run().catch(console.error)

It fails with:

{"level":"ERROR","timestamp":"2019-07-02T06:28:24.746Z","logger":"kafkajs","message":"[Producer] Connection error: getaddrinfo ENOTFOUND kafka-1.kafka-headless.default kafka-1.kafka-headless.default:9092","retryCount":0,"retryTime":247}
{ KafkaJSNumberOfRetriesExceeded
  Caused by: KafkaJSConnectionError: Connection error: getaddrinfo ENOTFOUND kafka-1.kafka-headless.default kafka-1.kafka-headless.default:9092
    at Socket.onError

Yet if I run this same script using telepresence, then it works as expected. Is there anything special I need to do in order to connect using kubefwd?

[cjimti]

An upgrade to v1.9.0 should resolve this. A headless service name will forward to the first Pod, however other pods backing the service will be available at their Pod names. Your Kafka client should now be able to find kafka-1, etc..