problem producing with new topic from docker-compose ches/kafka

[dianeswan]

Hello
I have been using ches/kafka with docker compose from October last year without problem. Recently I created some new topics but any attempt to produce/consume with those topics fails, although my program still appears to work with the old topics.

In an attempt to investigate I created a new docker-compose.yml with just zookeeper and kafka using new file space. When I run the producer/consumer test I get the same problem, here is some interaction and output;
$ docker-compose ps
Name Command State Ports

kafka_kafka_1 /start.sh Up 7203/tcp, 0.0.0.0:9092->9092/tcp
kafka_zookeeper_1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done

$ docker-compose run --rm kafka kafka-topics.sh --create --topic top1 --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done
Created topic "top1".

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done
top1

$ docker-compose run --rm kafka kafka-console-producer.sh --topic top1 --broker-list kafka:9092
Starting kafka_zookeeper_1 ... done
hello
[2018-04-11 14:09:04,661] ERROR Error when sending message to topic top1 with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

If I create zookeeper and kafka from the command line it works

docker run -d --name zookeeper --network test-net zookeeper:3.4
docker run -d --name kafka --network test-net --env ZOOKEEPER_IP=zookeeper ches/kafka
docker run --rm --network test-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
docker run --rm --interactive --network test-net ches/kafka kafka-console-producer.sh --topic test --broker-list kafka:9092

    in a separate terminal
    docker run --rm --network test-net ches/kafka kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092

This works fine.

This is my docker-compose file

version: "2"
networks:
kafka-net:

services:
zookeeper:
image: zookeeper:3.4
hostname: zookeeper
networks:

• kafka-net
  ports:
• 2181:2181
  environment:
  ZOO_MY_ID: 1
  ZOO_PORT: 2181
  ZOO_SERVERS: server.1=zookeeper:2888:3888
  volumes:
• /home/docker/volumes/test/zookeeper/data:/data
• /home/docker/volumes/test/zookeeper/logs:/log
  restart: always

kafka:
image: ches/kafka
hostname: kafka
links:

• zookeeper:zookeeper
  depends_on:
• zookeeper
  networks:
• kafka-net
  ports:
• 9092:9092
  environment:
  KAFKA_ADVERTISED_HOST_NAME: kafka
  ZOOKEEPER_IP: zookeeper
  KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
  volumes:
• /home/docker/volumes/test/kafka/data:/data
• /home/docker/volumes/test/kafka/logs:/logs
  restart: always

Sorry to bother you but I have spent a few days on this and not really making any progress.

Thanks
Diane

[dianeswan]

I should have added

$ docker --version
Docker version 1.12.3, build 6b644ec
$ docker-compose --version
docker-compose version 1.16.0-rc1, build 9bea70a

OS Ubuntu 14.04