Documentation on how to set up docker-compose.yml?

[iand675]

Sorry for seeking tech support, but I've looked around fairly extensively on how to this image up via docker compose, and haven't had much luck. Would you be able to add documentation to the README detailing a working setup with compose?

Here's my non-working configuration at the moment.

version: "2"

services:
  kafka:
    hostname: localhost
    image: ches/kafka
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - ZOOKEEPER_IP=127.0.0.1
    links:
      - zookeeper
    ports:
      - "9092:9092"
      - "7203:7203"
  zookeeper:
    hostname: localhost
    image: zookeeper
    ports:
      - "2181:2181"

[ches]

Hi,

No problem, I've been meaning to add Docker Compose examples for ages, in fact I have some sitting around in my local working copy that I just need to polish up and push. In the meantime this ought to be pretty close:

version: '3'

services:
  kafka:
    image: ches/kafka
    depends_on:
      - zookeeper
    environment:
      # Use container hostname support of the default Compose network
      ZOOKEEPER_IP: zookeeper
    ports:
      - '9092:9092'  # Kafka broker
      - '7203:7203'  # JMX
    volumes:
      - kafka-data:/data
      - kafka-logs:/logs

  zookeeper:
    image: zookeeper:3.4
    restart: unless-stopped
    ports:
      - '2181:2181'

volumes:
  kafka-data:
  kafka-logs:

This is probably compatible with the v2 file format also, I haven't tested.

If this doesn't work, can you let me know more details about the problems you have for your use case? There can be some troublesome bits with mapped ports for Kafka using Docker on Mac or Windows, where there is technically still a VM running the Docker daemon under the covers but they try to magically abstract the mapping of ports to your localhost. It Just Works most of the time but Kafka in particular has some complications with it, related to advertised host name.

All of that basically disappears if you can do everything within the Docker network and not map ports, but that certainly limits convenience for local development.

[iand675]

That helps a bunch. Thank you!

[pallavkothari]

Hi @ches like you say I'm running into trouble with this approach using Docker on Mac. I can verify that things work within the docker network using the precanned producer/consumer examples, but I want to be able to point my java client (i.e. from the host machine) to the kafka running in the container, and that blows up. I've tried pointing my client at kafka:9092, localhost:9092, 0.0.0.0:9092, 127.0.0.1:9092....

Talking to just zookeeper seems to work (e.g. for topics). Any pointers on how to properly set this up? Thanks!

[Joaobranquinho]

Hey @ches

I've tried this docker compose and I ran into some problems.
How should I create a topic after generating the cotainers?

I tried
docker run --rm --network kafka-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
and it failed

Exception in thread "main" org.I0Itec.zkclient.exception.ZkException: Unable to connect to zookeeper:2181
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:72)
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:106)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:88)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: java.net.UnknownHostException: zookeeper: unknown error
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:70)
	... 7 more

I also tried accessing the machine using

sudo docker exec -i -t 1e86e3ce623e /bin/bash

to run

kafka@6db4098af66b:~$ kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181

and the result was

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is: 
	java.net.BindException: Address already in use

Any ideas on how to help?

[dianeswan]

Hello
I have been using ches/kafka with docker compose from October last year without problem. Recently I created some new topics but any attempt to produce/consume with those topics fails, although my program still appears to work with the old topics.

In an attempt to investigate I created a new docker-compose.yml with just zookeeper and kafka using new file space. When I run the producer/consumer test I get the same problem, here is some interaction and output;

$ docker-compose ps
Name Command State Ports

kafka_kafka_1 /start.sh Up 7203/tcp, 0.0.0.0:9092->9092/tcp
kafka_zookeeper_1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done

$ docker-compose run --rm kafka kafka-topics.sh --create --topic top1 --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done
Created topic "top1".

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181
Starting kafka_zookeeper_1 ... done
top1

$ docker-compose run --rm kafka kafka-console-producer.sh --topic top1 --broker-list kafka:9092
Starting kafka_zookeeper_1 ... done
hello
[2018-04-11 14:09:04,661] ERROR Error when sending message to topic top1 with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

If I create zookeeper and kafka from the command line it works

docker run -d --name zookeeper --network test-net zookeeper:3.4
docker run -d --name kafka --network test-net --env ZOOKEEPER_IP=zookeeper ches/kafka
docker run --rm --network test-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
docker run --rm --interactive --network test-net ches/kafka kafka-console-producer.sh --topic test --broker-list kafka:9092

in a separate terminal
docker run --rm --network test-net ches/kafka kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092

This works fine.

This is my docker-compose file

version: "2"
networks:
kafka-net:

services:
zookeeper:
image: zookeeper:3.4
hostname: zookeeper
networks:
- kafka-net
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zookeeper:2888:3888
volumes:
- /home/docker/volumes/test/zookeeper/data:/data
- /home/docker/volumes/test/zookeeper/logs:/log
restart: always

kafka:
image: ches/kafka
hostname: kafka
links:
- zookeeper:zookeeper
depends_on:
- zookeeper
networks:
- kafka-net
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
ZOOKEEPER_IP: zookeeper
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
volumes:
- /home/docker/volumes/test/kafka/data:/data
- /home/docker/volumes/test/kafka/logs:/logs
restart: always

Sorry to bother you but I have spent a few days on this and not really making any progress.

Thanks
Diane