Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid port binding issues running create topic shell script within the container #11

Closed
wants to merge 2 commits into from
Closed

Conversation

robwdux
Copy link

@robwdux robwdux commented Oct 15, 2015

No description provided.

rob dux added 2 commits October 15, 2015 12:53
using KAFKA_JMX_PORT to prevent conflict

precludes running scripts that call kafka-run-class.sh within the
running container resulting in an exception (process spawned from
kafka-start-server.sh already bound to the port defined)

Error: Exception thrown by the agent : java.rmi.server.ExportException:
Port already in use: 7203; nested exception is:
  java.net.BindException: Address already in use
default undefined variables to an empty string for expected beahvior
when evaluating string length
@ches
Copy link
Owner

ches commented Oct 16, 2015

Getting all the JMX settings is rather fiddly to get right. You're going to have to give an appropriately thorough description in your pull request about how this works, what problem this solves for you, etc.

@robwdux
Copy link
Author

robwdux commented Oct 16, 2015

ches - I was looking for a working kafka container that could join an ensemble of zookeeper nodes and I didn't have any luck until trying yours. however attempting to verify beyond kafka registering with zk that it was indeed functional, I came up short attempting to create a topic when following the readme. I've corrected the issues encountered and tested them by creating an ensemble with 3 zk nodes and 3 kafka instances.

following the quick start, single kafka container talking with a zookeeper ensemble - I receive an exception attempting to create a topic within the instance to the zk ensemble (those doing the quickstart with a single node to try it out will likely encounter this issue).

Error: Exception thrown by the agent : java.rmi.server.ExportException:
Port already in use: 7203; nested exception is:
java.net.BindException: Address already in use

kafka-run-class.sh is aware of JMX_PORT the server script calls this script for which it then binds to 7203 as exported in the dockerfile to the environment. attempting to verify kafka is functional by running kafka-topics.sh to create a topic I receive the exception because this script calls kafka-run-class.sh which then attempts to spawn a process - instructed to bind to the same port declared by JMX_PORT and fails. Essentially, not exporting that variable solves the problem. My commits correct that problem as well as another. essentially, the KAFKA_JMX_OPTIONS don't get exported unless a value is passed during runtime given that it is being evaluated for length but isn't defined in this case.

@ches
Copy link
Owner

ches commented Oct 20, 2015

Thanks for the further explanation. I think I follow what you're getting at, but I'll need to make a few minutes to try to reproduce—can you give me an exact sequence of commands to reproduce your issue?

I believe my fundamental question is, how are you running two processes in a container such that they conflict on binding the port? I assume that's the only way this would happen, and it's not really intended usage, but I'll try to see if there's a reasonable way to accommodate.

following the quick start, single kafka container talking with a zookeeper ensemble - I receive an exception attempting to create a topic within the instance to the zk ensemble

Can you elaborate on what you mean by "within the instance"? I assume this has to do with how you're ending up with two processes competing for the port.

essentially, the KAFKA_JMX_OPTIONS don't get exported unless a value is passed during runtime given that it is being evaluated for length but isn't defined in this case.

I don't follow you. If the var is unset, then if [ -z "$KAFKA_JMX_OPTS" ]; then ... and if [[ -z ${KAFKA_JMX_OPTS:-} ]]; then ... are essentially equivalent:

$ unset BOZO_VAR
$ if [ -z "$BOZO_VAR" ]; then echo empty; fi
empty
$ if [[ -z ${BOZO_VAR:-} ]]; then echo empty; fi
empty

KAFKA_JMX_OPTS does get exported inside the conditional in start.sh as long as the user passes no explicit value for it at runtime, I've used this recently to connect VisualVM from outside containers.

@eliaslevy
Copy link

I believe my fundamental question is, how are you running two processes in a container such that they conflict on binding the port?

He is executing a single process in the container (Kafka), but is attempting to execute kafka-topic.sh within the container to verify the cluster state. That will result in the error he got if JMX_PORT is in the environment, as kafka-run-class.sh enables JMX support indiscriminetely and using the same JMX_PORT as the server within the server container will lead to a port conflict.

I run into the same issue on my own containers.

@greatpsi
Copy link

The changes outlined at the top of the post resolved the issue with the port binding. Further description of the build process follows:

Containers were built using docker-compose, where container_name elements were added for both zookeeper and kafka. Also made the following (partial) change to expose ZOOKEEPER_IP for the start.sh script:

kafka:
build: .
image: ches/kafka
container_name: kafka
links:

  • zookeeper
    environment:
  • ZOOKEEPER_IP=zookeeper

After docker-composer up -d and connecting the the kafka container using docker exec, the port binding problem occurred. Modifying the two files as indicated above allowed topics to be created, published and consumed from within the kafka container.

@markusthoemmes
Copy link

markusthoemmes commented Oct 13, 2016

Running into the same problem. Very annoying. As a workaround:

docker exec kafka bash -c "unset JMX_PORT; your_scriptcall"

Is there still something blocking the merge of this? I'ld be happy to rebase the fix to master to make it mergeable again if you agree on the change @ches .

@bluepuma77
Copy link

kafka-topics.sh doesn't run out of the box.

# docker run \
  --name=kafka1 \
  -p 9092:9092 -p 7203:7203 \
  -e ZOOKEEPER_CONNECTION_STRING=server1.server.de:2181 \
  -e KAFKA_BROKER_ID=0 \
  -e KAFKA_DEFAULT_REPLICATION_FACTOR=2 \
  -e KAFKA_AUTO_CREATE_TOPICS_ENABLE=true \
  -e KAFKA_ADVERTISED_HOST_NAME=server1.server.de \
  -e KAFKA_ADVERTISED_PORT=9092 \
  -d ches/kafka

# docker exec -it kafka1 bash -c "/kafka/bin/kafka-topics.sh --zookeeper server1.server.de:2181 --list"
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
	java.net.BindException: Address already in use

The mentioned workaround unset JMX_PORT does work.

# docker exec -it kafka1 bash -c "unset JMX_PORT; /kafka/bin/kafka-topics.sh --zookeeper server1.server.de:2181 --list"
test

Shouldn't that be default? Or at least be mentioned in the docs?

@ches
Copy link
Owner

ches commented Apr 8, 2017

Ahhhhh so the problem here is docker exec, I wish that was made clear from the start. @robwdux asserted that he was following the README.

I came up short attempting to create a topic when following the readme… I receive an exception attempting to create a topic within the instance to the zk ensemble (those doing the quickstart with a single node to try it out will likely encounter this issue).

(Emphasis mine)—my README does not instruct people to use docker exec. Don't do that. TL;DR, use docker run.

He is executing a single process in the container (Kafka), but is attempting to execute kafka-topic.sh within the container to verify the cluster state.

Using docker exec means you are executing a second process in the container 😝

This is a feature of last resort for unusual debugging situations. 95% of the time people are using docker exec, they don't need to. You're effectively mutating the expected state of a running container instance, which I regard as antithetical to good Docker practices.

I will not accept the changes proposed here because:

  1. The image was not designed with running multiple processes in the container in mind, and I don't intend to account for that as regular usage.
  2. JMX_PORT is the environment variable expected by users of the canonical Kafka distribution, KAFKA_JMX_PORT is non-standard and not used by the official Kafka scripts. The JMX configuration is already confusing and I don't wish to further complicate understanding.
  3. The image is behaving as expected. Download the Kafka tarball and follow the quick start, but export JMX_PORT=7203 in both the shell where you start the broker and the shell where you run kafka-topics.sh. You'll encounter the same issue. That's essentially what's happening with docker exec, you're getting the same shell environment in two processes that share a port.

There is a simple solution: use docker run --rm ches/kafka kafka-topics.sh, as documented.

If you really must exec for some reason, add --env JMX_PORT= (empty value).

By the way, you'll also likely encounter this problem if you run multiple container instances with --network=host. That's also a Docker feature of last resort and you're on your own to --env or remap ports.

@ches ches closed this Apr 8, 2017
@ches
Copy link
Owner

ches commented Apr 8, 2017

Something I'll consider that would make this less likely to be encountered: giving JMX_PORT an empty value by default so it's opt-in to enable, since that is the case with the Kafka distribution out of the box anyway.

Still though, break your docker exec habits 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants