Kubernetes

[xrl]

Kubernetes StatefulSet support is easiest when pulling the broker id out of the hostname pattern

$HOSTNAME="kafka-1" should be broker-id 1

So the start.sh script pulls that out of there with sed. If you set $KAFKA_BROKER_ID before start.sh runs, it'll honor that value.

I upgraded the server.properties.template with the latest from the kafka tar.gz, backporting the config vars and pruning deprecated ones:

• remove deprecated advertised.host.name/advertised.port
• auto.leader.rebalance.enable is true by default, don't need it in config file
• Add LOG_FLUSH_SCHEDULER_INTERVAL_MS so folks can actually flush data (I had huge recheck times on reboot)
• ZOOKEEPER_CONNECTION_STRING and ZOOKEEPER_CHROOT are not compatible if specifying multiple ZK hosts to connect to. Removing ZK_CHROOT and updating documentation to show that can be rolled in to connection string.
• Fix bug with JMX trying to use ipv6. Safer to just specify ipv4.
• KAFKA_PORT has been rolled in to the listeners/advertised.listeners config. Removing from ENV and documenting how to override in docker command.
• ENV vars are listed in the Dockerfile

[xrl]

This PR has been made worse by Kafka's bug with lost+found folders -- those appear in the root of all Linux filesystems. And I am using Kubernetes for it's dynamic volume management. I get that it wants to control the data directory but it's pretty silly for it to bomb out when it sees folders it doesn't know about. See: https://issues.apache.org/jira/browse/KAFKA-742. This PR includes a work around but I fear it is not backwards compatible with people's existing volumes. Advice would be helpful!