Accessing Kafka container from another container

[iNikem]

For my test I have to run 2 containers. One of them is KafkaContainer, another one is our application which uses that Kafka. Now, the problem is that KafkaContainer starts Kafka with ADVERTISED_LISTENER set to host ip of the Socat proxy container. Which is, obviously, not accessible from another container. #452 (comment) seems to be relevant here as well.

So my question is: how can I run another testcontainer which needs to access KafkaContainer?

[bsideup]

Hi @iNikem,

Easy - just use re-mapped port 9093. See example here:
https://github.com/bsideup/liiklus/blob/20fd7efa53410f05108f47743858eb612d83c6d6/examples/plugin/src/test/java/com/github/bsideup/liiklus/plugins/example/support/AbstractIntegrationTest.java#L31

[iNikem]

Well, I can confirm it does work. But it's absolutely not obvious from public API. kafka.getNetworkAliases().get(0) + ":9093" looks like dark magic incantation. You have to read source code even to understand it.

[iNikem]

Essentially we have 2 addresses here. One is "external", for using from the tests, another one is "internal" for using from other containers on the same network. May be both should be exposed as public API?

[bsideup]

@iNikem well, ask Kafka guys why it is so 😄 Also, for your usecase, you don't really need KafkaContainer because the purpose of it is to simplify Kafka usage from the host because of NAT. In your case, simple GenericContainer will do the job

[iNikem]

Well, I want to communicate with Kafka from tests as well, so your proxy is still convenient. And I will make a PR to you, exposing 9093 endpoint in public API as well, if you don't mind.

[bsideup]

@iNikem um... exposing 9093 doesn't make sense o_O What we could do, however, is to change the defaults and keep 9092 for the internal communication and some other port for the socat proxy

[iNikem]

@bsideup Why? By "exposing" I mean providing a nice method wrapping kafka.getNetworkAliases().get(0) + ":9093" in addition to existing getBootstrapServers method. So that other containers could use it.

[kiview]

Like getBootstrapServersForDockerNetwork()?

[bsideup]

@iNikem ah, I see now. I'm afraid if we add such public method, users will try to use it instead of getBootstrapServers :(

[iNikem]

@bsideup I will try to come up with some scary name :)

[bsideup]

The method doesn't really makes any sense. What makes sense is to change the port back to 9092. I just submitted a PR for it:
#733

This way, you can use KafkaContainer as any other container (alias:9092 ) - what you would normally espect

[iNikem]

You still have to know that there are 2 different ways to communicate with Kafka: inside docker network and from your tests.

[kiview]

Sounds like a good solution 👍

@iNikem You are right, but I'm afraid, this would also be the case if we add an additional public API method.

[bsideup]

@iNikem this applies to any container, actually. Either use exposed port on host or exposed inside the network

[iNikem]

Ok, thinking one more time about our requirements, the solution of #733 will satisfy us nicely :) Thanks!

[bsideup]

Released as 1.8.0 🎉