Flaky test: port is already allocated

[powersj]

Hi,

In telegraf, we make heavy use of test-containers to do integration testing of telegraf + various services. In the last couple of months we have found we have a very flaky test that results in the same error message:

Error:          Received unexpected error:
                container failed to start: Error response from daemon: driver failed programming external connectivity on endpoint peaceful_germain (8fe4467b369aab776e5f70add82cb20981fd7bea81a6cafc748f09406a9d1194): Bind for 0.0.0.0:9092 failed: port is already allocated: failed to start container
Test:           TestConnectAndWriteIntegration
Messages:       failed to start container

Here is the specific test that is flaky, and here is how we start up containers.

It only seems to be reproducible in CircleCI, which makes me think it is load related. However, this test case uses its own network, so I am confused as to why it would say the port is already allocated.

Is there something obvious I am missing? Or a way to further debug this situation?

Thanks!

[mdelapenya]

Hi @powersj thanks for reaching to us! The first thing I see is that the container request is using fixed ports, which means that even though running in its own network, the 9092 port is published to the host. Please see https://github.com/influxdata/telegraf/blob/d0aaabb4cafca1459dfc0f0dd37095fad56352ea/plugins/outputs/kafka/kafka_test.go#L50

- ExposedPorts: []string{"9092:9092", "9093:9093"},
+ ExposedPorts: []string{"9092", "9093"},

So I see plausible that at some point the port is allocated in the CI worker.

I wonder if you are using any of those fixed ports to connect from the host. If so, you can leverage the APIs to get the MappedPort or the Endpoint of the container.

[powersj]

Hi @mdelapenya,

Thank you for the response!

which means that even though running in its own network, the 9092 port is published to the host.

I now remember why we did this. The kafka client will reach out to the kafka broker to get metadata about the cluster. Our client uses the metadata to determine what the broker leader address and port are. While the first step of getting metadata is fine using the random mapped port, the metadata will turn around and say use port 9092 for the leader. Kafka does not know about the mapped ports, which will then cause our client fail to connect if we did not use the fixed port.

I think what is happening is we have a second kafka test, for our kafka input, that also maps port 9092 to the host for this same reason. The result is that we can sometimes have a conflict, where one test might still be running.

Is there a better way to set up networking such that our test and container are all in the same network and I could specify specific ports, without mapping?

Thank you again!

[mdelapenya]

I'm back-channeling the kafka issue to our kafka gurus (not me 😢 ). In the meantime, have you taken a look at the Kafka Kraft module? https://golang.testcontainers.org/modules/kafka/

[eddumelendez]

Hi @powersj, Fixed ports are not really recommended because that will cause the issue you described. Read more about Testcontainers Best Practices. Also, the networking doesn't change anything due to kafka is the only container using it.

Regarding to the issue, the trick here is ADVERTISED_LISTENERS in Kafka. You can read more about it here. ADVERTISED_LISTENERS are not configured in your tests and that's why is relying on fixed ports. In order to fix it, a listener using the random port should be added but that's not an easy task, you can see how Testcontainers for Go configure it dynamically here. For the first test you shared you can rely on kafka module pointed by @mdelapenya, at this time, the module only support Raft mode (Zookeeperless). For the second one, which is using an external zookeeper, we will need to add support based on Testcontainers for Java experience or you can take inspiration from Testcontainers for Go implementation and look at Java to add zookeeper until the support is ready.

[powersj]

Fixed ports are not really recommended because that will cause the issue you described.

Right, I think these are the only two tests with fixed ports, due to the issue above of not knowing the random port from the Kafka metadata.

For the first test you shared you can rely on kafka module pointed by @mdelapenya,

This appears to work nicely, I'll give it a shot in our CI shortly.

For the second one, which is using an external zookeeper

I am happy to migrate this test away from an external zookeeper and do the same in both. The integration test is validating we can talk to Kafka, so I am less concerned about the internals or how zookeeper is configured.

Thank you for the responses!

[mdelapenya]

@powersj I'm resolving this discussion, as we found a way to identify the flakiness, which came from the fixed ports and the kafka+zookeeper setup.

Thanks for your contributions to Testcontainers Go!