[Bug]: 1.19.0 regression with Kafka

[pgehl]

Module

Kafka

Testcontainers version

1.19.0

Using the latest Testcontainers version?

Yes

Host OS

Linux, Mac OS

Host Arch

x86, arm, amd64

Docker version

Client:
 Cloud integration: v1.0.31
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:52 2023
 OS/Arch:           darwin/amd64
 Context:           default

Server: Docker Desktop 4.19.0 (106363)
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

When upgrading from 1.18.3 to 1.19.0 out tests starting a kafka container are failing to start a container.

Relevant log output

===> Running preflight checks ... 
    ===> Launching ... 
    ===> Launching kafka ... 
    [2023-08-24 20:26:10,676] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
    [2023-08-24 20:26:11,003] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
    [2023-08-24 20:26:11,047] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
    java.lang.IllegalArgumentException: Error creating broker listeners from 'EXTERNAL_PLAINTEXT://localhost:32894,EXTERNAL_SASL_PLAINTEXT://localhost:32892,INTERNAL_PLAINTEXT://kafka-broker.testcontainer.docker:9094,BROKER://kafka-broker.testcontainer.docker:9092': No security protocol defined for listener EXTERNAL_PLAINTEXT
    	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:273)
    	at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:1924)
    	at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2033)
    	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1997)
    	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1471)
    	at kafka.Kafka$.buildServer(Kafka.scala:67)
    	at kafka.Kafka$.main(Kafka.scala:87)
    	at kafka.Kafka.main(Kafka.scala)
    Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener EXTERNAL_PLAINTEXT
    	at kafka.cluster.EndPoint$.$anonfun$createEndPoint$2(EndPoint.scala:49)
    	at scala.collection.immutable.Map$Map2.getOrElse(Map.scala:307)
    	at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:49)
    	at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:57)
    	at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:270)
    	at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
    	at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
    	at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
    	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:270)

Additional Information

No response

[eddumelendez]

Hi, can you share how the KafkaContainer is being created, please?

[alex-kuck]

Hi, we are having the same issue, Kafka container is configured like this:

private fun startKafka(): GenericContainer<*> = KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
			.withExposedPorts(9093)
			.withEnv(
				mapOf(
					"KAFKA_LISTENER_SECURITY_PROTOCOL_MAP" to "BROKER:PLAINTEXT,PLAINTEXT:SASL_PLAINTEXT",
					"KAFKA_LISTENER_NAME_PLAINTEXT_SASL_ENABLED_MECHANISMS" to "PLAIN",
					"KAFKA_LISTENER_NAME_PLAINTEXT_PLAIN_SASL_JAAS_CONFIG" to
						"org.apache.kafka.common.security.plain.PlainLoginModule required " +
						"username=\"$kafkaUsername\" " +
						"password=\"$kafkaPassword\"" +
						"user_admin=\"$kafkaPassword\"; ",
				),
			)
			.withReuse(true)
			.apply { start() }

[mfvanek]

We faced with the similar issue after upgrading from 1.17.6 to 1.18.3 and 1.19.1.
Only on Linux.
On macOS everything works fine.

Logs with 1.19.1

    2023-10-23 18:43:27.238 ERROR [example-kotlin-spring-one,,] 520 --- [    Test worker] t.d.a.t.ru/confluentinc/cp-kafka:7.3.3   : Could not start container
    com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"command terminated with exit code 2"}
    	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
    	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.put(DefaultInvocationBuilder.java:223)
    	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveToContainerCmdExec.execute(CopyArchiveToContainerCmdExec.java:34)
    	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveToContainerCmdExec.execute(CopyArchiveToContainerCmdExec.java:13)
    	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    	at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
    	at org.testcontainers.shaded.com.github.dockerjava.core.command.CopyArchiveToContainerCmdImpl.exec(CopyArchiveToContainerCmdImpl.java:162)
    	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:339)
    	at org.testcontainers.containers.KafkaContainer.containerIsStarting(KafkaContainer.java:261)
    	at org.testcontainers.containers.GenericContainer.containerIsStarting(GenericContainer.java:714)
    	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:490)
    	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
    	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
    	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:334)

Container is trivial

import org.springframework.boot.test.util.TestPropertyValues
import org.springframework.context.ApplicationContextInitializer
import org.springframework.context.ConfigurableApplicationContext
import org.testcontainers.containers.KafkaContainer
import org.testcontainers.containers.wait.strategy.Wait
import org.testcontainers.utility.DockerImageName

class KafkaInitializer : ApplicationContextInitializer<ConfigurableApplicationContext> {

    override fun initialize(applicationContext: ConfigurableApplicationContext) {
        kafka.start()
        TestPropertyValues.of(
            "spring.kafka.bootstrap-servers=" + kafka.bootstrapServers
        ).applyTo(applicationContext.environment)
    }

    companion object {
        internal val kafka: KafkaContainer = KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.3.3"))
            .waitingFor(Wait.forListeningPort())
    }
}

[eddumelendez]

@mfvanek why are you doing it like this?

internal val kafka: KafkaContainer = KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.3.3"))
            .waitingFor(Wait.forListeningPort())

KafkaContainer is a pre-configured class and it contains already a reliable wait strategy. You don't need .waitingFor(Wait.forListeningPort())

[mfvanek]

@eddumelendez we use colima for local development on macOS. Sometimes first test in test run may fail without proper waiting strategy. It's especially important for PostgreSQL. For Kafka container it looks like a copy-paste but nevertheless I've tried to remove that line of code and nothing changed :(

[eddumelendez]

@alex-kuck Although, the issue about not overriding the env vars is happening, I think you are missing some additional configuration, at least for what I read in the official documentation. I am not able to reproduce it.

The issue about overriding the env vars should be fixed in the next release due to some internal changes and I would like to add an additional test for jaas but can not reproduce it so far with the information provided.

Can you please provide a github repository that works with 1.18.3?

[alex-kuck]

@eddumelendez Hey, sorry for the late reply 😅
Here is a reproduction example: https://github.com/alex-kuck/testcontainer-regression-reproduction
However, it seems with 1.19.3 the issue seems to be fixed already. Thanks!