Multiple stop and starts results in a docker exception when port is exposed

[guenhter]

Hi,

when I create a container with a fixed exposed port and call start + stop multiple times, I get an error. First things first. Here is the container for my tests (I know there is a dedicated Nginx TestContainer but this is just to reproduce the problem and Nginx is easy to use):

class MyContainer : FixedHostPortGenericContainer<MyContainer>("nginx:1.17.0-alpine") {
    override fun configure() {
        withFixedExposedPort(5000, 80)
    }
}

and here the function using the container:

fun main() {
    val container = MyContainer()
    
    repeat(10) {
        println("--> Running Container: $it")
        container.start()
        container.stop()
    }
}

When I run this, the first iteration is ok, but in the second one I get this root cause:

	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testcontainers.dockerclient.AuditLoggingDockerClient.lambda$wrappedCommand$14(AuditLoggingDockerClient.java:98)
	... 8 more
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"driver failed programming external connectivity on endpoint adoring_rubin (43003e9ce801010fc73cd3dbc3717f89781d2330a68cb1d918e0cfc149cd630b): Bind for 0.0.0.0:5000 failed: port is already allocated"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:115)
	at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:28)
	at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:11)
	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:46)
	... 13 more

I thought It might be a docker issue but then just tried out this:

fun main() {
    repeat(10) {
        println("--> Running Container: $it")
        // Moved this inside the loop to create a new container each iteration
        val container = MyContainer()  
        container.start()
        container.stop()
    }
}

And this works.

But shouldn't the other approach by creating one container and calling start/stop multiple times also work?

[bsideup]

One of the reasons why we strongly advise agains fixed ports is issue like this one.
AFAIK Docker may race, especially on Windows/Mac, because it starts a proxy for exposed ports.

I think the reason why it works when you move it to repeat is because the constructor will resolve an image, it adds a few millis and prevents the race.
.stop() will kill & remove the container, and .start starts a new one, always. You're not just restaring it, you're recreating the container in both cases, so it shouldn't really differ.

The best I can suggest is to not use the fixed ports 🤷‍♂️

[guenhter]

Hm, I also tried it with a 5 second sleep. Doesn't help.

True, I also prefer random ports, but currently I create an integration test which restarts rabbitmq a couple of times to see if the application can handle outages. Therefore the port should not change after the restart.
You see my point here...

[bsideup]

Have you tried https://www.testcontainers.org/modules/toxiproxy/ ?

[guenhter]

C'mon. This is exactly what was looking for. Nice tool!

My problem is solved by this tool. So for me it's good enough, but I guess the problem somehow is still there (isn't it)?
What you think? Should there be some investigation to this issue (I also can help out) or do you want to close it?

[bsideup]

@guenhter we're of course happy to receive any help!
It's a bit hard to say whether there is a problem with .stop()/.start() in general or just the fixed ports thing (since the exception is from Docker daemon).
I suspect it is due to the fixed ports, which is a very rare API usage in Testcontainers and I doubt we will have capacity to debug it.

Your help with at least the investigation would help a lot 👍

[guenhter]

Tell you what. I am closing this for now and if I find something, I'll repoen it for discussion.

[bsideup]

@guenhter ok, thanks for reporting 👍 Happy that your use case is unblocked, one way or another :)

[bensche]

I would be willing to provide a code-fix for this. Could you pleas reopen this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[stale]

This issue has been automatically closed due to inactivity. We apologise if this is still an active problem for you, and would ask you to re-open the issue if this is the case.

[rehevkor5]

It looks to me like this can also happen if the container fails to launch the first time. The port problem is encountered during the retry. For example, if using MySQLContainer with withInitScript() and the script fails.