Latest Docker for Mac - apparent change to port forwarding breaks TCP liveness checks

[rnorth]

This long-standing part of our container liveness checks seems to be broken on Docker for Mac from at least beta 14 onwards:

new Socket(ipAddress, port).close();

The original behaviour of this was to simply verify that the relevant TCP port was listening, catching an IOException if the container was not listening yet. However, now Docker for Mac seems to be running its own perpetually listening TCP proxy that accepts the connection regardless of whether there is actually a listening process inside the container.

e.g. outside of testcontainers, this demonstrates the behaviour:

$ docker run -p 8080 -d alpine:latest sh -c "yes | nc -l -p 8080"   # real listening process inside container
$ docker run -p 8080 -d alpine:latest sh -c "yes > /dev/null"   # port mapped, but not listening

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                     NAMES
c77d07ba926c        alpine:latest       "sh -c 'yes | nc -l -"   About a minute ago   Up About a minute   0.0.0.0:32822->8080/tcp   small_mirzakhani
871c2c80ceaf        alpine:latest       "sh -c 'yes > /dev/nu"   6 minutes ago        Up 6 minutes        0.0.0.0:32821->8080/tcp   tender_raman

$ lsof | grep LISTEN
[.. snip ..]
com.docke 1481 rnorth   11u    IPv4 0x90cfe27e6cde7eb3         0t0      TCP *:32821 (LISTEN)
com.docke 1481 rnorth   13u    IPv4 0x90cfe27e6a3ab7ab         0t0      TCP *:32822 (LISTEN)

This seems to make the current wait check too dumb!

I'd be keen to know if others are experiencing this.

I suspect the fix should include opening an inputstream and verifying that the first read doesn't indicate end-of-stream, but I'd like to do some checking to be sure that this doesn't have unintended consequences.

[rnorth]

Scratch that idea as a potential fix - as I suspected, it only works where the container process has something to send and doesn't wait for data from the client first.

[rnorth]

Brief update: doing the equivalent of docker exec <id> cat /proc/net/tcp might potentially be a solution for checking whether the container is listening on a port, but for test cases at least it's impeded by gliderlabs/docker-alpine#143. It seems that a netcat process in the container (as used for simulating a simple running server in the testcontainers test suite) does not expose listening port data in the TCP socket table for some reason.

[rnorth]

Should be fixed in a later DfM beta, as per this post: https://forums.docker.com/t/port-forwarding-of-exposed-ports-behaves-unexpectedly/15807/2

[rnorth]

Current recommended workaround for this issue is to implement a custom service-specific WaitStrategy that checks the service is correctly running. e.g. use an actual client library for the service to check, such as in 64d3451.

[bsideup]

@rnorth got bitten by this issue. Looks like it's not fixed in the latest release of DfM. Do you have any ideas what to do with it?

Update:

I managed to workaround it by using:

setWaitStrategy(new AbstractWaitStrategy() {
  @Override
  protected void waitUntilReady() {
    Unreliables.retryUntilTrue(1, TimeUnit.MINUTES, () -> {
      ExecResult result = container.execInContainer("/bin/bash", "-c", "</dev/tcp/localhost/" + getExposedPorts().get(0) + " && echo success");

      return result.getStdout().contains("success");
    });
  }
});

It uses BASH's magical device /dev/tcp/$HOST/$PORT. Not sure if we can use it in TC until Docker is fixed, but at least it helps to workaround the issue.

[rnorth]

Ah, interesting @bsideup. The workaround (that IIRC @outofcoffee has used) is a service-specific wait strategy. e.g. to change the test from 'is the socket listening?' to 'does the socket talk HTTP?'

It's frustrating. Your workaround is very clever; I'd tried something similar but not as elegant 😆

I think bringing this into TC and making it the default might be a hack too far, but perhaps there's some merit in making it available so people can opt-in..? I like the principle of us doing the hard/dirty work so that users don't have to 😄

[bsideup]

@rnorth doing it right now (bash-based port check in Docker for Mac environment), will send PR soon