multiple ports in docker driver port fowarding overwrites previous ones

[fffonion]

For reporting security vulnerabilities please refer to the website.

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

v0.10.2

Operating system and Environment details

Ubuntu 18.04

Issue

When a container listens one port, it's not possible to forward to multiple ports.

Reproduction steps

Job file (if appropriate)

Consider there's a nginx task listening on 80 and i want to port forward to 8000 and 8001:

job "test" {
  group "test" {

    task "test" {
      driver = "docker"

      config {
        image = "nginx:latest"

        port_map = {
          http1    = 80
          http2    = 80
        }
      }

      resources {
        network {
          port "http1" {
            static = 8000
          }
          port "http2" {
            static = 8001
          }
        }
      }
    }
  }
}

^ the above will result in only http2 being forwarded

job "test" {
  group "test" {

    task "test" {
      driver = "docker"

      config {
        image = "nginx:latest"

        port_map = {
          http1    = 80
        }
      }

      resources {
        network {
          port "http1" {
            static = 8000
            static = 8001
          }
        }
      }
    }
  }
}

^ this will not result in an error but only 8001 is forwarded.

Currently it's only possible to let nginx to listen to two ports, and use different ports in port_map.

[tgross]

Hi @fffonion !

I don't think this can be supported in Docker, at least out of the box. You can verify this yourself by forwarding 2 ports:

$ docker run -it --rm -p 8002:8000 -p 8001:8000 busybox:1 nc -lvp 8000
listening on [::]:8000 ...

This results in the following network configuration for the container (via docker inspect :id | jq '.[0].NetworkSettings'):

{
  "Bridge": "",
  "SandboxID": "8d1d84cc9dbb63e5e6e41ed41af84feb211084b96519ec2ee9ba0a835b25fdc7",
  "HairpinMode": false,
  "LinkLocalIPv6Address": "",
  "LinkLocalIPv6PrefixLen": 0,
  "Ports": {
    "8000/tcp": [
      {
        "HostIp": "0.0.0.0",
        "HostPort": "8001"
      },
      {
        "HostIp": "0.0.0.0",
        "HostPort": "8002"
      }
    ]
  },
  "SandboxKey": "/var/run/docker/netns/8d1d84cc9dbb",
  "SecondaryIPAddresses": null,
  "SecondaryIPv6Addresses": null,
  "EndpointID": "0dd0c4110199ca58c24e57f0ebd26463838a3e6350ebe5e86f93b17fb5be1a8d",
  "Gateway": "172.17.0.1",
  "GlobalIPv6Address": "",
  "GlobalIPv6PrefixLen": 0,
  "IPAddress": "172.17.0.2",
  "IPPrefixLen": 16,
  "IPv6Gateway": "",
  "MacAddress": "02:42:ac:11:00:02",
  "Networks": {
    "bridge": {
      "IPAMConfig": null,
      "Links": null,
      "Aliases": null,
      "NetworkID": "e162ee46a46fe88e734a553d8fa58057a09c62c803261c3eace779c18be51022",
      "EndpointID": "0dd0c4110199ca58c24e57f0ebd26463838a3e6350ebe5e86f93b17fb5be1a8d",
      "Gateway": "172.17.0.1",
      "IPAddress": "172.17.0.2",
      "IPPrefixLen": 16,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "MacAddress": "02:42:ac:11:00:02",
      "DriverOpts": null
    }
  }
}

Then in another terminal, if you try nc 127.0.0.1 8002 and nc 127.0.0.1 8001, only one of the two clients will be able to connect.

This might be the sort of thing that could be supported with a CNI plugin, but I'd want to tag in my colleague @nickethier to see if he has thoughts on that.

[fffonion]

Hi @tgross thanks for investigating this. I feel like the behaviour of only one client is able to connect is depending on the nc -l command that it will only accept one connection and exit. If that's changed to a server that can accept multiple times like nginx or redis for example, both ports can be connected.

[tgross]

🤦 omg, you're right... that's embarrassing!

Ok, I've verified that does work just fine with nginx, so it's something in the way we configure the jobspec for Docker. Tagging this appropriately for further investigation.

[fffonion]

@tgross No worries :) Looking forward to the update!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.