Port forwarding should be through iptables

[jbeda]

As we've disabled docker managing IP tables we no longer get it setting up explicit port forwarding. Instead we are falling back on user mode proxying that is built into docker to do this port forwarding.

We should either build our networking mode into Docker proper or have the kubelet set up and maintain the IPTables forwarding rules we need.

[thockin]

Why did we disable IP tables?

[jbeda]

We needed to disable the blanket NAT for all traffic out of the bridge and only nat for ! 10.0.0.0/8. But the Docker flag for disable that also disabled the NAT for setting up iptables for incoming port maps. Docker then falls back on user mode proxying of that traffic.

We either need to make Docker itself more flexible or start setting up the port forwarding in the kubelet. If we do it in the kubelet we could do something like:

1. Set up the netns container for the pod
2. Set up the port forwards
3. Launch other containers for the pod

If we want to adapt Docker, it might be enough to split the iptables flag into a flag for configuring the iptables for the bridge in general vs. configuring iptables for the incoming port mapping.

[thockin]

I feel like a moron for not understanding the problem after that
explanation. Why do we need to disable NAT? I'm still digging into the
impl of Kubelet, so maybe it will emerge.

Aren't you on vacation?

On Tue, Jun 24, 2014 at 6:21 AM, Joe Beda notifications@github.com wrote:

We needed to disable the blanket NAT for all traffic out of the bridge and
only nat for ! 10.0.0.0/8. But the Docker flag for disable that also
disabled the NAT for setting up iptables for incoming port maps. Docker
then falls back on user mode proxying of that traffic.

We either need to make Docker itself more flexible or start setting up the
port forwarding in the kubelet. If we do it in the kubelet we could do
something like:

1. Set up the netns container for the pod
2. Set up the port forwards
3. Launch other containers for the pod

If we want to adapt Docker, it might be enough to split the iptables flag
into a flag for configuring the iptables for the bridge in general vs.
configuring iptables for the incoming port mapping.

Reply to this email directly or view it on GitHub
#15 (comment)
.

[brendandburns]

Joe is the expert on our config, but I think we needed to disable NAT to be
able to assign our own IP addresses.

Brendan
On Jun 24, 2014 8:26 AM, "Tim Hockin" notifications@github.com wrote:

I feel like a moron for not understanding the problem after that
explanation. Why do we need to disable NAT? I'm still digging into the
impl of Kubelet, so maybe it will emerge.

Aren't you on vacation?

On Tue, Jun 24, 2014 at 6:21 AM, Joe Beda notifications@github.com
wrote:

We needed to disable the blanket NAT for all traffic out of the bridge
and
only nat for ! 10.0.0.0/8. But the Docker flag for disable that also
disabled the NAT for setting up iptables for incoming port maps. Docker
then falls back on user mode proxying of that traffic.

We either need to make Docker itself more flexible or start setting up
the
port forwarding in the kubelet. If we do it in the kubelet we could do
something like:

1. Set up the netns container for the pod
2. Set up the port forwards
3. Launch other containers for the pod

If we want to adapt Docker, it might be enough to split the iptables flag
into a flag for configuring the iptables for the bridge in general vs.
configuring iptables for the incoming port mapping.

Reply to this email directly or view it on GitHub
<
#15 (comment)

.

—
Reply to this email directly or view it on GitHub
#15 (comment)
.

[jbeda]

Quick clarification and then back to vacation :)

We want traffic between containers to use the container API across nodes. Say we have Node A with a container IP space of 10.244.1.0/24 and Node B with a container IP space of 10.244.2.0/24. And we have Container A1 at 10.244.1.1 and Container B1 at 10.244.2.1. We want Container A1 to talk to Container B1 directly with no NAT. B1 should see the "source" in the IP packets of 10.244.1.1 -- not the "primary" IP for Node A. That means that we want to turn off NAT for traffic between containers (and also between VMs and containers).

But we don't support (yet?) the extra container IPs that we've provisioned talking to the internet directly. We don't map external IPs to the container IPs. So we solve that problem by having traffic that isn't to the internal network (! 10.0.0.0/8) get NATed through the primary host IP address so that it can get 1:1 NATed by the GCE networking when talking to the internet. Similarly, incoming traffic from the internet has to get NATed/proxied through the host IP.

So we end up with 3 cases:

1. Container -> Container or Container <-> VM. These should use 10. addresses directly and there should be no NAT.
2. Container -> Internet. These have to get mapped to the primary host IP so that GCE knows how to egress that traffic. There is actually 2 layers of NAT here: Container IP -> Internal Host IP -> External Host IP. The first level happens in the guest with IP tables and the second happens as part of GCE networking. The first one (Container IP -> internal host IP) does dynamic port allocation while the second maps ports 1:1.
3. Internet -> Container. This also has to go through the primary host IP and also has 2 levels of NAT, ideally. However, the path currently is a proxy with (External Host IP -> Internal Host IP -> Docker) -> (Docker -> Container IP). Once this bug is closed, it should be External Host IP -> Internal Host IP -> Container IP. But to get that second arrow we have to set up the port forwarding iptables rules per mapped port.

If this is still confusing let me know and I can find some time for a hangout to dive into the details.

[bgrant0607]

Docker issues that may be relevant to the iptables issue:
moby/moby#6002
moby/moby#6034

[thockin]

I think this is fixed, or at least soon. With my PR for true IP-per-pod, I am pretty sure we circumvent Docker's proxying entirely.

[jbeda]

Tim -- can you confirm that we use iptables when you specify a hostPort in a pod manifest? Docker was setting that up. Did we find a way to have Docker do it? Or are we doing it in the kubelet now?

[thockin]

Ah, right, HostPorts. Yes, in case of HostPorts we do still get proxied through docker, it seems.

[brendandburns]

I'm going to move this out of the 0.5 release, as I think that this is a P2 and thus not a blocker.

[jbeda]

Your call -- but this means that any hostPort traffic will go through the docker process. It'll be slow and might restrict some types of serving workloads. As we get a better external access story this may or may not become more important.

[errordeveloper]

@slintes suggests that it works without the --iptables=false --ip-masq=false flags now (weave/issues#260 (comment)).

[bluehallu]

@errordeveloper Your link is broken, correct link is weaveworks/weave#260 (comment)