Docker Network bypasses Firewall, no option to disable

[BenjamenMeyer]

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 14
 Running: 5
 Paused: 0
 Stopped: 9
Images: 152
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 204
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-58-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.793 GiB
Name: brm-pheonix-dev
ID: Y6Z4:6D53:RFOL:Z3CM:P7ZK:H6HL:RLV5:JT73:LZMC:DTBD:7ILK:2RS5
Username: benjamenmeyer
Registry: https://index.docker.io/v1/

Additional environment details (AWS, VirtualBox, physical, etc.):
Rackspace Cloud Server, Ubuntu 14.04, but that shouldn't really matter

Steps to reproduce the issue:

1. Setup the system with a locked down firewall
2. Create a set of docker containers with exposed ports
3. Check the firewall; docker will by use "anywhere" as the source, thereby all containers are exposed to the public.

Describe the results you received:
root@brm-pheonix-dev:~/rse# iptables --list DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:6379

Describe the results you expected:
root@brm-pheonix-dev:~/rse# iptables --list DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 127.0.0.0/24 172.17.0.2 tcp dpt:6379
ACCEPT tcp -- 172.16.0.0/16 172.17.0.2 tcp dpt:6379

Additional information you deem important (e.g. issue happens only occasionally):

By default docker is munging the firewall in a way that breaks security - it allows all traffic from all network devices to access the exposed ports on containers. Consider a site that has 2 containers: Container A exposes 443 running Nginx, and Container B runs an API on port 8000. It's desirable to open Container A to the public for use, but hide Container B entirely so that it can only talk to localhost (for testing by the user) and the docker network (for talking to Container A). It might also be desirable for testing purposes to have Container C be a database used by Container B with the same kind of restrictions.

I found this because of monitoring logs on a service I had thought was not open to the public. After finding log entries from sources trying to break in, I checked the firewall rules and found there was no limit on the source addresses or interfaces. I use UFW and only allow SSH onto this particular box, and would prefer to keep it that way. This can dramatically impact using Docker containers to deploy services and lead to potential security problems if people are not careful.

The best security practice would be to by default limit the networking to work like above desired effect example, and then allow the user to add the appropriate firewall, etc rules to override such behavior, or have an option to revert to the current behavior. I know that for legacy reasons that is not likely since it would break a lot of things on up-date; so at least having an option to enable the above that can be turned on now would be a good first step, and perhaps later after much warning make it the default behavior. Assuming the default behavior is secure, having functionality to manage this (firewall->enable public port, ip) in the docker-compose yml would be a great way to visibly make it known what is going on.

I did find the --iptables=false option, however, I don't want to have to be setting all the rules myself. The only thing I am objecting to is the source setting for the rules.

While I have not verified it, I suspect all the firewalls supported by docker will have the same issue.

[BenjamenMeyer]

Note: I noticed in https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/iptables/iptables.go that there is not even an existing option to set the source, so it's just using the iptables defaults for source ip/device.

[thaJeztah]

Duplicate of #14041?

[justincormack]

It is not quite #14041 as this issue is talking about exposed ports. Exposing ports is intended to make them publicly accessible, as this is how you expose services to the outside world. If you are working in a development environment, you can either disable access to ports from outside your computer with a host firewall, or simply not expose the ports and access the services directly, or from other containers on the same network.

I would recommend you use the newer docker networking features to set up private networks for services that you do not want exposed at all, see https://docs.docker.com/engine/userguide/networking/

[thaJeztah]

That's what I thought of first; but was a bit confused, because exposing a port (EXPOSE), doesn't actually do anything, but publishing a port (-p / -P) actually exposes it on the host.

If you're actually talking about publishing, then this is as designed;

In your example, container B and C should not publish their ports, and container A can communicate with them through the Docker Network, e,g.

docker network create mynet

docker run -d --net=mynet --name=api api-image
docker run -d --net=mynet --name=db database-image
docker run -d --net=mynet --name=web -p 443:443 nginx

This only publishes the "web" container to the host, The web container can access the "API" and "database" containers through their name (I.e. http://api:80/, and db:3306 (assuming MySQL))

[BenjamenMeyer]

@justincormack so I don't think using a private network solves the issue. In my case I'm using a private network between the containers, and they're still publicly exposed because the host firewall isn't configured to limit the exposure to the private network.

@thaJeztah the issue still comes down to the firewall support - there's no firewall support in docker to limit it to a specific network. You can probably still access those containers from another system as the firewall will not prevent other systems from accessing the port on the host.

Now I'm running this via docker-compose; however, it's not entirely a docker-compose issue since the libnetwork functionality has no capability of limiting the network in the firewall rules - iptables rules have no source specification so regardless of how one configures the network as long as one relies on docker to create the firewall rules (which one should because it's more likely to get them right) then this becomes an issue. Consider the following in a docker-compose.yml file:

nginx:
    build: ./docker/nginx/.
    ports:
        - "127.0.0.1:8080:80"
        - "127.0.0.1:443:443"
    environment:
        DESTINATION_HOST: repose
    links:
        - repose
repose:
    build: ./docker/repose/.
    ports:
        - "127.0.0.1:80:8080"
    environment:
        DESTINATION_HOST: phoenix
        DESTINATION_PORT: 8888
    links:
        - phoenix
curryproxy:
    build: ./docker/curryproxy/.
    ports:
        - "127.0.0.1:8081:8081"
    external_links:
        - rse_rse_1
        - rse_rse_2
        - rse_rse_3
phoenix:
    build: .
    ports:
        - '127.0.0.1:88:8888'
    links:
        - curryproxy:curry
    external_links:
        - rse_rse_1:rse
        - rse_rse_2
        - rse_rse_3
        - rse_cache_1:cache
    volumes:
        - .:/home/phoenix

The above is an excerpt from one of my projects. While I want to be able to test all of them locally from my host, I don't want anyone else to be able to access anything but the nginx instance.

I'm not sure how this translates to your nomenclature...it may be that this is part of the "publishing" aspect, and the publishing capability needs to be expanded to do what I'm saying.

If this is by design, then it's a poor security model as you now expose all developers to extreme risks when on unfamiliar networks (e.g traveling).

As I said, I don't expect the default to change immediately but having the option would be a good first step.

[justincormack]

I am a bit confused then, can you give some examples externally of what you can connect to? The backend services will be (by default) on the 172.17.0.0/16 network, which you will not be able to access externally I wouldn't think first because you will not have a route to that defined from an external host.

There is a potential issue if your external IP is also a private IP that traffic will not be dropped that is routed for the internal networks (whereas it should be from public to private) - is that the issue?

[BenjamenMeyer]

@justincormack so I'm primarily setting up proper proxying so that some services can only be hit via the proxy (nginx - ssl termination), which then filters through an authentication proxy (repose), and finally off to another service (phoenix). I could care less if all of them are bound to the 0.0.0.0 interface; but I only want nginx to be externally accessible (or at the very least the repose portion if I didn't have nginx in place here). An easy solution, for example, would be to not have to set "127.0.0.1" in the configuration, but have a firewall section where it's easy to specify that to allow through the firewall with a base configuration of only the docker network and local host (loopback) interfaces enabled to talk - something like:

firewall:
    external:
        ports:
            - 80
            - 443

Now the situation can be mitigated somewhat by limited the network mapping on the host to 127.0.0.1 instead of the default 0.0.0.0 map. Note that this is what really mitigates it because otherwise the bridging will forward the host port into the docker network.

And yes, I did verify that that limiting works; however, it still leaves potential vulnerabilities in place and the firewall rules do not match what is actually being done.

As another example, there was a Linux Kernel vulnerability a little while back (having trouble finding it at the moment) that was related to ports that were marked in IPtables as being opened for use by applications, but then not actually being connected to an application - for instance, being on a local host port but not a public IP port. This potentially sets that up, and it would be better practice to limit the IPtables rules to the expected networks instead of leaving them open to connect from any where. As I said, at the very least have the option to specify. They've likely fixed that particular issue but why leave the possibility open?

IOW, it's all about security.

[thaJeztah]

@BenjamenMeyer if you don't want the other services to be accessible, why do you publish their ports at all? i.e. "127.0.0.1:8081:8081" is not needed if it's only accessed through the docker network (other services connect directly through the docker network)

[seeruk]

One issue I have that's related to this one is that I would like to publish ports, but only allow certain IP addresses to access them.

For example, I run a Jenkins environment in a couple of containers. The master node is "published", but I have to make some pretty convoluted iptables rules to lock it down so that only the 2 offices we have can access it.

Is there a way around this currently built into Docker? Or at least a recommended practice? I've seen in the documentation how you might restrict access to 1 IP address; but not several. The other issue with this is that if you have a server that already has an iptables configuration, you might be resetting all of the rules before applying your rules (hence the convoluted rules I have had to set up).

[enzeart]

I have an issue similar to the one stated by @seeruk. There is a jarring violation of expectation when preexisting firewall rules don't apply to the published container ports. The desired behavior is as follows (for me at least)

1. User connection attempt is filtered based on INPUT configurations, etc
2. Traffic forwarding then happens as usual based on the docker-added FORWARD rules

Is there a succinct way to achieve this in iptables, or does it not easily permit such a construct. I'm particularly limited in my knowledge of iptables so bear with me. I've just recently picked up knowledge about it while trying to understand docker's interactions with it.

[seeruk]

What I've actually resorted to for the time being, since I am actually running these containers on a pretty powerful dedicated server, I've set up a KVM VM running Docker, then using some more standard iptables rules to restrict access from the host. The VM has it's own network interface that's only accessibly from the server, so I have to add rules to explicitly allow access to ports in iptables on the host. I have lost a little bit of performance, but not much.

[BenjamenMeyer]

@thaJeztah I want to be able to access it from the local system, and test against it easily. For example, setting up a RESTful HTTP API that has a Health end-point and being able to reliably run curl against it by using localhost (I have to document this for others and having IP addresses that change is not reliable). In most cases for my dev environment I only want the containers to talk to each other, but I also want to be able to access it from the host.

For @seeruk's case, being able set an IP block (5.5.0.0/16 - a valid parameter for a source address in iptables rules) would be a very good thing. IPtables already has the capability to do the limiting, but docker is not taking advantage of it.

[BenjamenMeyer]

@thaJeztah I set "127.0.0.1:8081:8081" explicitly to keep it off the external network; I had found logs in my docker containers from people trying to crack into the containers via the exposed ports.

My work around right now is to turn off docker containers before I leave for the day because I can't ensure the environment I want to be external actually is external, or that the environment is properly limited for security purposes.

[thaJeztah]

@BenjamenMeyer one way to do this is running those tests in a container, e.g.

docker run --net -it --rm --net=mynetwork healthchecker 

[jcheroske]

The issue that Ben is bringing to light is real and surprising (a bad combination). Many admins, like myself, are using the tried-and-true ufw firewall. Docker is doing and end-run around ufw and altering the iptables rules is in such a way that it 1) causes ufw to misreport the current status of the packet filtering rules, and 2) exposes seemingly private services to the public network. In order for docker to remain in the good graces of the sysadmin community, another approach must be devised. Right now there are many admins out there, who, like Ben and myself, inadvertently opened ports to the wider Internet. Unlike Ben and myself though, they have not figured it out yet.

[BenjamenMeyer]

@thaJeztah that assumes that I am doing it via the command-line and not using another tool where I only have to set an IP address.

For example, I'm working on an API. I have a tool that I can use to work with that API in production to support it; for development of the tool and the API I want to just point the tool at the dockerized API. The tool knows nothing about docker, nor should it. And I don't necessarily want to put the tool into docker just to use it - pointing it at a port exposed only to the local host should be sufficient.

[BenjamenMeyer]

@jcheroske I agree, but I don't know that there's a good solution to that aspect. For that, ufw probably needs to be made smarter to be able to lookup and report on rules that it wasn't involved in creating. There's lot of software out there that can adjust the iptables rules in ways that ufw (or AFAIK firewalld, etc) won't know about. There's not really a simple solution to fixing that either.

That said, it would be nice if Docker could integrate with those to dump out the appropriate config files to be able to enable/disable them, or integrate with those tools such that it gets hooked in and dumps out the information appropriately, however, given there are better solutions I don't think that aspect will really be solved. Here, it's more about just limiting the scope of the iptables rules that are being generated to at least minimize the potential impacts by allowing the specification of the source (lo, eth0, 127.0.0.0/24, etc).

[seeruk]

If you are willing to do so, using iptables does make this totally possible.

This is a trimmed-down example of how you can use it: https://gist.github.com/SeerUK/b583cc6f048270e0ddc0105e4b36e480

You can see that right at the bottom, 1.2.3.4 is explicitly given access to port 8000 (which is exposed by Docker), then anything else to that port is dropped. The PRE_DOCKER chain is inserted to be before the DOCKER chain so that it is hit first, meaning the DROP stops the blocked requests from ever reaching the DOCKER chain.

It's a bit annoying that Docker doesn't have this functionality built-in, but it is possible work around it right now.

Another alternative would be using an external firewall. Some places like AWS and Scaleway offer things like security groups where you can manage access to your boxes from outside, from there every port behaves the same way.

I never actually managed to figure out how to make this work with UFW. Though for now, I'm happy with using iptables as a solution. It seems to be working out very well for me so far.

Obviously, this isn't much of a great solution if you have already built a reasonably complicated set of firewall rules around UFW. Though, it does make using things like iptables-persistent quite easy. You can also use alternative ways of allowing access to this way that seem more "normal" in iptables.

[mavenugo]

@BenjamenMeyer have you thought of using an user-defined docker network with a subnet & ip-range option and assigning a static ip-address for containers & using them for local development so that you don't have to depend on a virtual static ip such as 127.0.0.1 ? That will avoid the need to have port-mapping all together for those containers that are private to a host.

docker network create --subnet=30.1.0.0/16 --ip-range=30.1.0.0/24 mynetwork
docker run --net=mynetwork --ip=30.1.1.1 --name=myservice1 xxxx
docker run --net=mynetwork --ip=30.1.1.2 --name=myservice2 yyyy

With this setup, myservice2 can reach myservice1 by name myservice1 and there is no need to even depend on the static ip. Also the host can reach the static-ip freely without the need to have port-mapping.

Also with compose 1.7, you can specify static ip address for containers and specify network subnets and ranges.

[jcheroske]

I did figure out a simple workaround.

1. Edit /etc/default/docker: DOCKER_OPTS="--iptables=false"

2. Add ufw rule: ufw allow to <private_ip> port <port>

So simple that it really makes me wonder why the --iptables=false option is not the default. Why create such a situation when all docker has to do is say, "Hey, if you're running a firewall you're going to have to punch a hole through it!" What am I missing?

https://fralef.me/docker-and-iptables.html
http://blog.viktorpetersson.com/post/101707677489/the-dangers-of-ufw-docker

[enzeart]

I can't get docker to stop modifying iptables to save my life. Tried updating /etc/default/docker to no avail on Ubuntu 16.04

[seeruk]

@enzeart Try /lib/systemd/system/docker.service.

[enzeart]

@seeruk Bless your soul

[thaJeztah]

@enzeart to configure a daemon running on a host that uses systemd, it's best to not edit the docker.unit file itself, but to use a "drop in" file. That way, you won't run into issues when upgrading docker (in case there's a newer docker.unit file). See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for more info.

You can also use a daemon.json configuration file, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

[BenjamenMeyer]

@mavenugo There's already a docker network in place.

@jcheroske that works, but as I noted it would mean that the end-user (me) would then have to make sure that all iptables rules were correct, which is not optimal and not nearly as likely to happen as having docker do it automatically, thus this issue.

[BenjamenMeyer]

update per @cpuguy83 on another issue - #4737 (comment)

[schklom]

Hi everyone,
My Debian 11 (Bullseye) and Ubuntu 22.04 (Jammy) Docker instances work really well with ufw, and I managed to track down why.

Fix

The fix is much easier (to me at least) than messing with iptables

$ sudo apt-get install -y iptables arptables ebtables
$ sudo update-alternatives --set iptables /usr/sbin/iptables-nft
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft
$ sudo update-alternatives --set arptables /usr/sbin/arptables-nft
$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-nft

If you want return to normal for some reason

$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
$ sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Context

I had done this years ago, before Debian wrote that nftables are the default, after needing it for the fix at https://github.com/crazy-max/docker-fail2ban#use-iptables-tooling-without-nftables-backend.
For some reason, the first chunk of code makes my ufw rules lock down Docker as I would expect.

To be clear,

ubuntu@docker_machine:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp                  DENY IN     1.2.3.4

prevents 1.2.3.4 from connecting to the machine on port 80 with protocol TCP, but does not prevent e.g. SSH connections on port 22.
ufw works also really well for a more locked-down scenario with default deny (incoming).

Note that my Debian server uses Rootless Docker and this does not seem to cause any problem.

Why this trick works is beyond me, I will let more knowledgeable people figure it out.

Edit: my bad, it only works on my Debian server

[BenjamenMeyer]

Hi everyone,
My Debian 11 (Bullseye) and Ubuntu 22.04 (Jammy) Docker instances work really well with ufw, and I managed to track down why.

Fix

The fix is much easier (to me at least) than messing with iptables

$ sudo apt-get install -y iptables arptables ebtables
$ sudo update-alternatives --set iptables /usr/sbin/iptables-nft
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft
$ sudo update-alternatives --set arptables /usr/sbin/arptables-nft
$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-nft

If you want return to normal for some reason

$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
$ sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

## Context had done this years ago, before Debian wrote that [`nftables` are the default](https://wiki.debian.org/iptables), after needing it for the fix at https://github.com/crazy-max/docker-fail2ban#use-iptables-tooling-without-nftables-backend. For some reason, the first chunk of code makes my `ufw` rules lock down Docker as I would expect.

To be clear,

ubuntu@docker_machine:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp                  DENY IN     1.2.3.4

prevents 1.2.3.4 from connecting to the machine on port 80 with protocol TCP, but does not prevent e.g. SSH connections on port 22.
ufw works also really well for a more locked-down scenario with default deny (incoming).

Note that my Debian server uses Rootless Docker and this does not seem to cause any problem.

Why this trick works is beyond me, I will let more knowledgeable people figure it out.

That doesn't solve the problem. You're only looking at what UFW shows. Check your full firewall rules. The Docker chains are not visible to UFW and bypass your rule set.

[schklom]

I was mistaken for the Ubuntu server (can't make it work there), and the Debian server uses the legacy tables not the new ones.

But I confirm that on the Debian server (a Raspberry Pi 4), the UFW IP and port restrictions really work with Docker.

You're only looking at what UFW shows.

No, I tested it, that's why I thought it may be helpful here. One of my rules

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
50000/tcp                  ALLOW       172.16.20.1

lets only traffic from my router pass to a containous/whoami container.
From my router

$ curl -v "http://172.16.20.51:50000"
*   Trying 172.16.20.51:50000...
* Connected to 172.16.20.51 (172.16.20.51) port 50000 (#0)
> GET / HTTP/1.1
> Host: 172.16.20.51:50000
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 06 May 2023 15:08:17 GMT
< Content-Length: 178
< Content-Type: text/plain; charset=utf-8
< 
Hostname: whoami
IP: 127.0.0.1
IP: 192.168.50.2
IP: 192.168.58.15
RemoteAddr: 172.16.20.1:21860
GET / HTTP/1.1
Host: 172.16.20.51:50000
User-Agent: curl/7.80.0
Accept: */*

and from any other device, after around a minute it fails

$ curl -v "http://172.16.20.51:50000"
*   Trying 172.16.20.51:50000...
* connect to 172.16.20.51 port 50000 failed: Connection timed out
* Failed to connect to 172.16.20.51 port 50000 after 129700 ms: Connection timed out
* Closing connection 0

but when I allow other devices to access with sudo ufw allow from <device_ip> to port 50000, the curl command works from that device.

---

EDIT

I figured out how to make UFW work with Rootless Docker on Ubuntu. Forgive me if my technical language is not ideal, I am not a pro.

I have installed and use Rootless Docker, switched to using legacy iptables + arptables + ebtables, allowed to expose unprivileged ports, enabled CPU, CPUSET, and I/O delegation, allowed ping, and pasted

[Service]
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns"

in ~/.config/systemd/user/docker.service.d/override.conf

Now, I confirm that UFW can correctly deny and allow my phone IP from accessing Docker containers. curl commands from my phone either time out or go through depending on UFW.

I am not sure if all steps were necessary, but it now works on Debian and Ubuntu servers with Docker containers.

[dm17]

The docker documentation should say "using a host firewall in front of docker is still highly experimental" and link this thread.

[aki-k]

The best solution I was able to find for CentOS 7 was this:

https://unrouted.io/2017/08/15/docker-firewall/

I use it with some modifications on my Swarm nodes.

[BenjamenMeyer]

You're only looking at what UFW shows.

No, I tested it, that's why I thought it may be helpful here. One of my rules

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
50000/tcp                  ALLOW       172.16.20.1

lets only traffic from my router pass to a containous/whoami container. From my router

Please post a printout of your entire routing table, not just what UFW shows:

$ sudo iptables --list

[schklom]

Please post a printout of your entire routing table, not just what UFW shows:

$ sudo iptables --list

Sure.

I simplified the UFW rules with more common ports for readability, and confirmed that they work.
Also, I replaced the IP of the device I banned with 1.2.3.4 here for privacy.

ubuntu@docker3:~$ sudo ufw prepend deny from 1.2.3.4 to any port 80
Rule inserted

ubuntu@docker3:~$ sudo ufw status verbose 
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80                         DENY IN     1.2.3.4             
22                         ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  

ubuntu@docker3:~$ sudo iptables --list
Chain INPUT (policy DROP)
target     prot opt source               destination         
ufw-before-logging-input  all  --  anywhere             anywhere            
ufw-before-input  all  --  anywhere             anywhere            
ufw-after-input  all  --  anywhere             anywhere            
ufw-after-logging-input  all  --  anywhere             anywhere            
ufw-reject-input  all  --  anywhere             anywhere            
ufw-track-input  all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ufw-before-logging-forward  all  --  anywhere             anywhere            
ufw-before-forward  all  --  anywhere             anywhere            
ufw-after-forward  all  --  anywhere             anywhere            
ufw-after-logging-forward  all  --  anywhere             anywhere            
ufw-reject-forward  all  --  anywhere             anywhere            
ufw-track-forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-output  all  --  anywhere             anywhere            
ufw-before-output  all  --  anywhere             anywhere            
ufw-after-output  all  --  anywhere             anywhere            
ufw-after-logging-output  all  --  anywhere             anywhere            
ufw-reject-output  all  --  anywhere             anywhere            
ufw-track-output  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain ufw-after-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-input (1 references)
target     prot opt source               destination         
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-ns
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-dgm
ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn
ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootps
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootpc
ufw-skip-to-policy-input  all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-after-output (1 references)
target     prot opt source               destination         

Chain ufw-before-forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ufw-user-forward  all  --  anywhere             anywhere            

Chain ufw-before-input (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ufw-logging-deny  all  --  anywhere             anywhere             ctstate INVALID
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
ufw-not-local  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251          udp dpt:mdns
ACCEPT     udp  --  anywhere             239.255.255.250      udp dpt:1900
ufw-user-input  all  --  anywhere             anywhere            

Chain ufw-before-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-before-output (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ufw-user-output  all  --  anywhere             anywhere            

Chain ufw-logging-allow (0 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere             ctstate INVALID limit: avg 3/min burst 10
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
ufw-logging-deny  all  --  anywhere             anywhere             limit: avg 3/min burst 10
DROP       all  --  anywhere             anywhere            

Chain ufw-reject-forward (1 references)
target     prot opt source               destination         

Chain ufw-reject-input (1 references)
target     prot opt source               destination         

Chain ufw-reject-output (1 references)
target     prot opt source               destination         

Chain ufw-skip-to-policy-forward (0 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-input (7 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-output (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain ufw-track-forward (1 references)
target     prot opt source               destination         

Chain ufw-track-input (1 references)
target     prot opt source               destination         

Chain ufw-track-output (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW

Chain ufw-user-forward (1 references)
target     prot opt source               destination         

Chain ufw-user-input (1 references)
target     prot opt source               destination         
DROP       tcp  --  1.2.3.4              anywhere             tcp dpt:http
DROP       udp  --  1.2.3.4              anywhere             udp dpt:80
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:22
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http

Chain ufw-user-limit (0 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 5 LOG level warning prefix "[UFW LIMIT BLOCK] "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain ufw-user-logging-forward (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-input (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-output (0 references)
target     prot opt source               destination         

Chain ufw-user-output (1 references)
target     prot opt source               destination         

---

EDIT:
The reason this works seems to be known and documented at #42563 (comment)

[akerouanton]

Hi, I created a discussion about what's going on with custom iptables rules (whether created through ufw or manually) and what we can do to improve that. It's available here:

• Summary of the issues about custom iptables rules #45524

[aradalvand]

Absolutely and utterly embarrassing that this is still just as big of an issue as it was 9 years ago, absolutely embarrassing.

NO FIX in NINE YEARS, for as severe and blatant of a security gotcha as this. Unbelievable.

[msimkunas]

If you're a developer, I think it makes sense to run Docker inside a VM with its own network interface that is only accessible to the VM host. This way Docker does not mess with the VM host's firewall and is contained inside a VM.

Is my understanding correct here? You could use Multipass to create a QEMU VM but in principle this should work with any VM that only exposes its own IP address which is only accessible from the VM host. The host running the VM (e.g. the developer's laptop) then acts as an external firewall.

[s4ke]

@msimkunas Yeah kinda. This is essentially why Docker Desktop is also more secure by default than just dockerd

[MahmoodAbuIssa96]

Disable iptables for Docker
by adding this line to '/etc/docker/daemon.json'

{ "iptables": false }

add daemon.json file if not found

Restart docker

sudo systemctl restart docker

Now docker ports accessible only in your server

in this point you need configure your firewall to open the needed ports
for example

sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --reload

helpful post
https://dev.to/soerenmetje/how-to-secure-a-docker-host-using-firewalld-2joo

[dm17]

Is it generally accepted that using k3s/k8s/etc instead of compose basically remedies this issue?

[BenjamenMeyer]

Is it generally accepted that using k3s/k8s/etc instead of compose basically remedies this issue?

If k3/k8/etc is backed by docker then no.
If it's backed by a different container engine then yes.

Using docker without docker compose would still be vulnerable.

[dm17]

If it's backed by a different container engine then yes.

Right, so CRI-O has been verified by someone as working intuitively with UFW/iptables running on the host OS? I haven't seen anyone touting that as a benefit of the CRI-O backed options vs the containerd (docker) backed options...

[BenjamenMeyer]

If it's backed by a different container engine then yes.

Right, so CRI-O has been verified by someone as working intuitively with UFW/iptables running on the host OS? I haven't seen anyone touting that as a benefit of the CRI-O backed options vs the containerd (docker) backed options...

I can't speak to CRI-0 myself, but I have only observed this with Docker itself due to how docker does networking. The only other implementation indirectly checked was podman because of its docker compatibility and did not observe the same behavior at the time.

K8 does networking differently where it's not an issue, in part the K8 network is more isolated by default and tends to use an application layer proxy. However because of how docker does networking unless k8 was extremely specific in their use of docker I would expect it would still end up vulnerable; even then it might be hard to get it secure since Docker essentially bypasses all the normal firewall rules and patterns with where the Docker Chain is implemented/utilized.

[aki-k]

@cpuguy83 Please try to find a workable solution before the 10 year anniversary of this issue. I left the job I was working in when I was working with this issue. Here's the solution that solved it for me:

https://unrouted.io/2017/08/15/docker-firewall/

[jacoscaz]

I've recently had a chance to run a few network scans across the WiFi network of a local developer meetup with roughly a hundred participants. Sure enough, I was able to connect to a ~dozen open ports that I really should not have been able to connect to. A few things struck me:

• We're talking about slightly more than 10% of active machines. That's a lot.
• None of the six developers I actually managed to track down and speak to were aware of what was going on.
• Another ~dozen developers said that it was only sheer luck that they didn't have anything exposed at the specific time I ran the scans.
• Two developers out of the six I tracked down immediately got in touch with sysadmin / devops colleagues as they were under the impression that some elements of their infrastructure could be at risk.
• Two developers out of the six I tracked down unknowingly had their machine set up in such a way as to allow @koo5 's routing trick to work. Sadly I wasn't able to track this down to the specific rules involved, both because I had run out of time but also because I am not a competent iptables user and nor a sysadmin.

Given the above, I think that docker, docker compose, docker ps and so on should print noticeable warnings about open ports as the final item in their stdout at end of each invocation. Though a relatively quick and minor, strictly non-breaking change, I think the ROI of doing this would be really high.

[msimkunas]

Personally, the only practical solution that I see right now is to run Docker inside a VM. Managing and maintaining the firewall rules directly and hoping that the next Docker update doesn’t silently mess them up is just not an option – it’s brittle and complex, and I’d rather deal with whatever challenges running Docker inside a VM entails than try and hack my way around iptables. I wrote a blog post about this particular issue.

[schklom]

@msimkunas If you don't want to use a VM, look into Rootless Docker.
As I wrote (even though I was condescendingly replied to then ignored for no valid reason), #42563 (comment) shows Rootless Docker does not conflict with UFW. I have done that since Rootless mode came out and UFW started doing its job without surprises.

Edit: I wonder how many downvotes I'll get this time

[msimkunas]

@schklom Actually, I prefer running multiple VMs with Docker inside for isolation reasons. In the worst case, if something doesn’t work, I can just destroy the VM from scratch.