DOCKER-USER iptables chain missing in 19.03.3

[dougburks]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

DOCKER-USER iptables chain should exist in docker-ce 19.03.3 just like it did in previous releases.

Actual behavior

Install docker-ce 19.03.3 and there is no DOCKER-USER iptables chain.

Steps to reproduce the behavior

Install docker-ce 19.03.3 on Ubuntu 16.04 or CentOS 7.
iptables -nvL
There is no DOCKER-USER listed in the output.

Output of docker version:

sudo docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:59:54 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 00:58:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-65-generic
 Operating System: Ubuntu 16.04.6 LTS
 OSType: linux
 Architecture: x86_64

Additional environment details (AWS, VirtualBox, physical, etc.)
physical

[dougburks]

We've verified this issue on multiple different systems, running both Ubuntu 16.04 and CentOS 7. In every case, iptables showed the DOCKER-USER chain on a previous version of docker-ce and after upgrading to docker-ce 19.03.3 iptables no longer has the DOCKER-USER chain.

There is no mention of iptables changes in the 19.03.3 release notes:
https://docs.docker.com/engine/release-notes/

However, iptables is mentioned in the 19.03.3 changelog:
https://github.com/docker/docker-ce/releases

from docker-archive/engine#330:

docker/libnetwork#2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
fixes docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
fixes moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
fixes docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway

We do NOT run dockerd with iptables=false but is it possible that there is some kind of logic bug where those changes are applying to us anyway and thus not creating the DOCKER-USER chain?

Thanks in advance for any assistance.

[arkodg]

@dougburks I can reproduce this issue as well

Can you please add these rules manually for now before you insert your user rules

iptables -N DOCKER-USER
iptables -I FORWARD -j DOCKER-USER
iptables -A DOCKER-USER -j RETURN

Working on the fix right now

[dougburks]

Thanks for the fast response @arkodg !

Can you tell me what the plan is for the fix?

Will there be an immediate re-release of 19.03.3 with the fix?

Thanks!

[dougburks]

Hello,

Thanks for the progress being made over at moby/libnetwork#2464!

Just a quick follow-up to see if a determination has been made about getting the fix out. It looks like 19.03.3 is still available for download and I don't see any notes about this issue in the Release Notes. I think this is a severe issue that at minimum will catch folks by surprise. It can potentially result in security issues if users were restricting access via rules in DOCKER-USER that are no longer being enforced.

We've asked members of our open source project to hold off on updates until we know what the next step is:
https://blog.securityonion.net/2019/10/docker-package-issues-please-do-not.html

From one open source project to another, please let us know what your plans are so that we can notify and protect our users accordingly.

Thanks in advance for your consideration!

[arkodg]

@dougburks we are yet to make a decision
We will be mentioning the issue in the Docker Docs soon with the above workaround.
Thanks for highlighting this regression

[dougburks]

Thanks for the update @arkodg !

Is the above workaround correct? If we append the DOCKER-USER reference to FORWARD, then that would be after the jump to DOCKER, so DOCKER-USER would have no way of restricting access to DOCKER, right? Wouldn't FORWARD need to jump to DOCKER-USER before DOCKER?

For what it's worth, my opinion is that this release has the potential of enough security issues with sufficient severity to warrant an immediate re-release to include the proper bug fix.

Thanks again for your consideration!

[arkodg]

good catch, I usually edit the output of iptables-save where everything is an append , edited the above commands to reflect the fact that DOCKER-USER should get matched before other chains

[andrewhsu]

Release notes for 19.03.3 updated with workaround instructions in Known Issues section: https://github.com/docker/docker-ce/releases/tag/v19.03.3

[dougburks]

Thanks @arkodg for the update and thanks @andrewhsu for the documentation!

Updating the release notes is a good first step, but I'm still concerned that there are folks that have added download.docker.com to their repo list and are going to run a standard system update via apt, yum, etc. and thus never read the release notes. If they were relying on rules in DOCKER-USER to restrict access to DOCKER, then those rules are then silently disabled. This has the potential to result in systems getting compromised without the system owner ever realizing what happened or why it happened.

May I offer my opinion again? I think the right thing to do is an immediate re-release with the bug fix. This helps limit the risk of exposure for folks described above which is better for your community and ultimately better for you.

As always, thanks for your consideration!

[andrewhsu]

@dougburks we're working on a timeline to get a fix out for just 19.03 sooner than our next planned monthly cycle. when we get that sorted, the dates for docker-ce 19.03.4 milestone will be updated: https://github.com/docker/docker-ce/milestone/41

[dougburks]

Thanks for the update @andrewhsu !

Have you considered taking down 19.03.3 until you have a fixed version available? That would help limit exposure for the folks mentioned above that won't ever see the release notes.

Thanks again!

[thaJeztah]

revert in libnetwork on master moby/libnetwork#2466 and backported to the 19.03 branch; moby/libnetwork#2470. vendor update for the engine 19.03 branch was just opened: docker-archive/engine#404

[andrewhsu]

The original issue has been addressed with release of docker-ce 19.03.4.

[dougburks]

Thanks @andrewhsu !

[sureshkachwa]

I still have the same issue , below are the logs and I did add DOCKER-USER chain manually but still the issue is same.

'/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.8.4 (nf_tables): Chain 'DOCKER' does not exist#012Try iptables -h' or 'iptables --help' for more information. Oct 27 07:35:41 airbyte-504841 firewalld[1541]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.8.4 (nf_tables): Chain 'DOCKER' does not exist#012Try iptables -h' or 'iptables --help' for more information.
Oct 27 07:35:41 airbyte-504841 firewalld[1541]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.8.4 (nf_tables): Chain 'DOCKER' does not exist#012Try `iptables -h' or 'iptables --help' for more information.