fakeroot has become extremely slow inside docker

[bwoodsend]

Description

Entering a fakeroot inside a container used to be instant but now takes about a minute.

Reproduce

docker run -it --rm alpine
apk add fakeroot
fakeroot 

Expected behavior

The original behaviour - fakeroot being quick.

docker version

Client:
 Version:           23.0.3
 API version:       1.42
 Go version:        go1.20.2
 Git commit:        3e7cbfdee1
 Built:             Wed Apr  5 13:17:36 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20
  Git commit:       bc3805a0a0
  Built:            Sat Feb 11 13:58:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.0
  GitCommit:        1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
 runc:
  Version:          1.1.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.10.4
    Path:     /usr/lib/docker/cli-plugins/docker-buildx

Server:
 Containers: 95
  Running: 4
  Paused: 0
  Stopped: 91
 Images: 125
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.9-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.495GiB
 Name: manjaro-2212
 ID: HBWP:AGRX:MHCB:XEVM:37M2:Z4PF:37RF:GDXA:7KP4:DERV:2JNI:XGKR
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Can also be reproduced on Fedora. Can't be reproduced with podman which is why I suspect a docker regression rather than a fakeroot one.

[thaJeztah]

Wondering if this could be related to #38814

Does it make a difference if you start the container with --ulimit '1024:1048576' ?

[bwoodsend]

I'm guessing you mean --ulimit "nofile=1024:1048576"? Yes that does indeed fix it.

[bwoodsend]

In that case, I have no objections to closing this one off in favour of #38814.

[thaJeztah]

In this case, I think containerd/containerd#7566 would be the cause of your issue (assuming containerd is running as its own systemd service). You can set the custom defaults in daemon.json or update the systemd unit with an override file (but that approach currently won't work until a new containerd version is released that is built with go1.19.9 or up)

Set custom default per-container limits in the daemon.json file;

Create the /etc/docker directory if it doesn't exist;

mkdir -p /etc/docker

Create (or edit the existing) /etc/docker/daemon.json file to set custom per-container limits;

{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Soft": 1024,
      "Hard": 1048576
    }
  }
}

Restart the docker daemon:

systemctl restart docker.service

Run a container to check the limits (this one shows that the hard and soft limit are both set to 524288, which is related to the Golang bug);

docker run --rm busybox sh -c 'ulimit -Sn; ulimit -Hn'
1024
1048576

Customize the systemd unit using an override unit (needs Golang update)

Currently, there's an issue with Go 1.19 and Go 1.20 that set the hard-limit to the soft-limit (which is resolved in the latest patch release of Go, but requires new packages to be built; see https://github.com/containerd/containerd/pull/7566/files#r1150907094), but setting other limits would probably already help;

To set custom limits, you can add a systemd override file that sets a fixed value instead of infinity;

Run sudo systemctl edit containerd.service; this will open an editor with a template for a systemd override file;

### Editing /etc/systemd/system/containerd.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file



### Lines below this comment will be discarded

Edit the file to add your override for the LimitNOFILE option in the Service

### Editing /etc/systemd/system/containerd.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
LimitNOFILE=1024:524288

### Lines below this comment will be discarded

Run systemctl cat containerd.service to check the result; it should print the output of both the original service and the override;

# /lib/systemd/system/containerd.service
...

# /etc/systemd/system/containerd.service.d/override.conf
[Service]
LimitNOFILE=1024:524288

Run systemctl daemon-reload to reload the systemd units.

Run systemctl restart containerd.service to restart containerd

Run a container to check the limits (this one shows that the hard and soft limit are both set to 524288, which is related to the Golang bug);

docker run --rm busybox sh -c 'ulimit -Sn; ulimit -Hn'
524288
524288

[thaJeztah]

Let me close this one, as it duplicates #38814 but feel free to continue the conversation.