runc clone binary mount too slow boot shim boot timeout，then runc.XXXXXX residual

[kamizjw]

Description

docker starts or execs a container, runc starts the init process and needs to copy the runc binary to the temporary file runc.XXXXXX.During this period, if the mount operation is too slow and the shim startup times out and fails to return, the runc.XXXXXX file will not be deleted and remain

Steps to reproduce the issue

Describe the results you received and expected

container exit and no runc.XXXXXX file residual

What version of runc are you using?

[root@localhost ~]# runc --version
runc version 1.1.3
commit: d2cbb1a3afab8170d4942fc0be8d0275cb11642c
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.3

Host OS information

No response

Host kernel information

No response

[kolyshkin]

Is there a repro?

[kamizjw]

Is there a repro?

It's easy to repro.I modified the try_bindfd function, add sleep(2000) between "/proc/self/exe" mount to runc.XXXXXX and try_bindfd return. docker will return timeout and runc.XXXXXX will residual.

	ret = -EPERM;
	if (mount("/proc/self/exe", template, "", MS_BIND, "") < 0)
		goto out;
	sleep(2000);
	if (mount("", template, "", MS_REMOUNT | MS_BIND | MS_RDONLY, "") < 0)
		goto out_umount;
	/* Get read-only handle that we're sure can't be made read-write. */
	ret = open(template, O_PATH | O_CLOEXEC);

# [root@localhost ~]# docker run -itd 4062f0d37663 /bin/bash
fbea6d71dcb6fc4a4af224374d4df804b4640585ad1221c62a1ea50838c089eb
docker: Error response from daemon: OCI runtime create failed: timeout 120s for cmd(pid=740164): /usr/bin/runc, [runc --root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/fbea6d71dcb6fc4a4af224374d4df804b4640585ad1221c62a1ea50838c089eb/log.json --log-format json create --bundle /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/fbea6d71dcb6fc4a4af224374d4df804b4640585ad1221c62a1ea50838c089eb --pid-file /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/fbea6d71dcb6fc4a4af224374d4df804b4640585ad1221c62a1ea50838c089eb/init.pid --console-socket /tmp/pty4181474061/pty.sock fbea6d71dcb6fc4a4af224374d4df804b4640585ad1221c62a1ea50838c089eb]: : unknown.

[cyphar]

I mean ... I don't really know what we can do here -- presumably containerd is doing a kill -9 of runc in this case and it's not possible for us to do any real cleanup. Do you actually have a real machine that takes that >2 minutes to do a mount? If so, you should report that as a kernel bug -- mounts are not meant to be that slow.

For what it's worth, I am working on a kernel patch that will eliminate the need for this userspace protection entirely, and I plan to remove bindfd in the next minor runc version (switching to always use memfds).

EDIT: Yeah, containerd is SIGKILLing us here. We can't do any cleanup because we're dead.

[kamizjw]

I mean ... I don't really know what we can do here -- presumably containerd is doing a kill -9 of runc in this case and it's not possible for us to do any real cleanup. Do you actually have a real machine that takes that >2 minutes to do a mount? If so, you should report that as a kernel bug -- mounts are not meant to be that slow.

For what it's worth, I am working on a kernel patch that will eliminate the need for this userspace protection entirely, and I plan to remove bindfd in the next minor runc version (switching to always use memfds).

EDIT: Yeah, containerd is SIGKILLing us here. We can't do any cleanup because we're dead.

ok，if always use memfds might solve this problem

[cyphar]

We've switched to only using memfds now with #3931. Let us know if this fixes the issue for you @kamizjw.

[kamizjw]

We've switched to only using memfds now with #3931. Let us know if this fixes the issue for you @kamizjw.

this definitely helps with my problem，thanks！