Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#30 Shim: fix the "deadlock avoided" error during the rootfs mount #31

Merged
merged 1 commit into from
Aug 15, 2022

Conversation

akhramov
Copy link
Contributor

@akhramov akhramov commented Aug 3, 2022

Issue number: #30

Description of changes:

Currently, the sporadic "mount_nullfs: /path/to/rootfs: resource
deadlock avoided" error occurs during containerd shim task creation.
The exact reason for this error is unknown, but a mounter that
executes the system call instead of running a shell command seems to
fix the problem.

This change replaces the containerd's mounter with moby/sys/mount

Testing done:

I ran the buildkit's integration test locally with local vagrant: moby/buildkit#2376
Reproduced the error, and ensured that the error is not reproducible with the updated containerd shim.

Terms of contribution:

By submitting this pull request, I agree that this contribution is licensed under the terms found in the LICENSE file.

@samuelkarp
Copy link
Owner

The exact reason for this error is unknown, but a mounter that executes the system call instead of running a shell command seems to fix the problem.

The existing implementation also executes a system call (calls unix.Unmount, which calls unmount(2)). I think the difference is that the moby/sys/mount implementation has a slow path which sorts and unmounts nested mountpoints.

Currently, the sporadic "mount_nullfs: /path/to/rootfs: resource
deadlock avoided" error occurs during containerd shim task creation.
The exact reason for this error is unknown, but a mounter that
sorts and unmounts nested mountpoints seems to fix the problem.

This change replaces the containerd's mounter with `moby/sys/mount`

Fixes samuelkarp#30

Signed-off-by: Samuel Karp <me@samuelkarp.com>
@samuelkarp samuelkarp merged commit aa66a93 into samuelkarp:main Aug 15, 2022
@samuelkarp
Copy link
Owner

samuelkarp force-pushed the fix/shim-mount-rootfs branch from 1caf307 to aa66a93

I reworded the commit message but otherwise didn't change any of the code. Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants