Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start rootless container under Podman when in WSL #2484

Open
Pvlerick opened this issue Oct 27, 2023 · 10 comments
Open

Unable to start rootless container under Podman when in WSL #2484

Pvlerick opened this issue Oct 27, 2023 · 10 comments

Comments

@Pvlerick
Copy link
Contributor

Although #719 has been fixed in #2370 - I checked it using versions of Youki before and after under Ubuntu; the issue still remains when running in WSL.

podman run -it --rm --runtime youkidev alpine:latest
Error: ERROR libcontainer::process::container_intermediate_process: failed to add task to cgroup pid=Pid(43606) err=V1(WrappedIo(CreateDir { err: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }, path: "/sys/fs/cgroup/cpuacct/3652f57e7e8a876e538ca3d186cad1bfb0cdb8554171cab8a39d7c3c58c29110" })) init=true: OCI permission denied

youkidev being defined in /usr/share/containers/containers.conf as pointing to the latest build of youki from master

./youki info
Version           0.3.0
Commit            97488350
Kernel-Release    5.15.90.1-microsoft-standard-WSL2
Kernel-Version    #1 SMP Fri Jan 27 02:56:13 UTC 2023
Architecture      x86_64
Operating System  Debian GNU/Linux 11 (bullseye)
Cores             8
Total Memory      15859
Cgroup setup      hybrid
Cgroup mounts
Capabilities
CAP_BPF           available
CAP_PERFMON       available
CAP_CHECKPOINT_RESTORE available

I would be interested to work on this myself, but I'm ramping up on Rust right now. Any pointers to where to start appreciated, otherwise I'll just dig and dig more.

@YJDoc2
Copy link
Collaborator

YJDoc2 commented Oct 30, 2023

Hey @Pvlerick ,thanks for reporting this. Can I ask you to check running youki 0.2.0 or older version, and see what is the error message? I think under wsl, we do not have systemd, so the changes in #2370 would not have any relation with this. If in youki 0.2.0 the error message is the same, then the wsl setup uses cgroups v2 manager and not systemd cgroups manager. If that is the case, then the cgroups v2 manager would be a good place to start looking.

@Pvlerick
Copy link
Contributor Author

Using 0.2.0:

./youki info
Version           0.2.0
Commit            ed75057
Kernel-Release    5.15.90.1-microsoft-standard-WSL2
Kernel-Version    #1 SMP Fri Jan 27 02:56:13 UTC 2023
Architecture      x86_64
Operating System  Debian GNU/Linux 11 (bullseye)
Cores             8
Total Memory      15859
Cgroup setup      hybrid
Cgroup mounts
Capabilities
CAP_BPF           available
CAP_PERFMON       available
CAP_CHECKPOINT_RESTORE available
podman run -it --rm --runtime /tmp/youki-0.2.0/youki_0_2_0_linux/youki-0.2.0/youki alpine:latest
Error: IO error

               Caused by:
                             Permission denied (os error 13)
                                                            ERRO[0000] Error removing container 80e6b0528249d12fa0ebbc16d3cf56a0a0d3e6c421270ec4611e455cc7d4782b from runtime after creation failed
Error: IO error

Caused by:
    Permission denied (os error 13)
Error:     Permission denied (os error 13): OCI permission denied

I'm running systemd in WSL:

systemctl --version
systemd 247 (247.3-7+deb11u4)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

If that gives you any idea, let me know :-)

In any event, I'll start looking around cgroups manager.

@utam0k
Copy link
Collaborator

utam0k commented Oct 31, 2023

@Pvlerick May I ask you to use the latest version(v0.3.0)?

@Pvlerick
Copy link
Contributor Author

@utam0k that's what I did initially #2484 (comment)

I probably didn't explain myself very well; I tried under WSL with 0.3.0, 0.2.0 and the latest from master and I have similar errors. I tested under Ubuntu "bare metal" (no vm) and it works like a charm since 0.3.0.

@Pvlerick
Copy link
Contributor Author

Pvlerick commented Nov 16, 2023

Turns out it's cgroup manager V1 that is used in the end:

youki[13853]: error in executing command: failed to receive. "waiting for intermediate process". BrokenChannel
                                         Caused by:
                                             channel connection broken
...
youki[13310]: cgroup manager V1 will be used
youki[13853]: failed to run container process
youki[13855]: failed to run intermediate process
youki[13855]: failed to add task to cgroup

Hacking in the code to force it to use V2 or systemd gets the same error, so I think there might be something other at play here. The plot thickens! :-)

@YJDoc2
Copy link
Collaborator

YJDoc2 commented Nov 17, 2023

Hey @Pvlerick Thanks for the debugging! There is also #2528 which is related to a bug in cgroups v1 creation. Not sure if that would be the root cause, but can you try this PR branch #2532 which aims to fix that? If it does, then we might need to do something similar with cgroups v2

On the other hand, I'm not even sure if for rootless, we can use v1 or v2 directly... As we delegate the cgroup management to systemd when running rootless (as non-root, we don't have perms for manipulating cgroups) we need systemd to handle it for us...

@Pvlerick
Copy link
Contributor Author

Under Ubuntu (not in WSL), it's indeed systemd that is used. I'll checkout that branch and see what happens, thanks for the pointers.

@Pvlerick
Copy link
Contributor Author

Same error with #2532, unfortunately.

@YJDoc2
Copy link
Collaborator

YJDoc2 commented Dec 18, 2023

@Pvlerick Hey, sorry I couldn't be of much help here, did you find out anything? I currently don't have access to WSL, so cannot re-produce this locally, but for rootless, the systemd cgroups manager must be used.

@Pvlerick
Copy link
Contributor Author

@YJDoc2 no problem, I still have this issue on my radar but I don't have time to work on it right now. I'll eventually get to it, don't worry :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants