Youki fails when some unused capability is missing

[jprendes]

I am trying to run youki in a container context (think Docker in Docker), and it fails to run if the container doesn't have all the capabilities enabled.

In particular, this also happens if some capabilities added in kernel 5.8 / 5.9 are not present (like CAP_BPF or CAP_CHECKPOINT_RESTORE). I haven't tested in a post-5.3 and pre-5.9 kernel, but I think it would hit the same issue.

Note that runc does work in the same situation.

Please see the attachment for reproduction steps: reproduction.zip

[jprendes]

The issue is caused by reset_effective setting all the capabilities listed in caps::all(), which includes the capabilities added in kernel 5.8 / 5.9.
I don't understand why this is needed, in either in container_init_process or set_id.

[yihuaf]

@jprendes Thank you for the bug report. dind is a special case where we have not tested before. So I am not surprised that it does not work out of box. Also thank you for the PR, but I remember the reset capabilities has a purpose. However, it's been a while since I touched that part of the code, so I don't recall exactly the reason off the top of my head. I will have to take a deeper look.

I believe the first step to address this issue is to set up an integration test like we do for containerd and k8s. Then we can start to explore how to consistently support the dind usecase.

In addition, it would be good to understand why runc works whereas we fail in this case.

The issue is caused by reset_effective setting all the capabilities listed in caps::all(), which includes the capabilities added in kernel 5.8 / 5.9.

This sounds like a bug to me. May be caps::all() is not the right choice here.

[jprendes]

@yihuaf thanks for looking into this.
I've simplified the reproductions to not involve docker. Just plain youki and plain runc running inside a container.
reproduction.zip

Additionally, doing an strace -f --trace=capset ... on youki and runc, I see that youki does 4 calls to capset:

capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=ALL, permitted=ALL, inheritable=0}) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, permitted=ALL, inheritable=0}) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, permitted=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, inheritable=0}) = 0
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, permitted=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, inheritable=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE}) = 0

while runc does only one call (the same as the last call from youki):

capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, permitted=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE, inheritable=1<<CAP_KILL|1<<CAP_NET_BIND_SERVICE|1<<CAP_AUDIT_WRITE}) = 0

[jprendes]

May be caps::all() is not the right choice here.

Yeah, I think this is the way to go.

The problem is that the thread is that caps::all() returns a hardcoded list. The thread can only acquire capabilities in the permitted set, so it should be caps::read(None, CapSet::Permitted), anything beyond that will result in an error.

I've updated the PR to reflect this.