Clarify documented privileges of kubectl debug node

[kgibm]

What would you like to be added:

oc supports chroot /host (and even prints a helpful hint of "To use host binaries, run chroot /host") whereas kubectl causes "Operation not permitted" (even though /host is readable):

oc results:

$ oc debug node/$NODE -t --image=ubuntu -- chroot /host ls
Creating debug namespace/openshift-debug-node-vv26n ...
Starting pod/worker2-debug ...
To use host binaries, run `chroot /host`
bin   dev  home  lib64	mnt  ostree  root  sbin  sys	  tmp  var
boot  etc  lib	 media	opt  proc    run   srv	 sysroot  usr

kubectl results:

$ kubectl debug node/$NODE -it --image=ubuntu -- sh -c "ls /host; chroot /host ls"
Creating debugging pod node-debugger-worker2-xnmxh with container debugger on node worker2.
bin   dev  home  lib64	mnt  ostree  root  sbin  sys	  tmp  var
boot  etc  lib	 media	opt  proc    run   srv	 sysroot  usr
chroot: cannot change root directory to '/host': Operation not permitted

Why is this needed:

This is part of a broader lack of permissions in kubectl debug node that oc debug node does not suffer from which limits certain diagnostics. For example, I can't even look at all PIDs with kubectl even though I'm ostensibly root:

$ kubectl debug node/$NODE -it --image=ubuntu -- sh -c "id; ls -l /proc/3899524 >/dev/null"
Creating debugging pod node-debugger-worker2-4b7vb with container debugger on node worker2.
uid=0(root) gid=0(root) groups=0(root)
ls: cannot read symbolic link '/proc/3899524/cwd': Permission denied
ls: cannot read symbolic link '/proc/3899524/root': Permission denied
ls: cannot read symbolic link '/proc/3899524/exe': Permission denied

Although this works fine with oc:

$ oc debug node/$NODE -t --image=ubuntu -- sh -c "id; ls -l /proc/3899524 >/dev/null"
Creating debug namespace/openshift-debug-node-dhj8k ...
Starting pod/worker2-debug ...
To use host binaries, run `chroot /host`
uid=0(root) gid=0(root) groups=0(root)

The relevant bits of code appear to be the following and show significant differences in the debug node pod templates:

• kubectl's generateNodeDebugPod
• oc's approximatePodTemplateForObject in the corev1.Node case

[k8s-ci-robot]

@kgibm: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[eddiezane]

I think this is a decision that oc is making for you.

https://github.com/openshift/oc/blob/e0edc20620af770409c1530ae5361c364831544e/pkg/cli/debug/debug.go#L1016

We currently only mount the host when you debug on a node.

https://github.com/kubernetes/kubernetes/blob/e9702cfc1bce5d92fae707de3c5c57b8e952d53c/staging/src/k8s.io/kubectl/pkg/cmd/debug/profiles.go#L32

There is a plan to add further debug profiles to kubectl debug.

https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/1441-kubectl-debug/README.md#debugging-profiles

Not sure this is something we'd want to change right now by default though.

cc @soltysh @verb

[kgibm]

@eddiezane Thanks, makes sense. I agree the default doesn't need to change, but some flag like --profile=sysadmin will be fine. Thanks!

[kgibm]

Resurrecting this because I noticed that the documentation writes (emphasis added):

If none of these approaches work, you can find the Node on which the Pod is running and create a privileged Pod running in the host namespaces

However, whereas oc debug node is truly privileged, I don't think kubectl debug node is, as per my original description of not being able to inspect all processes, for example.

So it seems to me that either the documentation should be updated to remove "privileged" (or some caveats added) or the default should change to create a privileged pod.

oc debug node seems to do that with:

		zero := int64(0)
		isTrue := true
		[...]
						SecurityContext: &corev1.SecurityContext{
							Privileged: &isTrue,
							RunAsUser:  &zero,
						},

[verb]

Hi @kgibm, we plan on adding a privileged debug profile, follow along in kubernetes/kubectl#1108

We won't change the default, though, so maybe the docs should be updated. I'll send this issue over to the website repo.

/transfer website

[sftim]

The feature request is to document that Kubernetes ephemeral containers are unprivileged by default. Have I got that right?

[kgibm]

@sftim Yes, this is a documentation change request although I'm not sure what the correct term should be instead of privileged. I don't think unprivileged is right either because the pod does have privileged access to the worker node filesystem but it lacks other privileged permissions that one would normally expect root to have (such as introspecting all processes). I guess the best thing would be to ask a subject matter expert to describe what sorts of things the debug pod can do.

[sftim]

No, it's not about ephemeral containers. It's about debug pods that you can try to create with kubectl, as a way to get a form of node level access.

There aren't any docs for kubectl debug node yet; it looks like the folks doing the improvement might have missed out adding those when this subcommand got added.

[sftim]

/retitle “Debug Running Pods” gives poor advice about debugging via node shell

[kgibm]

@sftim

No, it's not about ephemeral containers. It's about debug pods that you can try to create with kubectl, as a way to get a form of node level access.

Correct. Not sure how ephemeral containers came into the picture.

There aren't any docs for kubectl debug node yet; it looks like the folks doing the improvement might have missed out adding those when this subcommand got added.

Yes, there are, as quoted earlier: https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#node-shell-session

If none of these approaches work, you can find the Node on which the Pod is running and create a privileged Pod running in the host namespaces. To create an interactive shell on a node using kubectl debug, run:

kubectl debug node/mynode -it --image=ubuntu

These are, in a sense, privileged, as the user is root and the worker node's filesystem is mounted into /host; however, they are also not fully privileged as the user does not have access to all processes (as documented earlier in the issue, and as compared to oc debug node). At least, not in the way that an average person defines as "privileged".

[sftim]

@kgibm would you be willing to revise the issue description to frame this as the improvement you would like to see?

[kgibm]

/retitle Clarify documented privileges of kubectl debug node

[k8s-ci-robot]

@kgibm: Re-titling can only be requested by trusted users, like repository collaborators.

In response to this:

/retitle Clarify documented privileges of kubectl debug node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kgibm]

@sftim I've updated the title. If you think it's better to close this issue and open a new issue that's cleaner and reference this as background, I can do that as well.

As far as what the actual clarification should be instead of the word "privileged", I'm not sure. I think a subject matter expert should opine on that. My guess is it should be something along the lines of:

If none of the previous approaches work, you can find the Node on which the Pod is running and create a debug Pod on that node. This Pod runs as root and has access to the Node's filesystem through /host; however, it is not a fully privileged pod and there are some limitations such as reading some process information. To create an interactive shell on a Node using kubectl debug, run:

[sftim]

I suggest opening a new issue, given that the original issue was a CLI feature request and not a suggestion for docs.

BTW, the description is the longer text slightly below the issue title.

[kgibm]

Opened #35170. Closing this issue.