Trouble in debugging in k8s pod with limited privilege #159
Replies: 2 comments 6 replies
-
Hi @rickyzhang82, did you try to use the core file feature instead of trying to work with a running process? |
Beta Was this translation helpful? Give feedback.
-
That's exactly what I did! I used gcore in GDB to create core dump files and then replicate the same binary environment as the target pod by extending the new Dockerfile from the base image of the target pod. One thing I want to mention that there is python-gdb.py source file in the cpython source code. Adding it to I didn't know how pystack implements. I used Thank you! |
Beta Was this translation helpful? Give feedback.
-
I'm using PyStack to debug deadlock issue in Python code. But I found it hard to use PyStack in the following scenario:
The target Python process is inside a k8s pod (let's call it the target pod). The target pod didn't run as root user, removed apt-get, pip, made the root FS read-only. So I can't launch PyStack from the k8s pod at the runtime.
I used the tool kubectl-exec to attach a separate pod with the root privilege to the k8s node. But I can't remote attach PyStack from the separate pod to the target Python process because the Python binary in the target pod is not accessible to the separate pod.
I can use GDB to remote attach to the target Python process from the separate pod. But the stack frame from PVM is not quite helpful. Because PVM is compiled with the optimization flag enabled. Most the parameters are optimized out. There are two hundred levels deep stack frame. I'm not familiar with PVM source code. I found it very hard to use GDB to debug Python as well.
Any suggestion to use PyStack in this k8s env to make my debug life easier?
Beta Was this translation helpful? Give feedback.
All reactions