py-spy 0.1.9 fails when started inside docker container

[iljau]

When py-spy is installed and launched from inside docker container, py-spy==0.1.9 fails, whereas previous version py-spy==0.1.8 worked (SYS_PTRACE capability is set for container).

$ py-spy --version
py-spy 0.1.9

$ RUST_BACKTRACE=1 py-spy -- python3 test.py 

Error: EPERM: Operation not permitted
stack backtrace:
   0:           0x4e7bac - backtrace::backtrace::trace::ha0fbe6521a297b97
   1:           0x4e6682 - backtrace::capture::Backtrace::new_unresolved::ha618912f49ed2d80
   2:           0x4e602f - failure::backtrace::internal::InternalBacktrace::new::h33dcc523c9982708
   3:           0x4e62d1 - <failure::backtrace::Backtrace as core::default::Default>::default::h530f3770400a1ba8
   4:           0x4e6328 - failure::backtrace::Backtrace::new::hfc945c6756648d04
   5:           0x444e30 - py_spy::process::os_impl::Namespace::new::h2c3c0252db472d26
   6:           0x41d75f - py_spy::python_spy::PythonSpy::new::hfbd77dacf6afa4b9
   7:           0x421508 - py_spy::python_spy::PythonSpy::retry_new::hc6dd70655afd2779
   8:           0x41bb35 - py_spy::main::h54ad4b843dae6e20
   9:           0x4416c2 - std::rt::lang_start::{{closure}}::h1e5e01061341fa09
  10:           0x5f1b72 - std::rt::lang_start_internal::{{closure}}::h388a7d34c6d3c7af
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::he4497885895661a8
                        at libstd/panicking.rs:310
  11:           0x6159e9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:103
  12:           0x6012e6 - std::panicking::try::hcf37798695a9b029
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::hdd9654d0e8ee1176
                        at libstd/panic.rs:392
                         - std::rt::lang_start_internal::h4ec5eae1ccf41136
                        at libstd/rt.rs:58
  13:           0x41c533 - main

# test.py
while True:
    pass

[benfred]

thanks for the bug report! I've managed to reproduce this - it's caused by some code that was added so that you can profile from the host os into processes running in a docker container. I will have a fix out shortly

[benfred]

fixed in v0.1.10

[lovepocky]

root@dd77dc76b-4ck4h:/opt/app# RUST_BACKTRACE=1 py-spy -p 1
Error: Failed to suspend process
Reason: EPERM: Operation not permitted
stack backtrace:
   0:           0x4eda5d - backtrace::backtrace::trace::h72eb519ea1c30979
   1:           0x4ec4b2 - backtrace::capture::Backtrace::new_unresolved::hcca2671986d02778
   2:           0x4ebe21 - failure::backtrace::internal::InternalBacktrace::new::h8aa243afed4a675f
   3:           0x4ec0d1 - <failure::backtrace::Backtrace as core::default::Default>::default::h9ee3359cb47a172e
   4:           0x4ec129 - failure::backtrace::Backtrace::new::h33aed36859968218
   5:           0x42648b - py_spy::process::os_impl::Lock::new::h19299266bf895b52
   6:           0x40b9f6 - py_spy::python_spy::PythonSpy::get_stack_traces::h562232d9b147cf02
   7:           0x40b627 - py_spy::python_spy::PythonSpy::retry_new::h9464b08b0438a692
   8:           0x44660d - py_spy::main::h2f1f6f64c8aa4e65
   9:           0x422b72 - std::rt::lang_start::{{closure}}::h479c2785e776f3b2
  10:           0x611952 - std::rt::lang_start_internal::{{closure}}::h6b5f7ae7c14f8493
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::he4712eb84191fb3b
                        at libstd/panicking.rs:310
  11:           0x621169 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  12:           0x60412b - std::panicking::try::hd44c69d8635b9942
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::h4424b54e4de01dca
                        at libstd/panic.rs:392
                         - std::rt::lang_start_internal::h76f990c2b2c74932
                        at libstd/rt.rs:58
  13:           0x4482c4 - main
root@dd77dc76b-4ck4h:/opt/app# py-spy --version
py-spy 0.1.10

same issue in kubernetes pod
@benfred

[benfred]

@lovepocky did you launch your container with the SYS_PTRACE capability enabled? https://github.com/benfred/py-spy/#how-do-i-run-py-spy-in-kubernetes

If you didn't launch with the SYS_PTRACE capability, you can still trace the process from the host os

[lovepocky]

I have added capabilities to k8s containers.

root@6ccb596f9d-4lqjd:/opt/app# capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend+eipBounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspendSecurebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked)uid=0(root)gid=0(root)groups=

and get cap_sys_ptrace in container

[CtheSky]

Same here. With cap_sys_ptrace added to k8s containers as documented here. py-spy==0.1.8 works well, but 0.1.9, 0.1.10, 0.1.11 give the same error.