eBPF

Introduction

Sysdig now supports (in beta) the ability to run the fully featured system event capture engine using eBPF as instrumentation backend as opposed to the traditional kernel module. During the setup phase, an "eBPF probe" is compiled based on the currently running kernel. The eBPF probe is an ELF object that contains a series of eBPF programs that are loaded inside the kernel using the bpf() system call, and subsequently attached to relevant tracepoints, such as system call events, context switches, page faults, ...

Warning: the feature is currently in beta, and the steps described below could change at any time.

Requirements

The requirements are:

• Kernel version >= 4.14 (it might be possible to relax this up to 4.12, they have been excluded since they are not longterm kernels, please get in touch if you feel otherwise)
• Kernel with eBPF support enabled, in particular CONFIG_BPF=y, CONFIG_BPF_JIT=y and CONFIG_BPF_SYSCALL=y
• Kernel headers must be installed and available (this might be again relaxed in the future)
• If not using the sysdig container, a recent working Clang + LLVM setup (recommended version is >= 7, but older versions might work as well)
• x86-64 target platform, no other platforms are supported at the moment

Running sysdig + eBPF

Container mode (recommended)

This is by far the easiest and most recommended way to run sysdig with eBPF support. The command line is:

docker run -it --name sysdig --privileged --net=host -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro -e SYSDIG_BPF_PROBE="" sysdig/sysdig

The specific changes from the traditional container installation are:

• -e SYSDIG_BPF_PROBE="": needed to instruct the docker entry point to setup the eBPF probe instead of the standard dkms kernel module builder/loader
• --net=host: needed to enable eBPF JIT at runtime for performance reasons. Can be skipped if eBPF JIT is enabled from outside the container (e.g. via /proc/sys/net/core/bpf_jit_enable)
• -v /etc:/host/etc:ro: needed to correctly detect the kernel version for the eBPF probe on Google Container OS (COS). Can be skipped if not running on Google COS

The output should be similar to this:

gianluca@sid:~$ docker run -it --name sysdig --privileged --net=host -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro -e SYSDIG_BPF_PROBE="" sysdig/sysdig
* Setting up /usr/src links from host
* Mounting debugfs
Found kernel config at /host/boot/config-4.17.0-rc2+
* Trying to compile BPF probe sysdig-probe-bpf (sysdig-probe-bpf-0.1.1dev-x86_64-4.17.0-rc2+-cbc1061ff198a7ea70b3c2d98f94ca76.o)
* BPF probe located, it's now possible to start sysdig

root@sid:/# sysdig 
3 15:28:19.157661456 3 <NA> (0) > switch next=3253(cat) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
5 15:28:19.157699229 3 cat (3253) > write fd=1(<f>/dev/pts/8) size=73 
7 15:28:19.157739756 3 cat (3253) > read fd=3(<f>/sys/kernel/debug/tracing/trace_pipe) size=131072 
...

Native installation

If installing sysdig through a traditional tarball/deb/rpm package, you need to have Clang + LLVM in your path. Then, you can either run sysdig using the --bpf or -B command line parameters, or set the environment variable SYSDIG_BPF_PROBE (tip: it can be persisted in your shell initialization script):

gianluca@sid:~$ sudo sysdig -B
* Mounting debugfs
Found kernel config at /boot/config-4.15.0-1006-gcp
* Trying to compile BPF probe sysdig-probe-bpf (sysdig-probe-bpf-0.1.1dev-x86_64-4.15.0-1006-gcp-610846bd0fce75bc94cba8639ca7347b.o)
* BPF probe located, it's now possible to start sysdig

16 17:31:24.809317422 0 sysdig (18052) > switch next=0 pgft_maj=0 pgft_min=1039 vm_size=75220 vm_rss=9192 vm_swap=0 
...

This compiles the eBPF probe for the currently running kernel by leveraging the sysdig-probe-loader script, and puts it in ~/.sysdig/.

The probe will be automatically recompiled after each sysdig or kernel update.

Development

The overall workflow from here is still largely applicable. libelf-dev is a new requirement. The main difference is that to compile the eBPF probe you need to explicitly enable the BUILD_BPF option in cmake:

mkdir build
cd build
cmake -DBUILD_BPF=ON ..
make

Among the various steps, this also runs a "bpf" compilation step shown as below, which triggers Clang + LLVM to create the probe.o eBPF probe ELF object that is then loaded inside the kernel:

gianluca@sid:~/build$ make
[  5%] Built target b64
[ 10%] Built target openssl
...
[100%] Built target bpf
...

gianluca@sid:~/build$ file driver/bpf/probe.o 
driver/bpf/probe.o: ELF 64-bit LSB relocatable, *unknown arch 0xf7* version 1 (SYSV), with debug_info, not stripped

The probe.o can then be passed to the just built sysdig using a combination of -B and --bpf command line parameters, or the environment variable SYSDIG_BPF_PROBE. For example:

gianluca@sid:~/build$ sudo ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o
17 15:43:19.240898250 0 sysdig (5669) > switch next=0 pgft_maj=0 pgft_min=1249 vm_size=178628 vm_rss=11104 vm_swap=0
18 15:43:19.246693116 3 <NA> (0) > switch next=9 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
...

gianluca@sid:~/build$ sudo ./userspace/sysdig/sysdig -Bdriver/bpf/probe.o
17 15:43:26.292088538 0 sysdig (5678) > switch next=0 pgft_maj=0 pgft_min=1252 vm_size=178628 vm_rss=11160 vm_swap=0
18 15:43:26.295238906 3 <NA> (0) > switch next=9 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
...

gianluca@sid:~/build$ sudo SYSDIG_BPF_PROBE="driver/bpf/probe.o" ./userspace/sysdig/sysdig
17 15:43:45.511666007 0 sysdig (5686) > switch next=0 pgft_maj=0 pgft_min=1252 vm_size=178628 vm_rss=11076 vm_swap=0
18 15:43:45.514424330 3 <NA> (0) > switch next=9 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
...