slabratetop_example.txt

Demonstrations of slabratetop, the Linux eBPF/bcc version.


slabratetop shows the rate of allocations and total bytes from the kernel
memory allocation caches (SLAB or SLUB), in a top-like display that refreshes.
For example:

# ./slabratetop
<screen clears>
07:01:35 loadavg: 0.38 0.21 0.12 1/342 13297

CACHE                            ALLOCS      BYTES
kmalloc-4096                       3554   14557184
kmalloc-256                        2382     609536
cred_jar                           2568     493056
anon_vma_chain                     2007     128448
anon_vma                            972      77760
sighand_cache                        24      50688
mm_struct                            49      50176
RAW                                  52      49920
proc_inode_cache                     59      38232
signal_cache                         24      26112
dentry                              135      25920
sock_inode_cache                     29      18560
files_cache                          24      16896
inode_cache                          13       7696
TCP                                   2       3840
pid                                  24       3072
sigqueue                             17       2720
ext4_inode_cache                      2       2160
buffer_head                          16       1664
xfs_trans                             5       1160

By default the screen refreshes every one second, and only the top 20 caches
are shown. These can be tuned with options: see USAGE (-h).

The output above showed that the kmalloc-4096 cache allocated the most, about
14 Mbytes during this interval. This is a generic cache; other caches have
more meaningful names ("dentry", "TCP", "pid", etc).

slabtop(1) is a similar tool that shows the current static volume and usage
of the caches. slabratetop shows the active call rates and total size of the
allocations.


Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the
kernel stacks that led to this allocation. In the future (maybe by now) the
bcc trace tool could do this. As I'm writing this, it can't, so I'll use my
older ftrace-based kprobe tool as a workarond. This is from my perf-tools
collection: https://github.com/brendangregg/perf-tools.

# ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100
Tracing kprobe kmem_cache_alloc. Ctrl-C to end.
          kprobe-3892  [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478333: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
          kprobe-3892  [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478341: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
          kprobe-3892  [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478346: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
          kprobe-3892  [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478351: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
          kprobe-3892  [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478355: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
          kprobe-3892  [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
          kprobe-3892  [002] d... 7888274.478359: <stack trace>
 => kmem_cache_alloc
 => user_path_at_empty
 => vfs_fstatat
 => SYSC_newstat
 => SyS_newstat
 => entry_SYSCALL_64_fastpath
[...]

This is just an example so that you can see it's possible to dig further.
Please don't copy-n-paste that kprobe command, as it's unlikely to work (the
"+0(+96(%di))" text is specific to a kernel version and architecture).

So these allocations are coming from user_path_at_empty(), which calls other
functions (not seen in the stack: I suspect it's a tail-call compiler
optimization).


USAGE:

# ./slabratetop -h
usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count]

Kernel SLAB/SLUB memory cache allocation rate top

positional arguments:
  interval              output interval, in seconds
  count                 number of outputs

optional arguments:
  -h, --help            show this help message and exit
  -C, --noclear         don't clear the screen
  -r MAXROWS, --maxrows MAXROWS
                        maximum rows to print, default 20

examples:
    ./slabratetop            # kmem_cache_alloc() top, 1 second refresh
    ./slabratetop -C         # don't clear the screen
    ./slabratetop 5          # 5 second summaries
    ./slabratetop 5 10       # 5 second summaries, 10 times only