Perf Recording

Basic steps to get Perf Reports

1. Install Perf (make sure perf version matches kernel version)
2. Install debug symbols for frr (for RHEL, frr-debuginfo)
3. Run perf top --call-graph=dwarf -p <pid> to get a generic overview of the system. Screenshots of this are useful
4. Slightly more advanced usage: perf record -g --call-graph=dwarf -- /usr/lib/frr/bgpd -A 127.0.0.1 then run vtysh -b, and wait for whatever event. Kill -9 bgpd after you'd like to stop recording
5. If a routing daemon is already running you can tell perf to record a running process perf record -g --call-graph=dwarf -p <pid>
6. run perf with -K to hide kernel symbols so we can focus on bgpd, ospfd, or zebra daemon.

Some related interesting tools/techniques can be found at: http://gernotklingler.com/blog/gprof-valgrind-gperftools-evaluation-tools-application-level-cpu-profiling-linux/ and http://www.thegeekstuff.com/2012/08/gprof-tutorial

Flamegraphs

A good way to analyze (and share the data) is to convert the perf output to a Flamegraph. See https://github.com/brendangregg/FlameGraph for details on flamegraph.

Basic usage:

1. Do a perf record as outlined above (don't forget the -g)
2. Convert the perf data and create graph:
   1. perf script > out.perf
   2. ./stackcollapse-perf.pl out.perf > out.folded
   3. ./flamegraph.pl out.folded > bgpd.svg

Looking at flamegraph:

The svg file is a standalone file and can be best inspected with a webbrowser which supports svg (and the encoded scripts within the svg). Please note, that hovering the mouse over a block shows the full name (if only partially readable) and clicking on a block zooms into the specific function.