Skip to content

Debugging and troubleshooting

Jump to bottom Edit New page
Alexander Krizhanovsky edited this page May 13, 2017 · 30 revisions

If you're developing or debugging Tempesta's code (thanks for that by the way!), then you need to make several steps to simplify the process or even save your VM in a cloud. Tempesta FW is a Linux kernel extension, so the below are typical steps helping in Linux kernel development and debugging.

Automatic reboot to safe kernel

If a kernel crash occurs, then the system may hang, so it has a sense to setup automatic reboot. Normally your Linux should reboot automatically on kernel panic. You can check it by

    # cat /proc/sys/kernel/panic
    1

1 means 1 second before reboot on system panic. You can access and set the setting by sysctl kernel.panic. Next, you can emulate the panic by

    # echo c > /proc/sysrq-trigger

and see that the system reboots in 10 seconds. One more important setting is sysctl kernel.panic_on_oops (you can find it also in /proc/sys/kernel/panic_on_oops). Usually it's set to 1, i.e. reboot on any kernel fault ("Oops") occurred. Setting this isn't necessary, but you may prefer to use it.

Next, you need to reboot to testing kernel but make the system reboot automatically to safe kernel if the first one occasionally crashes. You can do this using

    GRUB_DEFAULT=saved
    GRUB_CMDLINE_LINUX_DEFAULT="panic=1"

in /etc/default/grub. The settings allow you to set a kernel as safe, i.e. booted by default, and add the kernel parameter to reboot automatically in 1 second after panic. The kernel parameter automatically sets kernel.panic sysctl. To apply the changes run

    # grub2-mkconfig -o /boot/grub2/grub.cfg

for CentOS or RHEL and

   # update-grub

for Debian or Ubuntu. Now let's list all installed kernels in the system (for GRUB2 in CentOS/RHEL 7.x):

    # grep '^menuentry' /boot/grub2/grub.cfg 
    menuentry 'CentOS Linux (4.8.15-tfw) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.8.15-tfw-advanced-3b2c9f44-81ab-4581-b94a-9c95754a3fd8' {
    menuentry 'CentOS Linux (4.1.27+) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.1.27+-advanced-3b2c9f44-81ab-4581-b94a-9c95754a3fd8' {
    menuentry 'CentOS Linux (3.10.0-123.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-123.el7.x86_64-advanced-3b2c9f44-81ab-4581-b94a-9c95754a3fd8' {
    menuentry 'CentOS Linux (0-rescue-c582b0f7976e47fb9beaa84df9af1a3a) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-c582b0f7976e47fb9beaa84df9af1a3a-advanced-3b2c9f44-81ab-4581-b94a-9c95754a3fd8' {

In Debian or Ubuntu you'll see something like:

    # grep 'menuentry\>' /boot/grub/grub.cfg
    menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-cfda0544-9803-41e7-badb-43563085ff3a' {
        menuentry 'Ubuntu, with Linux 4.8.15+' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.8.15+-advanced-cfda0544-9803-41e7-badb-43563085ff3a' {
    	menuentry 'Ubuntu, with Linux 4.8.15+ (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.8.15+-recovery-cfda0544-9803-41e7-badb-43563085ff3a' {
        menuentry 'Ubuntu, with Linux 4.4.0-75-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-75-generic-advanced-cfda0544-9803-41e7-badb-43563085ff3a' {
    	menuentry 'Ubuntu, with Linux 4.4.0-75-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-75-generic-recovery-cfda0544-9803-41e7-badb-43563085ff3a' {
        menuentry 'Ubuntu, with Linux 4.4.0-45-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-45-generic-advanced-cfda0544-9803-41e7-badb-43563085ff3a' {
    	menuentry 'Ubuntu, with Linux 4.4.0-45-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-45-generic-recovery-cfda0544-9803-41e7-badb-43563085ff3a' {

I.e. the new kernel is listed in submenu. However, we still can use the same tools to set default and next booted kernels as for RHEL-based distros. We can choose the first kernel as testing and the second one as safe:

    # grub2-set-default 'CentOS Linux (4.1.27+) 7 (Core)'
    # grub2-reboot 'CentOS Linux (4.8.15-tfw) 7 (Core)'

After the system reboot it will boot into 4.8.15-tfw kernel, but if it fails it will automatically reboot to 4.1.27+ kernel. In Debian-based distros you can do the same:

   # grub-set-default 'Ubuntu, with Linux 4.4.0-75-generic'
   # grub-reboot 'Ubuntu, with Linux 4.8.15+'

Crash dumps

It might be useful to setup automatic storing kernel crash dumps. Please explore these links for setting it up:

Add a custom footer

Contents

Clone this wiki locally