Skip to content

Commit

Permalink
mantle/kola: add detection for a kernel soft lockup
Browse files Browse the repository at this point in the history
Recently we saw a test with many soft lockup messages like:

```
[ 4159.779792] watchdog: BUG: soft lockup - CPU#0 stuck for 883s! [kworker/u2:0:10]
[ 4159.780488] Modules linked in:
[ 4159.780787] CPU: 0 PID: 10 Comm: kworker/u2:0 Tainted: G             L    -------  ---  6.4.0-0.rc0.20230502git865fdb08197e.11.fc39.x86_64 #1
[ 4159.780787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc37 04/01/2014
[ 4159.780787] Workqueue: ftrace_check_wq ftrace_check_work_func
[ 4159.780787] RIP: 0010:get_symbol_pos+0x5d/0x140
[ 4159.780787] Code: 63 0c 8d 08 d0 6e a0 85 c9 79 0a 48 f7 d1 48 03 0d 50 c2 5a 01 48 39 cf 48 0f 42 f0 48 0f 43 d0 48 89 f0 48 29 d0 48 83 f8 01 <77> ca 48 85 d2 75 08 eb 4b 48 83 ea 01 74 45 8d 42 ff 48 98 48 63
[ 4159.780787] RSP: 0018:ffffb1970005bb80 EFLAGS: 00000202
[ 4159.780787] RAX: 0000000000000032 RBX: ffffffff9f22edb4 RCX: ffffffff9f22ed00
[ 4159.780787] RDX: 0000000000004436 RSI: 0000000000004468 RDI: ffffffff9f22edb4
[ 4159.780787] RBP: ffffb1970005bbce R08: 0000000000000000 R09: ffffb1970005bbc0
[ 4159.780787] R10: 0000000000000000 R11: 00000000000320a7 R12: 0000000000000000
[ 4159.780787] R13: 0000000000000000 R14: ffffb1970005bbc0 R15: 0000000000000000
[ 4159.780787] FS:  0000000000000000(0000) GS:ffff9fa07ec00000(0000) knlGS:0000000000000000
[ 4159.780787] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4159.780787] CR2: ffff9fa04fe01000 CR3: 000000000e022000 CR4: 0000000000350ef0
[ 4159.780787] Call Trace:
[ 4159.780787]  <TASK>
[ 4159.780787]  kallsyms_lookup_buildid+0x4d/0x130
[ 4159.780787]  test_for_valid_rec+0x64/0xb0
[ 4159.780787]  ftrace_check_work_func+0x3b/0x60
[ 4159.780787]  process_one_work+0x1c7/0x3d0
[ 4159.780787]  worker_thread+0x51/0x390
[ 4159.780787]  ? __pfx_worker_thread+0x10/0x10
[ 4159.780787]  kthread+0xf7/0x130
[ 4159.780787]  ? __pfx_kthread+0x10/0x10
[ 4159.780787]  ret_from_fork+0x2c/0x50
[ 4159.780787]  </TASK>
```

Let's try to detect and report this.
  • Loading branch information
dustymabe committed May 3, 2023
1 parent bb94c5e commit d0fc64f
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions mantle/kola/harness.go
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,10 @@ var (
desc: "kernel oops",
match: regexp.MustCompile("Oops:"),
},
{
desc: "kernel soft lockup",
match: regexp.MustCompile("watchdog: BUG: soft lockup - CPU"),
},
{
desc: "kernel warning",
match: regexp.MustCompile(`WARNING: CPU: \d+ PID: \d+ at (.+)`),
Expand Down

0 comments on commit d0fc64f

Please sign in to comment.