C3 Kernel panic in WiFi driver

[briannewtonpsyd]

Describe the bug

This is the first time this has happened to me on either the comma 2 or comma 3, but my comma appeared to freeze/crash in the middle of a drive. I was driving with OpenPilot engaged, I first heard a "ding" from my car essentially that the Comma 3 had stopped handling driving functions/disconnected, then the Comma3 made a loud beeping noise, even though the image of my drive on the device was frozen and I had to take control manually as I was drifting out of lane. The device then rebooted. The crash happened at the end of the attached route, and then the next route (286c142afbb0252e|2022-02-03--15-58-24) was after the device rebooted.

What hardware does this issue affect?

comma three

Provide a route where the issue occurs

286c142afbb0252e|2022-02-03--15-47-21

openpilot version

0.8.12

Additional info

My car is a Honda Civic 2016 Touring

[pd0wm]

Looks like kernel panic somewhere in the wifi stack. Will look into it!

Internal sentry event for this crash: https://sentry.io/organizations/commaai/issues/2931560622/?project=6137392

Relevant logs:

<3>[  639.545372] Fatal error on the modem.
<3>[  639.545432] modem subsystem failure reason: modem_xpu_int.c:456:XPU violation End.
<6>[  639.545441] subsys-restart: subsystem_restart_dev(): Restart sequence requested for modem, restart_level = RELATED.
<6>[  639.550219] ipa-wan ipa3_ssr_notifier_cb:2692 IPA received MPSS BEFORE_SHUTDOWN
<3>[  639.554331] ipa client_lock_unlock_cb:692 Invalid EP
<3>[  639.554374] ipa client_lock_unlock_cb:692 Invalid EP
<6>[  639.554482] rmnet_ipa started deinitialization
<3>[  639.554496] ipa ipa3_teardown_sys_pipe:1113 bad parm.
<3>[  639.554519] ipa-wan ipa3_wwan_remove:2516 Failed to teardown IPA->APPS pipe
<3>[  639.554542] ipa ipa3_teardown_sys_pipe:1113 bad parm.
<3>[  639.554563] ipa-wan ipa3_wwan_remove:2521 Failed to teardown APPS->IPA pipe
<3>[  639.555265] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister rmnet_ipa0
<3>[  639.644104] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister rmnet_ipa0
<3>[  639.644311] ipa-wan ipa3_del_dflt_wan_rt_tables:413 Deleting Route hdl:(0x7) with ip type: 0
<3>[  639.646792] ipa-wan ipa3_del_dflt_wan_rt_tables:422 Deleting Route hdl:(0x9) with ip type: 1
<6>[  639.648250] rmnet_ipa completed deinitialization
<6>[  639.648700] ipa-wan ipa3_ssr_notifier_cb:2708 IPA BEFORE_SHUTDOWN handling is complete
<6>[  639.648737] icnss: PD service down, pd_state: 2, state: 0x4f8a: cause: Root PD crashed
<3>[  639.649362] icnss: PD Down while recovery inprogress, crashed: 1, state: 0x4f8a
<3>[  639.649408] icnss: ASSERT at line 2472
<4>[  639.649448] ------------[ cut here ]------------
<2>[  639.649458] Kernel BUG at ffffff94c198e894 [verbose debug info unavailable]
<4>[  639.649484] ------------[ cut here ]------------
<2>[  639.649497] Kernel BUG at ffffff94c198e894 [verbose debug info unavailable]
<0>[  639.649512] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
<4>[  639.649528] Modules linked in: wlan(CE) snd_soc_sdm845(E) snd_soc_wcd9xxx(E)
<4>[  639.649568] CPU: 1 PID: 6 Comm: kworker/u16:0 Tainted: G         C  E   4.9.103+ #26
<4>[  639.649581] Hardware name: Qualcomm Technologies, Inc. sda845 v2.1 TurboX-SOM_V01 (DT)
<4>[  639.649607] Workqueue: icnss_driver_event icnss_driver_event_work
<4>[  639.649628] task: ffffffe1b528c600 task.stack: ffffffe1b5388000
<4>[  639.649643] PC is at icnss_driver_event_work+0x131c/0x18d0
<4>[  639.649657] LR is at icnss_driver_event_work+0x143c/0x18d0
<4>[  639.649671] pc : [<ffffff94c198e894>] lr : [<ffffff94c198e9b4>] pstate: 40c00145
<4>[  639.649683] sp : ffffffe1b538bc60
<4>[  639.649696] x29: ffffffe1b538bc60 x28: ffffff94c43a6000 
<4>[  639.649727] x27: ffffff94c2b844c8 x26: ffffff94c3878000 
<4>[  639.649756] x25: ffffffe112f8a280 x24: dead000000000100 
<4>[  639.649785] x23: ffffff94c26c8870 x22: ffffffe1b2dbf818 
<4>[  639.649814] x21: ffffff94c3a74bf0 x20: ffffff94c43a67b0 
<4>[  639.649844] x19: ffffff94c2b82160 x18: 0000000000000001 
<4>[  639.649873] x17: 0000007fafad4150 x16: 00000000004f9540 
<4>[  639.649902] x15: ffffffffffffffff x14: ffffff94c3878788 
<4>[  639.649931] x13: ffffffe1b538bc21 x12: ffffffe1b538bb50 
<4>[  639.649964] x11: 000000000000098f x10: ffffffe1b538b990 
<4>[  639.649993] x9 : 0000000000003ff0 x8 : 0000000000003fff 
<4>[  639.650022] x7 : ffffffe1bdc56090 x6 : ffffffe1bdc56090 
<4>[  639.650052] x5 : ffffffe1b538ba40 x4 : ffffff94c3879000 
<4>[  639.650082] x3 : ffffff94c3879628 x2 : 054884ef9eae3e00 
<4>[  639.650113] x1 : 0000000000000000 x0 : 0000000000000000 
<4>[  639.650145] 
<4>[  639.650145] PC: 0xffffff94c198e854:
<4>[  639.650159] e854  11000421 b904f101 35000682 f94033e1 90008fb3 91058273 5280aaa2 aa1303e0
<4>[  639.650257] e874  97f1187d f9400680 b40000c0 900092e2 aa1303e1 5280aaa3 91240042 97f094e2
<4>[  639.650354] e894  d4210000 b94097e3 b0008fb6 913ee2d6 b0008c81 aa1603e0 91344021 f90033e1
<4>[  639.650453] e8b4  97f1186d f9400680 b40000c0 295213e3 900092e2 aa1603e1 91240042 97f094d2
<4>[  639.650554] 
<4>[  639.650554] LR: 0xffffff94c198e974:
<4>[  639.650569] e974  97f094a9 39598ac0 35ffa4a0 f94033e1 90008fb3 91058273 52813502 aa1303e0
<4>[  639.650677] e994  97f11835 f9400680 b4fff7c0 aa1303e1 900092e2 52813503 91240042 97f0949a
<4>[  639.650779] e9b4  17ffffb8 36ff8ac0 17ffff41 f9414f23 d0008fbb 9110437b b0008c81 aa1b03e0
<4>[  639.650876] e9d4  91344021 2a1603e2 f90033e1 97f11822 f9400680 b40000e0 f9414f24 900092e2
<4>[  639.650971] 
<4>[  639.650971] SP: 0xffffffe1b538bc20:
<4>[  639.650984] bc20  c198e9b4 ffffff94 b538bc60 ffffffe1 c198e894 ffffff94 40c00145 00000000
<4>[  639.651079] bc40  c3879000 ffffff94 ffffffff 00ffffff ffffffff ffffffff bdc56090 ffffffe1
<4>[  639.651180] bc60  b538bd70 ffffffe1 c14cb6bc ffffff94 b2dbfa50 ffffffe1 b5236500 ffffffe1
<4>[  639.651277] bc80  b15e6800 ffffffe1 00000000 00000000 b52b8400 ffffffe1 c3a635a8 ffffff94
<0>[  639.651374] Process kworker/u16:0 (pid: 6, stack limit = 0xffffffe1b5388000)
<4>[  639.651388] Call trace:
<4>[  639.651404] Exception stack(0xffffffe1b538ba60 to 0xffffffe1b538bb90)
<4>[  639.651419] ba60: ffffff94c2b82160 0000007fffffffff ffffffe1b538bc60 ffffff94c198e894
<4>[  639.651434] ba80: 0000000040c00145 000000000000003d ffffffe112f8a280 ffffffe1b538bc60
<4>[  639.651449] baa0: ffffffe1b538bc30 054884ef9eae3e00 ffffffe1b538bc60 ffffff94c198e9b4
<4>[  639.651464] bac0: ffffff94c2b82160 ffffff94c43a67b0 ffffff94c3a74bf0 ffffffe1b2dbf818
<4>[  639.651479] bae0: ffffffe1b538bc60 ffffffe1b538bc60 ffffffe1b538bc30 00000000ffffffd0
<4>[  639.651494] bb00: ffffffe1b538bc60 ffffffe1b538bc60 ffffffe1b538bc30 00000000ffffffd0
<4>[  639.651509] bb20: a477080131023102 054884ef9eae3e00 0000000000000000 0000000000000000
<4>[  639.651525] bb40: 054884ef9eae3e00 ffffff94c3879628 ffffff94c3879000 ffffffe1b538ba40
<4>[  639.651541] bb60: ffffffe1bdc56090 ffffffe1bdc56090 0000000000003fff 0000000000003ff0
<4>[  639.651556] bb80: ffffffe1b538b990 000000000000098f
<4>[  639.651571] [<ffffff94c198e894>] icnss_driver_event_work+0x131c/0x18d0
<4>[  639.651595] [<ffffff94c14cb6bc>] process_one_work+0x20c/0x4d8
<4>[  639.651611] [<ffffff94c14cb9d8>] worker_thread+0x50/0x4d0
<4>[  639.651628] [<ffffff94c14d2980>] kthread+0x100/0x108
<4>[  639.651648] [<ffffff94c1483f00>] ret_from_fork+0x10/0x50
<0>[  639.651666] Code: aa1303e1 5280aaa3 91240042 97f094e2 (d4210000) 
<4>[  639.651691] ---[ end trace 3cc23375261c2037 ]---
<1>[  639.671341] Unable to handle kernel paging request at virtual address ffffffffffffffd8
<1>[  639.671362] pgd = ffffffe19162c000
<1>[  639.671369] [ffffffffffffffd8] *pgd=0000000155820003, *pud=0000000155820003, *pmd=0000000000000000
<4>[  639.671436] ------------[ cut here ]------------
<2>[  639.671444] Kernel BUG at ffffff94c14d363c [verbose debug info unavailable]
<0>[  639.671452] Internal error: Oops: 96000005 [#2] PREEMPT SMP
<4>[  639.671459] Modules linked in: wlan(CE) snd_soc_sdm845(E) snd_soc_wcd9xxx(E)
<4>[  639.671487] CPU: 1 PID: 6 Comm: kworker/u16:0 Tainted: G      D  C  E   4.9.103+ #26
<4>[  639.671494] Hardware name: Qualcomm Technologies, Inc. sda845 v2.1 TurboX-SOM_V01 (DT)
<4>[  639.671511] task: ffffffe1b528c600 task.stack: ffffffe1b5388000
<4>[  639.671526] PC is at kthread_data+0x24/0x30
<4>[  639.671537] LR is at wq_worker_sleeping+0x20/0xd8
<4>[  639.671543] pc : [<ffffff94c14d363c>] lr : [<ffffff94c14ccc58>] pstate: 604003c5
<4>[  639.671547] sp : ffffffe1b538b870
<4>[  639.671552] x29: ffffffe1b538b870 x28: ffffffe1b5288000 
<4>[  639.671566] x27: ffffff94c2b844c8 x26: ffffff94c14e84c4 
<4>[  639.671577] x25: 0000000000000000 x24: ffffffe1b528cc30 
<4>[  639.671587] x23: ffffff94c32a2010 x22: ffffff94c3879c70 
<4>[  639.671605] x21: ffffffe1b528c600 x20: ffffff94c32ad880 
<4>[  639.671616] x19: ffffffe1b528c600 x18: 0000000000000000 
<4>[  639.671633] x17: 0000000000000007 x16: ffffffe1a915fe00 
<4>[  639.671644] x15: 000000000019c7ff x14: ffffffe1bdc5e298 
<4>[  639.671662] x13: 000000000019c7ff x12: ffffffe1b26f8588 
<4>[  639.671671] x11: 000000000000017d x10: 000000000000017d 
<4>[  639.671687] x9 : 0000000000003ff0 x8 : 00000000007348ef 
<4>[  639.671697] x7 : ffffffe1bdc56090 x6 : 000000000a7d7eb0 
<4>[  639.671713] x5 : 00ffffffffffffff x4 : 0000004cfa9b0000 
<4>[  639.671725] x3 : 0000000000000000 x2 : 0000000000000000 
<4>[  639.671740] x1 : 0000000000000000 x0 : 0000000000000000 
<4>[  639.671756] 
<4>[  639.671756] PC: 0xffffff94c14d35fc:
<4>[  639.671762] 35fc  a8c27bfd d65f03c0 d4210000 17fffffc d4210000 17fffff8 d503201f a9be7bfd
<4>[  639.671807] 361c  910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f f942e660 f9400bf3 a8c27bfd
<4>[  639.671840] 363c  f85d8000 d65f03c0 d503201f a9bd7bfd 910003fd a90153f3 aa0003f4 aa1e03e0
<4>[  639.671882] 365c  d503201f b0011d33 911e2273 f942e681 f9400260 f90017e0 d2800000 d2800102
<4>[  639.671925] 
<4>[  639.671925] LR: 0xffffff94c14ccc18:
<4>[  639.671931] cc18  d0012ca1 91386021 39409022 35fffe82 52800022 39009022 d4210000 17fffff0
<4>[  639.671971] cc38  a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f aa1303e0 94001a71
<4>[  639.672012] cc58  b9406802 52803901 6a01005f 540004c1 f9402401 d538d082 d000eea0 91004000
<4>[  639.672054] cc78  b8626800 b9400422 6b00005f 540002e1 910e0023 f9800071 885f7c60 51000400
<4>[  639.672099] 
<4>[  639.672099] SP: 0xffffffe1b538b830:
<4>[  639.672105] b830  c14ccc58 ffffff94 b538b870 ffffffe1 c14d363c ffffff94 604003c5 00000000
<4>[  639.672149] b850  b538b870 ffffffe1 c14ccc58 ffffff94 ffffffff 0000007f c263fab0 ffffff94
<4>[  639.672188] b870  b538b890 ffffffe1 c263fac8 ffffff94 bdc5d880 ffffffe1 c154cee4 ffffff94
<4>[  639.672236] b890  b538b900 ffffffe1 c14e84c4 ffffff94 b528c600 ffffffe1 b538b978 ffffffe1
<0>[  639.672275] Process kworker/u16:0 (pid: 6, stack limit = 0xffffffe1b5388000)
<4>[  639.672286] Call trace:
<4>[  639.672298] Exception stack(0xffffffe1b538b680 to 0xffffffe1b538b7b0)
<4>[  639.672306] b680: ffffffe1b528c600 0000007fffffffff ffffffe1b538b870 ffffff94c14d363c
<4>[  639.672315] b6a0: 00000000604003c5 ffffff94c2644ce4 ffffffe1bdc5d880 ffffffe1a29bf840
<4>[  639.672322] b6c0: ffffffe1ab387000 ffffff94c263fa24 ffffffe1b538b730 ffffff94c2644c64
<4>[  639.672330] b6e0: 00000000000003c0 ffffffe1bdc56db8 0000000000000001 ffffff94c32a6dc0
<4>[  639.672336] b700: ffffffe1b538b710 ffffff94c154cee4 ffffffe1b538b750 ffffff94c150b9a8
<4>[  639.672344] b720: ffffff94c32a6db8 ffffff94c3879c70 ffffffe1b538b750 054884ef9eae3e00
<4>[  639.672353] b740: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
<4>[  639.672360] b760: 0000004cfa9b0000 00ffffffffffffff 000000000a7d7eb0 ffffffe1bdc56090
<4>[  639.672368] b780: 00000000007348ef 0000000000003ff0 000000000000017d 000000000000017d
<4>[  639.672374] b7a0: ffffffe1b26f8588 000000000019c7ff
<4>[  639.672385] [<ffffff94c14d363c>] kthread_data+0x24/0x30
<4>[  639.672401] [<ffffff94c263fac8>] __schedule+0x2a8/0x728
<4>[  639.672411] [<ffffff94c14e84c4>] do_task_dead+0x74/0x78
<4>[  639.672420] [<ffffff94c14b2078>] do_exit+0x6d8/0x9b0
<4>[  639.672433] [<ffffff94c148bc78>] die+0x240/0x258
<4>[  639.672443] [<ffffff94c148bcdc>] bug_handler.part.2+0x4c/0x78
<4>[  639.672449] [<ffffff94c148bd44>] bug_handler+0x3c/0x48
<4>[  639.672459] [<ffffff94c1485204>] brk_handler+0x9c/0xd0
<4>[  639.672468] [<ffffff94c1481874>] do_debug_exception+0xa4/0x170
<4>[  639.672474] Exception stack(0xffffffe1b538ba60 to 0xffffffe1b538bb90)
<4>[  639.672481] ba60: ffffff94c2b82160 0000007fffffffff ffffffe1b538bc60 ffffff94c198e894
<4>[  639.672489] ba80: 0000000040c00145 000000000000003d ffffffe112f8a280 ffffffe1b538bc60
<4>[  639.672498] baa0: ffffffe1b538bc30 054884ef9eae3e00 ffffffe1b538bc60 ffffff94c198e9b4
<4>[  639.672504] bac0: ffffff94c2b82160 ffffff94c43a67b0 ffffff94c3a74bf0 ffffffe1b2dbf818
<4>[  639.672511] bae0: ffffffe1b538bc60 ffffffe1b538bc60 ffffffe1b538bc30 00000000ffffffd0
<4>[  639.672518] bb00: ffffffe1b538bc60 ffffffe1b538bc60 ffffffe1b538bc30 00000000ffffffd0
<4>[  639.672525] bb20: a477080131023102 054884ef9eae3e00 0000000000000000 0000000000000000
<4>[  639.672531] bb40: 054884ef9eae3e00 ffffff94c3879628 ffffff94c3879000 ffffffe1b538ba40
<4>[  639.672538] bb60: ffffffe1bdc56090 ffffffe1bdc56090 0000000000003fff 0000000000003ff0
<4>[  639.672545] bb80: ffffffe1b538b990 000000000000098f
<4>[  639.672553] [<ffffff94c1483660>] el1_dbg+0x18/0x74
<4>[  639.672560] [<ffffff94c14cb6bc>] process_one_work+0x20c/0x4d8
<4>[  639.672567] [<ffffff94c14cb9d8>] worker_thread+0x50/0x4d0
<4>[  639.672576] [<ffffff94c14d2980>] kthread+0x100/0x108
<4>[  639.672582] [<ffffff94c1483f00>] ret_from_fork+0x10/0x50
<0>[  639.672590] Code: d503201f f942e660 f9400bf3 a8c27bfd (f85d8000) 
<4>[  639.672600] ---[ end trace 3cc23375261c2038 ]---

[redacid95]

Could this have been fixed with #23701? @pd0wm

[pd0wm]

No, this is a kernel issue

[adeebshihadeh]

Haven't seen this in a while