UBLOX_ODIN_EVK_W2 crashing in Wifi tests after tickless change

[kimlep01]

Description

UBLOX_EVK_ODIN_W2 started to crash last night in CI when running netsocket/network tests with Wifi. Happens with all compilers.

Commit ID #11480 :

kimlep01@e109827-lin:~/dev/git_repos/mbed-os-example-wifi/mbed-os$ git log -1
commit f513bce4a3c712dc724851735ab85c32c214c61f (HEAD -> master, origin/master, origin/HEAD)
Merge: daafb95 f1a8b04
Author: Martin Kojtal <martin.kojtal@arm.com>
Date:   Mon Sep 23 11:42:58 2019 +0200

    Merge pull request #11480 from gpsimenos/gp-enable-tickless-stage2
    
    Enable tickless mode in additional tested targets

GCC_ARM crash message:

1569313321.84][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-SECURE-FAIL}}, queued...
[1569313321.90][CONN][RXD] >>> Running case #1: 'WIFI-CONSTRUCTOR'...
[1569313321.94][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONSTRUCTOR}}, queued...
[1569313322.43][CONN][RXD]
[1569313322.43][CONN][RXD]
[1569313322.46][CONN][RXD] ++ MbedOS Error Info ++
[1569313322.50][CONN][RXD] Error Status: 0x80010133 Code: 307 Module: 1
[1569313322.58][CONN][RXD] Error Message: Mutex: 0x2000C504, Not allowed in ISR context
[1569313322.60][CONN][RXD] Location: 0x8022EC9
[1569313322.62][CONN][RXD] Error Value: 0x2000C504
[1569313322.73][CONN][RXD] Current Thread: main Id: 0x20005A78 Entry: 0x8022DF7 StackSize: 0x1000 StackMem: 0x2000B4D0 SP: 0x2002FE50
[1569313322.82][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80010133&tgt=UBLOX_EVK_ODIN_W2
[1569313322.84][CONN][RXD] -- MbedOS Error Info --
[1569313322.84][CONN][RXD]
[1569313322.90][CONN][RXD] = System will be rebooted due to a fatal error =
q^Cmbedgt: ctrl+c keyboard interrupt!

ARMC6 crash message:

[1569314034.30][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-SECURE-FAIL}}, queued...
[1569314034.34][CONN][RXD] >>> Running case #1: 'WIFI-CONSTRUCTOR'...
[1569314034.38][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONSTRUCTOR}}, queued...
[1569314034.88][CONN][RXD]
[1569314034.92][CONN][RXD] ++ MbedOS Fault Handler ++
[1569314034.92][CONN][RXD]
[1569314034.94][CONN][RXD] FaultType: HardFault
[1569314034.94][CONN][RXD]
[1569314034.95][CONN][RXD] Context:
[1569314034.97][CONN][RXD] R0   : 001E8480
[1569314034.98][CONN][RXD] R1   : 20003C7C
[1569314035.00][CONN][RXD] R2   : 00000000
[1569314035.01][CONN][RXD] R3   : 08124C01
[1569314035.03][CONN][RXD] R4   : 00000000
[1569314035.05][CONN][RXD] R5   : 20003CC8
[1569314035.06][CONN][RXD] R6   : 20003E2C
[1569314035.08][CONN][RXD] R7   : 20003FF4
[1569314035.09][CONN][RXD] R8   : 20004194
[1569314035.11][CONN][RXD] R9   : 00000000
[1569314035.12][CONN][RXD] R10  : 20003DE4
[1569314035.14][CONN][RXD] R11  : 00000000
[1569314035.17][CONN][RXD] R12  : 00000000
[1569314035.18][CONN][RXD] SP   : 20006030
[1569314035.20][CONN][RXD] LR   : 0801A059
[1569314035.22][CONN][RXD] PC   : 0800AC70
[1569314035.23][CONN][RXD] xPSR : 61000000
[1569314035.25][CONN][RXD] PSP  : 20005FC8
[1569314035.26][CONN][RXD] MSP  : 2002FFD8
[1569314035.28][CONN][RXD] CPUID: 410FC241
[1569314035.30][CONN][RXD] HFSR : 80000000
[1569314035.31][CONN][RXD] MMFSR: 00000000
[1569314035.33][CONN][RXD] BFSR : 00000000
[1569314035.34][CONN][RXD] UFSR : 00000000
[1569314035.36][CONN][RXD] DFSR : 0000000A
[1569314035.37][CONN][RXD] AFSR : 00000000
[1569314035.39][CONN][RXD] Mode : Thread
[1569314035.42][CONN][RXD] Priv : Privileged
[1569314035.43][CONN][RXD] Stack: PSP
[1569314035.43][CONN][RXD]
[1569314035.46][CONN][RXD] -- MbedOS Fault Handler --
[1569314035.46][CONN][RXD]
[1569314035.46][CONN][RXD]
[1569314035.46][CONN][RXD]
[1569314035.48][CONN][RXD] ++ MbedOS Error Info ++
[1569314035.53][CONN][RXD] Error Status: 0x80FF013D Code: 317 Module: 255
[1569314035.56][CONN][RXD] Error Message: Fault exception
[1569314035.58][CONN][RXD] Location: 0x800AC70
[1569314035.61][CONN][RXD] Error Value: 0x200001B0
[1569314035.72][CONN][RXD] Current Thread: main Id: 0x2000A600 Entry: 0x8039175 StackSize: 0x1000 StackMem: 0x20005140 SP: 0x20006030
[1569314035.81][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=UBLOX_EVK_ODIN_W2
[1569314035.84][CONN][RXD] -- MbedOS Error Info --
[1569314035.84][CONN][RXD]
[1569314035.89][CONN][RXD] = System will be rebooted due to a fatal error =

IAR crash message:

[1569314274.70][CONN][RXD]
[1569314274.70][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-SECURE-FAIL}}, queued...
[1569314274.75][CONN][RXD] >>> Running case #1: 'WIFI-CONSTRUCTOR'...
[1569314274.79][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONSTRUCTOR}}, queued...
[1569314275.29][CONN][RXD]
[1569314275.29][CONN][RXD]
[1569314275.32][CONN][RXD] ++ MbedOS Error Info ++
[1569314275.36][CONN][RXD] Error Status: 0x80010133 Code: 307 Module: 1
[1569314275.42][CONN][RXD] Error Message: Mutex: 0x2000BA1C, Not allowed in ISR context
[1569314275.44][CONN][RXD] Location: 0x0
[1569314275.46][CONN][RXD] Error Value: 0x2000BA1C
[1569314275.58][CONN][RXD] Current Thread: main Id: 0x2000C018 Entry: 0x80EDA81 StackSize: 0x1000 StackMem: 0x20005368 SP: 0x2000CCC0
[1569314275.67][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80010133&tgt=UBLOX_EVK_ODIN_W2
[1569314275.69][CONN][RXD] -- MbedOS Error Info --
[1569314275.69][CONN][RXD]
[1569314275.74][CONN][RXD] = System will be rebooted due to a fatal error =

Steps to reproduce:
git checkout f513bce (latest master)
mbed test --compile -m UBLOX_EVK_ODIN_W2 -t GCC_ARM --app-config=tools/test_configs/mbed_app.json.txt -n tests-network-wifi
mbedgt -V

Use attached mbed_app.json.txt to compile network tests (no need to have valid Wifi credentials)
mbed_app.json.txt

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[Tharazi97]

@maciejbocianski

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1812

[0xc0170]

Enabling tickless is not always straight forward.

cc @ARMmbed/team-ublox

[SeppoTakalo]

This is not testcase related issue. I just noticed that Ublox ODIN W2 hardfaults when trying to run our WiFi example when compiled against master.

WiFi example
Mbed OS version 99.99.99


++ MbedOS Fault Handler ++

FaultType: HardFault

Context:
...

[0xc0170]

@gpsimenos Please review this issue, we shall revert the change for ODIN target at least.

[gpsimenos]

I agree, let's revert it until the problem is identified and fixed. This crash didn't appear in the standard greentea tests.

[jamesbeyond]

The TICKLESS mode got turned off temporarily.
However issue will need to @ARMmbed/team-ublox help to instigate, as plan to enable the TICKLESS mode by default.

[michalpasztamobica]

Could this ticket please be assigned to someone from @ARMmbed/team-ublox ?

[0xc0170]

@ARMmbed/team-ublox Please review

[MarceloSalazar]

@ARMmbed/team-st-mcd can you please have a look and suggest how this could be possibly fixed?

[jeromecoutant]

commit from #11480 has been reverted by #11625

[0xc0170]

commit from #11480 has been reverted by #11625

Correct, to fix an issue on master. However there is still an issue as tickless should work for this target.

[jeromecoutant]

However there is still an issue as tickless should work for this target.

No... Tickless is not targeted for STM32F4 family.

@bulislaw @MarceloSalazar
@LMESTM

I think you can close this issue.

[LMESTM]

Correct, to fix an issue on master. However there is still an issue as tickless should work for this target.

We (ST) have not activated Tickless for F4 based targets. We do not recommend to use it for F4 STM32 targets as we have never been able to get "green light" - not all TESTS run OK.
See commit message here:
6397a61

Since then, and following more investigations (like #11682 ) we still believe that TICKLESS should not be activated on F4 devices. We should rather look at future low power support based on RTC alarm wake-up where deep sleep would be entered for longer periods of time. that would be less dynamic than tickless but we still expect very good low power savings .

[0xc0170]

Closing as not supported.