Possible compiler bug (IDFGH-6049) #5783

olicooper · 2021-10-19T21:01:21Z

Environment

Development Kit: Heltec Wireless Stick v1.2
Module or chip used: [ESP32]
IDF version: main branch v4.4-dev-2313-gc69f0ec32 / mem-optimized branch v4.4-dev-3401-gb86fe0c66c
Build System: idf.py (PlatformIO)
Compiler version: not sure... (esp-2021r1) 8.4.0 (maybe platformio version number??) OR crosstool-ng-1.22.0-97-gc752ad5
Operating System: Windows
(Windows only) environment type: PowerShell.
Using an IDE?: PlatformIO
Power Supply: USB

Problem Description

I recently opened an issue with the MCCI Arduino LoRaWAN library here: mcci-catena/arduino-lorawan#181
After some debugging it has become apparent that there seems to be a compiler issue rather than an issue with the library.
I am unsure of the details as @terrillmoore is the library maintainer and understands the issue better than me (see this comment).

Expected Behavior

Program should run as normal (as it did in Arduino v1.0.6).

Actual Behavior

There are no build or link errors, but at runtime kernel panic occurs with a LoadStoreError.

========= BOOT INFO =========
MAC: FF:FF:FF:FF:FF:FF
BootPart: offset=0x10000 size=0x140000
IDF version: v4.4-dev-2313-gc69f0ec32
Free memory: 241704 bytes
=============================
NetBeginRegionInit
Publishing state
Guru Meditation Error: Core  1 panic'ed (LoadStoreError). Exception was unhandled.

Core  1 register dump:
PC      : 0x400d5e97  PS      : 0x00060330  A0      : 0x800d5f17  A1      : 0x3ffb25b0
A2      : 0x40013b1e  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x000028d8
A6      : 0xfffffffc  A7      : 0x00000003  A8      : 0x00000000  A9      : 0x3ffc0f94
A10     : 0x00000010  A11     : 0x00000016  A12     : 0x00000000  A13     : 0x00040df7
A14     : 0x63ffffff  A15     : 0x00000003  SAR     : 0x0000001b  EXCCAUSE: 0x00000003
EXCVADDR: 0x40013b1e  LBEG    : 0x400d5e48  LEND    : 0x400d5ea6  LCOUNT  : 0x00000004

Backtrace:0x400d5e94:0x3ffb25b00x400d5f14:0x3ffb25e0 0x400d5aeb:0x3ffb26e0 0x400d5bbe:0x3ffb2700 0x400d5c00:0x3ffb2720 0x400d1855:0x3ffb2740 0x400d2d89:0x3ffb2760 0x400d3086:0x3ffb2780 0x400d40bc:0x3ffb27a0 0x400d5d3b:0x3ffb27c0 0x400d14a9:0x3ffb27e0 0x400d15fe:0x3ffb2800 0x400d7900:0x3ffb2820
  #0  0x400d5e94:0x3ffb25b00 in Arduino_LoRaWAN::SessionChannelMask_EU_like<16u, 4u>::setFrequency(unsigned char (&) [48], unsigned int, unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/Arduino_LoRaWAN.h:369
      (inlined by) Arduino_LoRaWAN::SessionChannelMask_EU_like<16u, 4u>::setFrequency(unsigned char (&) [48], unsigned int, unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/Arduino_LoRaWAN.h:359
      (inlined by) Arduino_LoRaWAN::BuildSessionState(Arduino_LoRaWAN::SessionState_u&) const at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_sessionstate.cpp:114

Steps to reproduce

I created a repo for testing here: https://github.com/olicooper/arduino-v2-esp32-platformio/tree/60fbacf2acfa1341cf9cec9d9d86aec16b042eb6
It is a PlatformIO project that runs Arduino v2.0.0 - be sure to use commit 60fbacf2acfa1341cf9cec9d9d86aec16b042eb6 as the latest commit has a workaround for the compiler issue.

This is the fix that was applied to the library: olicooper/arduino-lorawan@8e834e7 based on this comment: mcci-catena/arduino-lorawan#181 (comment)

All the details can be found in the original ticket.

Other items if possible

sdkconfig file N/A
elf file in the build folder: firmware-202110181418.zip
coredump (This provides stacks of tasks.)

The text was updated successfully, but these errors were encountered:

igrr · 2021-10-20T08:23:27Z

@olicooper Could you please attach the ELF file of the program with/without the workaround commit?

(i am not sure if the ELF file you have attached has the workaround or not)

olicooper · 2021-10-20T09:03:51Z

That firmware was built on before the patch was applied so it should exhibit the problem I describe. The workaround includes a function called setFrequencyRaw which isn't present in this firmware.
Thanks for taking a look 👍

olicooper · 2021-10-20T10:09:50Z

So you have all the information here is a new elf file for Arduino v1.0.6 and v2.0.0 and associated logs. The builds are based on commit 60fbacf2acfa1341cf9cec9d9d86aec16b042eb6

Arduino v2.0.0

Build log

PLATFORM: Espressif 32 (3.3.1+sha.3784198) > Heltec Wireless Stick
HARDWARE: ESP32 240MHz, 320KB RAM, 8MB Flash
PACKAGES:
- framework-arduinoespressif32 0.0.0+sha.44c1198
- tool-esptoolpy 1.30100.210531 (3.1.0)
- toolchain-xtensa-esp32 8.4.0+2021r1
- Dependency Graph
|-- <MCCI Arduino LoRaWAN Library> 0.9.1
|   |-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |   |-- <SPI> 2.0.0
|   |-- <MCCI Arduino Development Kit ADK> 0.2.2+sha.97a5943
|-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |-- <SPI> 2.0.0
|-- <SPI> 2.0.0
RAM:   [          ]   4.3% (used 14140 bytes from 327680 bytes)
Flash: [==        ]  20.3% (used 266037 bytes from 1310720 bytes)

Debug log

========= BOOT INFO =========
BootPart: offset=0x10000 size=0x140000
IDF version: v4.4-dev-2313-gc69f0ec32
Free memory: 241484 bytes
=============================
NetBeginRegionInit
Publishing state
Guru Meditation Error: Core  1 panic'ed (LoadStoreError). Exception was unhandled.

Core  1 register dump:
PC      : 0x400d5e97  PS      : 0x00060b30  A0      : 0x800d5f17  A1      : 0x3ffb25b0  
A2      : 0x40013b1e  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x00002aa9
A6      : 0xfffffffc  A7      : 0x00000003  A8      : 0x00000000  A9      : 0x3ffc0f94  
A10     : 0x00000010  A11     : 0x00000016  A12     : 0x00000000  A13     : 0x00040df7
A14     : 0x63ffffff  A15     : 0x00000003  SAR     : 0x0000001b  EXCCAUSE: 0x00000003  
EXCVADDR: 0x40013b1e  LBEG    : 0x400d5e48  LEND    : 0x400d5ea6  LCOUNT  : 0x00000004


Backtrace:0x400d5e94:0x3ffb25b00x400d5f14:0x3ffb25e0 0x400d5aeb:0x3ffb26e0 0x400d5bbe:0x3ffb2700 0x400d5c00:0x3ffb2720 0x400d1855:0x3ffb2740 0x400d2d89:0x3ffb2760 0x400d3086:0x3ffb2780 0x400d40bc:0x3ffb27a0 0x400d5d3b:0x3ffb27c0 0x400d14a9:0x3ffb27e0 0x400d15fe:0x3ffb2800 0x400d7900:0x3ffb2820

Exception Cause: Not found

0x400d5e94: Arduino_LoRaWAN::SessionChannelMask_EU_like<16u, 4u>::setFrequency(unsigned char (&) [48], unsigned int, unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/Arduino_LoRaWAN.h:369
(inlined by) Arduino_LoRaWAN::SessionChannelMask_EU_like<16u, 4u>::setFrequency(unsigned char (&) [48], unsigned int, unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/Arduino_LoRaWAN.h:359
(inlined by) Arduino_LoRaWAN::BuildSessionState(Arduino_LoRaWAN::SessionState_u&) const at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_sessionstate.cpp:114
0x400d5f14: Arduino_LoRaWAN::SaveSessionState() at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_sessionstate.cpp:190  
0x400d5aeb: Arduino_LoRaWAN::UpdateFCntDown(unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/Arduino_LoRaWAN.h:925
(inlined by) Arduino_LoRaWAN::StandardEventProcessor(unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_begin.cpp:214
0x400d5bbe: Arduino_LoRaWAN::DispatchEvent(unsigned int) at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_begin.cpp:1230x400d5c00: onEvent at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_begin.cpp:109
0x400d1855: reportEventNoUpdate at .pio/libdeps/heltec_wireless_stick/MCCI LoRaWAN LMIC library/src/lmic/lmic.c:481
0x400d2d89: reportEventAndUpdate at .pio/libdeps/heltec_wireless_stick/MCCI LoRaWAN LMIC library/src/lmic/lmic.c:466
0x400d3086: startJoining at .pio/libdeps/heltec_wireless_stick/MCCI LoRaWAN LMIC library/src/lmic/lmic.c:2139
0x400d40bc: os_runloop_once at .pio/libdeps/heltec_wireless_stick/MCCI LoRaWAN LMIC library/src/lmic/oslmic.c:157
0x400d5d3b: Arduino_LoRaWAN::loop() at .pio/libdeps/heltec_wireless_stick/MCCI Arduino LoRaWAN Library/src/lib/arduino_lorawan_loop.cpp:42
0x400d14a9: Lora::loop() at src/lora.cpp:58
0x400d15fe: loop() at src/main.cpp:43
0x400d7900: loopTask(void*) at C:/Users/Oliver/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:46

Elf file:
firmware-202110201103.zip

Arduino v1.0.6

Build log

PLATFORM: Espressif 32 (3.3.2+sha.1e0f147) > Heltec Wireless Stick
HARDWARE: ESP32 240MHz, 320KB RAM, 8MB Flash
PACKAGES:
- framework-arduinoespressif32 0.0.0+sha.46d5afb
- tool-esptoolpy 1.30100.210531 (3.1.0)
- toolchain-xtensa32 2.50200.97 (5.2.0)
Dependency Graph
|-- <MCCI Arduino LoRaWAN Library> 0.9.1
|   |-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |   |-- <SPI> 1.0
|   |-- <MCCI Arduino Development Kit ADK> 0.2.2+sha.97a5943
|-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |-- <SPI> 1.0
|-- <SPI> 1.0
RAM:   [          ]   4.8% (used 15580 bytes from 327680 bytes)
Flash: [=         ]   7.2% (used 239726 bytes from 3342336 bytes)

Debug log

========= BOOT INFO =========
BootPart: offset=0x10000 size=0x330000
IDF version: v3.3.5-1-g85c43024c
Free memory: 288160 bytes
=============================
NetBeginRegionInit
Publishing state
NetBeginRegionInit

Elf file: firmware-v106-202110201123.zip

VojtechBartoska · 2022-03-30T15:27:43Z

Hello @olicooper, adding this to roadmap and we will investigate this more.

VojtechBartoska · 2022-04-08T08:42:02Z

Any chance you can validate this again on v2.0.3-RC1? thanks!

VojtechBartoska · 2022-04-22T08:53:12Z

@olicooper Any chance you have tested this? Thanks

olicooper · 2022-05-03T13:48:14Z

My test environment uses PlatformIO, so I have validated against Arduino v2.0.2 as this is the latest release available. Is there something specific in v2.0.3-RC1 that will address this issue?

The kernel panic does not seem to be occurring anymore (using a 'Lolin 32 Lite' MCU) but I get an assert issue now. The assert can be found here

Build output:

PLATFORM: Espressif 32 (4.2.0) > WEMOS LOLIN32 Lite
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
...
PACKAGES:
 - framework-arduinoespressif32 3.20002.220503 (2.0.2)
 - tool-esptoolpy 1.30300.0 (3.3.0)
 - toolchain-xtensa-esp32 8.4.0+2021r2-patch3
Dependency Graph
|-- <MCCI Arduino LoRaWAN Library> 0.9.1
|   |-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |   |-- <SPI> 2.0.0
|   |-- <MCCI Arduino Development Kit ADK> 0.2.2+sha.97a5943
|-- <MCCI LoRaWAN LMIC library> 4.1.0+sha.26326e4
|   |-- <SPI> 2.0.0
|-- <SPI> 2.0.0
...
RAM:   [=         ]   5.7% (used 18536 bytes from 327680 bytes)
Flash: [==        ]  20.0% (used 261777 bytes from 1310720 bytes)

Debug output:

========= BOOT INFO =========

MAC: 00:00:00:00:00:00
BootPart: offset=0x10000 size=0x140000
IDF version: v4.4-beta1-189-ga79dc75f0a
Free memory: 269896 bytes

=============================

FAILURE
.pio/libdeps/lolin32_lite/MCCI LoRaWAN LMIC library/src/lmic/radio.c:1164

Compiling for Arduino v1.0.6 it works fine and I get the following debug output:

========= BOOT INFO =========

MAC: 00:00:00:00:00:00
BootPart: offset=0x10000 size=0x140000
IDF version: v3.3.5-1-g85c43024c
Free memory: 288364 bytes

=============================

NetBeginRegionInit
Publishing state
NetBeginRegionInit
EV_TXSTART: Ch=1

I don't have the 'Heltec Wireless Stick' to test with anymore, so I have used a 'Lolin 32 Lite' with an external RFM95W lora module instead. Without the exact hardware I cannot be sure that the kernel panic issue has been resolved but the kernel panic occurs on the 'Lolin 32 Lite' when using the original arduino-upstream platform so I assume it is fixed now?

This is the lorawan-test repo that I am using.

VojtechBartoska · 2022-05-18T09:29:50Z

Issue added to our Roadmap to be tested. Thanks for providing more info.

PilnyTomas · 2022-08-05T11:50:31Z

Hi @olicooper, I'm having trouble running the project under Platform IO (I never used it before).
Can you please create a project for Arduino IDE?

PilnyTomas · 2022-08-25T12:49:48Z

Closing as expired

github-actions bot changed the title ~~Possible compiler bug~~ Possible compiler bug (IDFGH-6049) Oct 19, 2021

o-marshmallow self-assigned this Oct 20, 2021

wujiangang transferred this issue from espressif/esp-idf Oct 20, 2021

o-marshmallow removed their assignment Mar 24, 2022

VojtechBartoska added Status: Awaiting triage Issue is waiting for triage Type: Bug 🐛 All bugs labels Mar 30, 2022

VojtechBartoska added this to Arduino ESP32 Core Project Roadmap Mar 30, 2022

VojtechBartoska moved this to Todo in Arduino ESP32 Core Project Roadmap Mar 30, 2022

VojtechBartoska added the Resolution: Awaiting response Waiting for response of author label Apr 8, 2022

olicooper mentioned this issue May 3, 2022

Kernel panic on Arduino v2.0 mcci-catena/arduino-lorawan#181

Open

VojtechBartoska added Status: Test needed Issue needs testing and removed Status: Awaiting triage Issue is waiting for triage Resolution: Awaiting response Waiting for response of author labels May 18, 2022

VojtechBartoska assigned PilnyTomas Jun 29, 2022

VojtechBartoska moved this from Todo to Under investigation in Arduino ESP32 Core Project Roadmap Aug 8, 2022

PilnyTomas closed this as completed Aug 25, 2022

Repository owner moved this from Under investigation to Done in Arduino ESP32 Core Project Roadmap Aug 25, 2022

VojtechBartoska added Resolution: Expired More info wasn't provided and removed Status: Test needed Issue needs testing labels Aug 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possible compiler bug (IDFGH-6049) #5783

Possible compiler bug (IDFGH-6049) #5783

olicooper commented Oct 19, 2021

igrr commented Oct 20, 2021 •

edited

Loading

olicooper commented Oct 20, 2021

olicooper commented Oct 20, 2021 •

edited

Loading

Build log

Debug log

Build log

Debug log

VojtechBartoska commented Mar 30, 2022

VojtechBartoska commented Apr 8, 2022

VojtechBartoska commented Apr 22, 2022

olicooper commented May 3, 2022 •

edited

Loading

VojtechBartoska commented May 18, 2022

PilnyTomas commented Aug 5, 2022

PilnyTomas commented Aug 25, 2022

Possible compiler bug (IDFGH-6049) #5783

Possible compiler bug (IDFGH-6049) #5783

Comments

olicooper commented Oct 19, 2021

Environment

Problem Description

Expected Behavior

Actual Behavior

Steps to reproduce

Other items if possible

igrr commented Oct 20, 2021 • edited Loading

olicooper commented Oct 20, 2021

olicooper commented Oct 20, 2021 • edited Loading

Build log

Debug log

Build log

Debug log

VojtechBartoska commented Mar 30, 2022

VojtechBartoska commented Apr 8, 2022

VojtechBartoska commented Apr 22, 2022

olicooper commented May 3, 2022 • edited Loading

VojtechBartoska commented May 18, 2022

PilnyTomas commented Aug 5, 2022

PilnyTomas commented Aug 25, 2022

igrr commented Oct 20, 2021 •

edited

Loading

olicooper commented Oct 20, 2021 •

edited

Loading

olicooper commented May 3, 2022 •

edited

Loading