st-util: GDB breakpoints do not break at appropriate location

[heliochronix]

• Programmer/board type: NUCLEO-F091RC [stlink-v2-1]
• Programmer firmware version: V2J37M26
• Operating system and version: Linux
• Stlink tools version: v1.6.1
• Stlink commandline tool name: st-util
• Target chip (and board if applicable): STM32F091RC

Description:
When attempting to break at a particular point in the code, it seems a GDB session will instead break at a completely different location. The location of the break is consistent, but it is not the correct location. When using OpenOCD as the debugging bridge, the break occurs in the correct location.

Commandline-Output:

user@hostname:~
➜ st-util
st-util
2020-07-31T14:17:18 INFO common.c: F09X: 32 KiB SRAM, 256 KiB flash in at least 2 KiB pages.
2020-07-31T14:17:18 INFO gdb-server.c: Listening at *:4242...

user@hostname:oresat-firmware/src/f0/app_template on  master [⇡!]
➜ arm-none-eabi-gdb build/app_template.elf
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build/app_template.elf...
(gdb) tar ext localhost:4242
Remote debugging using localhost:4242
CO_NMT_process (NMT=0x20000988 <COO_CANmodule_txArray0+192>, timeDifference_us=<optimized out>, HBtime_ms=<optimized out>, NMTstartup=134218129, errorRegister=147 '\223', errorBehavior=0x8000193 <VectorBC> <incomplete sequence \360>, timerNext_us=0x8000193 <VectorBC>)
    at ../../../common/CANopenNode/301/CO_NMT_Heartbeat.c:232
232         if (NMT->operatingState == CO_NMT_INITIALIZING) {
(gdb) load
Loading section .vectors, size 0xc0 lma 0x8000000
Loading section .text, size 0x7088 lma 0x80000c0
Loading section .rodata, size 0xb24 lma 0x8007148
Loading section .ARM.exidx, size 0x8 lma 0x8007c6c
Loading section .data, size 0x2b0 lma 0x8007c74
Start address 0x08000190, load size 32548
Transfer rate: 15 KB/sec, 5424 bytes/write.
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (Remote target) killed]
(gdb) b CO_CANsend
Breakpoint 1 at 0x8001aa0: file ../../../common/CO_driver.c, line 230.
(gdb) run
Starting program: /home/user/Projects/PSAS/oresat-firmware/src/f0/app_template/build/app_template.elf
Note: automatically using hardware breakpoints for read-only addresses.

Program received signal SIGTRAP, Trace/breakpoint trap.
CO_NMT_process (NMT=0x20000988 <COO_CANmodule_txArray0+192>, timeDifference_us=<optimized out>, HBtime_ms=<optimized out>, NMTstartup=134218129, errorRegister=147 '\223', errorBehavior=0x8000193 <VectorBC> <incomplete sequence \360>, timerNext_us=0x8000193 <VectorBC>)
    at ../../../common/CANopenNode/301/CO_NMT_Heartbeat.c:232
232         if (NMT->operatingState == CO_NMT_INITIALIZING) {
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
CO_NMT_process (NMT=0x20000988 <COO_CANmodule_txArray0+192>, timeDifference_us=<optimized out>, HBtime_ms=<optimized out>, NMTstartup=134218129, errorRegister=147 '\223', errorBehavior=0x8000193 <VectorBC> <incomplete sequence \360>, timerNext_us=0x8000193 <VectorBC>)
    at ../../../common/CANopenNode/301/CO_NMT_Heartbeat.c:232
232         if (NMT->operatingState == CO_NMT_INITIALIZING) {
(gdb)

Expected/description:

It is expected that the debug session breaks at the appropriate line in the appropriate file, which is at address 0x8001aa0. However, it does not break here, and instead it always seems to break at completely different locations.

It almost seems as if the addresses reported back are off in some way, because (for example) OpenOCD reports that it was last idling at address 0x08002462, which is where the system waits for an interrupt. However, with st-util, it always seems to think it is idling on a movs instruction in the middle of a thread starting function.

An example of equivalent expected output from OpenOCD is below:

user@hostname:oresat-firmware/src/f0/app_template on  master [⇡!]
➜ make gdb
arm-none-eabi-gdb -q /home/user/Projects/PSAS/oresat-firmware/src/f0/app_template/./build/app_template.elf -cd ../../../boards/ST_NUCLEO64_F091RC -x gdboocd.cmd
Reading symbols from /home/user/Projects/PSAS/oresat-firmware/src/f0/app_template/./build/app_template.elf...
Open On-Chip Debugger 0.10.0+dev-01365-g583a65644-dirty (2020-07-30-21:27)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : STLINK V2J37M26 (API v2) VID:PID 0483:3752
Info : Target voltage: 3.254104
Info : stm32f0x.cpu: hardware has 4 breakpoints, 2 watchpoints
Info : starting gdb server for stm32f0x.cpu on pipe
Info : accepting 'gdb' connection from pipe
target halted due to debug-request, current mode: Thread
xPSR: 0x41000000 pc: 0x08002462 psp: 0x20001ae8
Info : device id = 0x10006442
Info : flash size = 256kbytes
Warn : Prefer GDB command "target extended-remote pipe" instead of "target remote pipe"
0x08002462 in port_wait_for_interrupt () at ../../../ChibiOS/os/common/ports/ARMCMx/chcore_v6m.h:458
458   __WFI();
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Unable to match requested speed 1000 kHz, using 950 kHz
target halted due to debug-request, current mode: Thread
xPSR: 0xc1000000 pc: 0x08000190 msp: 0x20000200
target halted due to debug-request, current mode: Thread
xPSR: 0xc1000000 pc: 0x08000190 msp: 0x20000200
(gdb) b CO_CANsend
Breakpoint 1 at 0x8001aa0: file ../../../common/CO_driver.c, line 230.
(gdb) c
Continuing.
Note: automatically using hardware breakpoints for read-only addresses.
Error: No symbols for ChibiOS

Breakpoint 1, CO_CANsend (CANmodule=0x20000758 <COO_CANmodule>, buffer=0x20000988 <COO_CANmodule_txArray0+192>) at ../../../common/CO_driver.c:230
230 {
(gdb)

The GDB script used for the OpenOCD session is as follows:

target remote | openocd -f oocd.cfg -c "gdb_port pipe"
monitor stm32f0x.cpu configure -rtos chibios
monitor reset halt

[simbazor]

Hi,
I had a similar problem after installing version 1.6.1 from the deb package provided here, on a Debian 10 system.
Using a nucleo board to program another board with a STM32L475 processor, after setting up a break point, it would either stop in another place or wouldn't brake at all.
Reverting to version 1.5.1 from Debian repositories solved the problem.

[stappersg]

On Tue, Aug 04, 2020 at 07:58:08PM -0700, simbazor wrote: similar problem with version 1.6.1 ... Reverting to version 1.5.1 good

Please `git bisect` it

[daylanKifky]

Same problem with:

Programmer/board type: NUCLEO-G031K8 [stlink-v2-1]
Programmer firmware version: V2J34M25
Operating system and version: Linux
Stlink tools version: v1.6.1
Stlink commandline tool name: st-util
Target chip (and board if applicable): STM32G031K8T6

Cannot test with lower versions of st-utils because not compatibles with STM32G031xx (or am I wrong? --> #856)
As in @heliochronix's case, the behaviour is correct when using openOCD

[cmdrf]

I seem to have the same problem with an STM32F767. v1.5.1 shipped with Debian works fine, while my own v1.6.1 build doesn't (current master can't even flash, probably some other problem). I will try and bisect this, but I'm off with a bad start as v1.5.1 won't build because of some CMake problem in set_target_properties...

[Nightwalker-87]

@cmdrf: Use the v1.5.1-patch branch if you want to compile v1.5.1. This branch remains for testing purposes and fixes a compilation issue in the original v1.5.1. Please note that we don't offer any support for older versions of this toolset. Detected bugs and issues may solely be fixed in an upcoming release.

[cmdrf]

Cool, thanks! That successfully compiles. But I'm still seeing the issue there, while v1.5.1 from Debian still works in the exact same setup 😕 . Maybe I'm chasing a red herring here. I'm doing this on an ARM machine right now. Trying something else next.

EDIT: I'm just stupid. Running make inside the CMake build directory doesn't update the binaries under bin. That seems to be something that the top-level Makefile does.

[cmdrf]

After figuring out that the st-util binary changes location inside the build directory in some revision, I got a result from bisecting:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
09ea99a
0a91936
We cannot bisect more!

[Nightwalker-87]

Thx. I'll have a look at it soon.

[cmdrf]

I fiddled around a bit and got 09ea99a to compile. This is indeed the first commit where the problem appeared. Since it is related to the STLINK version, I tried the --stlinkv1 option, but it didn't change anything.

I have a clone STLINKv2 (0483:3748) updated with an official firmware at some point.

[Nightwalker-87]

@cmdrf: Can you checkout the current develop branch and go to /src/st-util/gdb-server.c Line 1387 and replace
TARGET_HALTED with STLINK_CORE_HALTED to see if that fixes this issue? I am not sure yet if this is a good solution to this problem, but I could find out that both of these preprocessing directives are used simultaneously with the latter being the older one. Maybe we need some proper refactoring here.

[cmdrf]

For this I had to switch from my IDE to hand crank gdb, because my IDE always wants to flash before running and flashing does not work on develop. This changes the picture a bit:

st-util + IDE (Qt Creator with Bare Metal plugin):

• v1.5.1: breaks at correct location
• v1.6.1: breaks at wrong location
• develop: can't test
• develop + STLINK_CORE_HALTED: can't test

st-util + command line gdb:

• v1.5.1: breaks at correct location
• v1.6.1: doesn't break at all. After Ctrl-C, gdb points to a broken stack.
• develop: breaks at correct location!
• develop + STLINK_CORE_HALTED: doesn't break at all, but when I pause with Ctrl-C, gdb points to the correct breakpoint location.

So my conclusion would be: The issue was silently fixed in develop. The STLINK_CORE_HALTED fix makes things slightly worse.

[Nightwalker-87]

Interesting. I need some time to dig into this.

[Ant-ON]

@Nightwalker-87 Changing TARGET_HALTED to STLINK_CORE_HALTED is break the code. I analyzed the code and see that the core_stat cannot be STLINK_CORE_HALTED value.

I think this problem have been fixed by #1027.

[Nightwalker-87]

@heliochronix @cmdrf: Can you verify this with ba5c679 and 5c03678 ?

[cmdrf]

• ba5c679: breaks at the correct location
• 5c03678: does not break

[Nightwalker-87]

So @Ant-ON 's educated guess appears to be right and we can close this issue as resolved. 👍
@arsv Thx again for fixing this! 🥇

[stappersg]

On Sat, Oct 24, 2020 at 11:13:36AM -0700, Fabian wrote: For this I had to switch from my IDE to hand crank gdb, because my IDE always wants to flash before running and flashing does not work on `develop`.

Please name (and shame) the IDE

[cmdrf]

I did name it, and in the meantime I found out how to disable this behavior.

And I really don't want to crash the party, but my STM32H750 never stops at any breakpoints :). Yes, I did try it without the IDE. But flashing works so I'll resort to printf debugging for now.

[Ant-ON]

@cmdrf Need look at the update_code_breakpoint function in st-util/gdb-server.c. Maybe H7 requires a fixes same as F7.

[stappersg]

Preamble: My previous comment in this issue was only based upon one comment email.

Rereading through whole #1011 did not reveal a name of something that I can link to an IDE.

This comment is for inviting @cmdrf to share expriences with IDEs and IDE tweaking.

Regarding the

my STM32H750 never stops at any breakpoints :)

is my advice to explore documentation and build a working configuration.

[Ant-ON]

@cmdrf try replace if (sl->core_id == STM32F7_CORE_ID) to if (sl->core_id == STM32F7_CORE_ID || sl->core_id == 0x6ba02477) in st-util/gdb-server.c

[Nightwalker-87]

Please don't recycle this topic for anything else than a potential additional fix.

[cmdrf]

@Ant-ON Thanks for the suggestion! Just to clarify: do you mean to replace all occurences (six, including those where the logic is negated), or just specific ones? I tried different combinations, but all it does is breaking the ability to flash. Still no breakpointing. gdb did point to a breakpoint when I Ctrl-C'd once, but I cannot reproduce it.

@Nightwalker-87 Shall we open a new issue specific to breakpoints on STM32H7?

[Ant-ON]

@cmdrf replace only in update_code_breakpoint function in st-util/gdb-server.c

Yes, I think need create new issue. It is another bug.

[Nightwalker-87]

@cmdrf @Ant-ON I agree. - Let's open a new ticket for that and finally close this one.