Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2023.04 release bug tracking #19469

Open
MrKevinWeiss opened this issue Apr 14, 2023 · 6 comments
Open

2023.04 release bug tracking #19469

MrKevinWeiss opened this issue Apr 14, 2023 · 6 comments
Labels
Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Type: tracking The issue tracks and organizes the sub-tasks of a larger effort

Comments

@MrKevinWeiss
Copy link
Contributor

Description

  • compile_and_test_for_board.py found the following:
    • BOARD=all
    • BOARD=arduino-mega2650
      • tests/congure_quic
      • tests/congure_test
      • tests/pbkdf2
      • tests/phydat_unix
      • tests/pkg_littlefs2
      • tests/pkg_mbedtls
      • tests/sys_sema_inv
    • BOARD=b-l475e-iot01a
      • tests/ztimer_overhead
    • BOARD=nrf52840dk
      • tests/periph_timer_periodic
    • BOARD=samr34-xpro
      • tests/dbgpin
      • tests/mtd_raw
      • tests/periph_rtt

Steps to reproduce the issue

actions results
@MrKevinWeiss MrKevinWeiss added Type: tracking The issue tracks and organizes the sub-tasks of a larger effort Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) labels Apr 14, 2023
@dylad
Copy link
Member

dylad commented Apr 14, 2023 •
edited
Loading

For samr34-xpro,

  • tests/dbgpin hangs because of the test default pin (port 0, pin0) -> PA00. On SAML21-based MCU, this pin is used as the input for 32kHz crystal which is use internally to feed the PLL which also supply the core. So we simply need to change the default pin.
  • tests/mtd_raw MTD initialization seems to fail with errno -5 (Input/Output). Don't know why yet.
  • tests/periph_rtt Tried on iotlab, cannot reproduce. (I don't own one unfortunately)

@MrKevinWeiss
Copy link
Contributor Author

MrKevinWeiss commented Apr 26, 2023 •
edited
Loading

Local tests and iotlab results (conflicts take the more successful results) of RC3 result in:

Failure Summary

  • frdm-kw41z: ❌ FAIL
    • tests/mtd_flashpage
    • tests/periph_rtc
    • tests/periph_timer
  • nucleo-l452re: ❌ FAIL
    • tests/pkg_libschc
  • esp32-wroom-32: ✅ PASS
  • saml10-xpro: ❌ FAIL
    • tests/bench_ztimer
    • tests/congure_quic
    • tests/event_periodic_callback
    • tests/event_wait_timeout_ztimer
    • tests/evtimer_mbox
    • tests/evtimer_msg
    • tests/evtimer_underflow
    • tests/gnrc_ipv6_fwd_w_sub
    • tests/gnrc_ipv6_nib
    • tests/gnrc_ipv6_nib_6ln
    • tests/gnrc_lorawan
    • tests/gnrc_lorawan_11
    • tests/gnrc_ndp
    • tests/gnrc_netif_ipv6_wait_for_global_address
    • tests/gnrc_rpl_p2p
    • tests/gnrc_sixlowpan_frag
    • tests/gnrc_sixlowpan_iphc_w_vrb
    • tests/gnrc_sock_async_event
    • tests/gnrc_sock_ip
    • tests/gnrc_sock_neterr
    • tests/gnrc_sock_udp
    • tests/lwip_sock_ip
    • tests/lwip_sock_udp
    • tests/netdev_test
    • tests/netutils
    • tests/periph_rtc
    • tests/periph_rtt
    • tests/periph_rtt_min
    • tests/pkg_libschc
    • tests/pkg_mbedtls
    • tests/pkg_umorse
    • tests/posix_sleep
    • tests/ps_schedstatistics
    • tests/pthread_cooperation
    • tests/shell_lock
    • tests/sys_sched_round_robin
    • tests/trickle
    • tests/ztimer_periodic
    • tests/ztimer_xsec
  • nrf52840dk: ✅ PASS
  • dwm1001: ❌ FAIL
    • tests/periph_timer_periodic
  • remote-revb: ❌ FAIL
    • examples/micropython
    • tests/congure_test
    • tests/mtd_raw
    • tests/pkg_mbedtls
    • tests/unittests
    • tests/pkg_libfixmath
    • tests/pthread_cooperation
  • arduino-mega2560: ❌ FAIL
    • tests/congure_quic
    • tests/congure_test
    • tests/pbkdf2
    • tests/phydat_unix
    • tests/pkg_littlefs2
    • tests/pkg_mbedtls
    • tests/sys_sema_inv
    • tests/ztimer_overhead
  • samr34-xpro: ❌ FAIL
    • examples/micropython
    • tests/dbgpin
    • tests/mtd_raw
    • tests/periph_rtt
  • b-l475e-iot01a: ❌ FAIL
    • tests/cpp_ctors
    • tests/cpp_ext
    • tests/deferred_yield_higher
    • tests/pkg_libschc
    • tests/ztimer_overhead
  • nucleo-f429zi: ✅ PASS
  • nucleo-wl55jc: ❌ FAIL
    • tests/pkg_libschc
  • nrf52dk: ❌ FAIL
    • tests/pkg_yxml
  • iotlab-m3: ❌ FAIL
    • tests/pkg_libschc
    • tests/stdin
    • tests/struct_tm_utility
    • tests/suit_manifest
  • samr21-xpro: ✅ PASS
  • b-l072z-lrwan1: ❌ FAIL
    • tests/pkg_libschc

@benpicco
Copy link
Contributor

How did those fail? Many of those tests are completely hardware independent, do you have the error logs?

@MrKevinWeiss
Copy link
Contributor Author

I suppose I should share like @maribu
I will be improving this a bit in the future to make it easier to share/combine/present/repeat runs.

@benpicco
Copy link
Contributor

benpicco commented Apr 26, 2023 •
edited
Loading

Just took a quick look, but b-l475e-iot01a:tests/cpp_ctors looks more like a network error

@MrKevinWeiss
Copy link
Contributor Author

The local tests I reran with --incremental but the iotlabs ones are just run once, so I would assume the flakey tests would show more.

@MrKevinWeiss MrKevinWeiss mentioned this issue May 4, 2023
Merged
bors bot added a commit that referenced this issue Oct 27, 2023
@bors @MrKevinWeiss
@benpicco @krzysztof-cabaj
Merge #19546 #20022 #20025
86bab88 
19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss

### Contribution description

The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards.  Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c
```
and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out.  Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`)

I tried to do as much as I could with make and only alter the python script when needed.

### Testing procedure

Clear results folder and run:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

it should execute the test...

Then the same command again should skip it.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

...finally, running without the changes check should run the test as usual, even if nothing changes:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

### Issues/PRs references

Might help the surprises we got with #19469


20022: pkg/lwip: add support for slipdev r=benpicco a=benpicco



20025: tests/drivers/at: fix device table overflow r=benpicco a=krzysztof-cabaj

### Contribution description

This PR fix device table overflow in `tests/driver/at`, which could lead to device crash.

### Testing procedure

PR was tested on two nucleo boards with 2 and 3 UARTs (nucleo-l476rg and nucleo-l496zg).
Flash `tests/driver/at` with and without this PR.

Output with this PR:

```
> main(): This is RIOT! (Version: 2022.07-devel-5083-g2b9e8-tests-drivers-at)
AT command test app
> init 5 9600

Wrong UART device number - should by in range 0-2.
>
```

Output without this PR:

```
> main(): This is RIOT! (Version: 2022.07-devel-5083-g2b9e8)
AT command test app
> init 5 9600

8001afd
*** RIOT kernel panic:
FAILED ASSERTION.

*** halted.


Context before hardfault:
   r0: 0x0000000a
   r1: 0x00000000
   . . . 
```

### Issues/PRs references

None

Co-authored-by: MrKevinWeiss <weiss.kevin604@gmail.com>
Co-authored-by: Benjamin Valentin <benjamin.valentin@ml-pa.com>
Co-authored-by: krzysztof-cabaj <kcabaj@gmail.com>
bors bot added a commit that referenced this issue Oct 27, 2023
@bors @MrKevinWeiss
Merge #19546
fcf7644 
19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss

### Contribution description

The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards.  Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c
```
and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out.  Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`)

I tried to do as much as I could with make and only alter the python script when needed.

### Testing procedure

Clear results folder and run:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

it should execute the test...

Then the same command again should skip it.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

...finally, running without the changes check should run the test as usual, even if nothing changes:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

### Issues/PRs references

Might help the surprises we got with #19469


Co-authored-by: MrKevinWeiss <weiss.kevin604@gmail.com>
bors bot added a commit that referenced this issue Nov 1, 2023
@bors @MrKevinWeiss
Merge #19546
652767b 
19546: Enable compile_and_test_for_board to skip if nothing changed r=MrKevinWeiss a=MrKevinWeiss

### Contribution description

The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards.  Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c
```
and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out.  Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`)

I tried to do as much as I could with make and only alter the python script when needed.

### Testing procedure

Clear results folder and run:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

it should execute the test...

Then the same command again should skip it.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

...finally, running without the changes check should run the test as usual, even if nothing changes:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

### Issues/PRs references

Might help the surprises we got with #19469


Co-authored-by: MrKevinWeiss <weiss.kevin604@gmail.com>
bors bot added a commit that referenced this issue Nov 4, 2023
@bors @MrKevinWeiss @gschorcht
Merge #19546 #19944
49075b6 
19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss

### Contribution description

The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards.  Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c
```
and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out.  Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`)

I tried to do as much as I could with make and only alter the python script when needed.

### Testing procedure

Clear results folder and run:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

it should execute the test...

Then the same command again should skip it.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

...finally, running without the changes check should run the test as usual, even if nothing changes:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

### Issues/PRs references

Might help the surprises we got with #19469


19944: cpu/esp32: add low-level LCD parallel interface using LCD peripheral r=benpicco a=gschorcht

### Contribution description

This PR the implementation of the LCD low-level MCU 8080 parallel interface using the LCD peripheral for ESP32, ESP32-S2 and ESP32-S3 or `periph_gpio_ll` for the ESP32-C3.

### Testing procedure

```
BOARD=esp32s3-wt32-sc01-plus make -C tests/drivers/st77xx flash
```
should work on top of PR #19941. Drawing operations should be much faster.

### Issues/PRs references

Co-authored-by: MrKevinWeiss <weiss.kevin604@gmail.com>
Co-authored-by: Gunar Schorcht <gunar@schorcht.net>
bors bot added a commit that referenced this issue Nov 4, 2023
@bors @MrKevinWeiss
Merge #19546
f17dc3d 
19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss

### Contribution description

The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards.  Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c
```
and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out.  Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`)

I tried to do as much as I could with make and only alter the python script when needed.

### Testing procedure

Clear results folder and run:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

it should execute the test...

Then the same command again should skip it.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again.
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

...finally, running without the changes check should run the test as usual, even if nothing changes:
```
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell
```
<details>

```
INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful
```
</details>

### Issues/PRs references

Might help the surprises we got with #19469


Co-authored-by: MrKevinWeiss <weiss.kevin604@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Type: tracking The issue tracks and organizes the sub-tasks of a larger effort
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benpicco @MrKevinWeiss @dylad