[BUG] Zephyr baseline update blocked

[kv2019i]

Describe the bug
Updating SOF main to Zephyr main shows issues in SOF CI tests, blocking merging.

Last good #8797
First failed attempt: #8804 (Zephyr zephyrproject-rtos/zephyr@6a0b1da )
Latest failed attempt: #8813

To Reproduce
Update Zephyr version in SOF west.yml and submit PR to SOF CI.

Reproduction Rate
100%

Expected behavior
SOF test suite passed with latest Zephyr.

Impact
Blocks updates of Zephyr in SOF main.

Environment
See above

Screenshots or console output

Notes
Discovery of this was delayed by an earlier regression spotted in #8747 and later fixed on SOF side (by #8790 ).

Bisect is slowed down by interface change decsribed in zephyrproject-rtos/zephyr#64755

[kv2019i]

Fixed done on Zephyr side:

• Revert "pm: Remove CURRENT_CPU macro" zephyrproject-rtos/zephyr#68234 (UPDATE: replaced by intel_adsp: Fix interruption locks in power management code zephyrproject-rtos/zephyr#67863 )
• soc: xtensa: intel_adsp: cavs: fix XCC build zephyrproject-rtos/zephyr#68108
• ace: fix DSP panic during startup zephyrproject-rtos/zephyr#67617

[kv2019i]

Status as of 31th Jan, we have two blocking issues:

• failure to power down non-primary core, example log showing SET_DX IPC failure -> https://sof-ci.01.org/sofpr/PR8813/build2413/devicetest/index.html?model=MTLP_RVP_NOCODEC&testcase=check-pause-resume-playback-100
• failure in chain-dma stop on Intel cAVS2.5 platforms, SOF-side workaround exists 386a525 , but tracked as [BUG] multiple-pause-resume.sh sof-test fails to FW_GEN_MSG failed with err 3018 #8792

[kv2019i]

The issues with SMP interface change are still unresolved, but I took a quick peek at latest Zephyr 3.6.0-rc2 and we have more problems:

• -fno-lto added -> xtensa toolchain fails
• various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI
• the cache interface change (to sys_cache) -> requires SOF side changes

[marc-hb]

I'll look at the toolchain issues.

[marc-hb]

-fno-lto added -> xtensa toolchain fails

Fix submitted:

• cmake/compiler/gcc: use -flto/-fno-lto only for gcc >= 4.5.0 zephyrproject-rtos/zephyr#69058

[marc-hb]

various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI

Bisected to big commit zephyrproject-rtos/zephyr@25173f71cda630d4fb (zephyrproject-rtos/zephyr#67424)

Déjà vu?

• Revert "init: add missing initialization of dev pointer in SYS_INIT (alternative to PR68125) zephyrproject-rtos/zephyr#68118

[marc-hb]

various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI

Fix submitted:

• pm: use braces for .pm_base inside anonymous union initializer zephyrproject-rtos/zephyr#69270

[kv2019i]

@LaurentiuM1234 @dbaluta Any preference how we handle this for SOF2.9? We are still struggling with the SMP interface change on Intel platforms (merged in Zephyr upstream), but the problems only affect multicore configurations. The 2.9 branch point is next week, so we have two options to proceed:
a) fix all the remain issues and update to 3.6-rc for SOF2.9
b) branch SOF2.9 with the current version of Zephyr used by SOF and merge new Zephyr baseline ASAP after stable-v2.9 branch is created

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

FYI @marc-hb @lgirdwood .

[LaurentiuM1234]

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

as far as the NXP native switch is concerned there shouldn't be an issue with option (b). Also, as far as I'm aware the only changes to Zephyr (that can impact SOF) we've had lately are all native-related. @dbaluta and @iuliana-prodan can correct me if I'm wrong here.

[lgirdwood]

a) fix all the remain issues and update to 3.6-rc for SOF2.9 b) branch SOF2.9 with the current version of Zephyr used by SOF and merge new Zephyr baseline ASAP after stable-v2.9 branch is created

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

Lets try a) and extend code freeze by 1 week. If too many problems then b) will be backup.

[kv2019i]

#8732 is getting close, but there are still issues (plus 8732 uses unmerged commits from Zephyr). So let's go with option (b) for SOF2.9. We can backport Zephyr upstream fixes to the stable branch used by SOF.

[kv2019i]

Test results look good now #8903 . After code reviews, we can merge this.

We do need one revert, filed following bug in Zephyr to track zephyrproject-rtos/zephyr#69807

[kv2019i]

Baseline now updated via #8903 so we can close this. I filed a new bug #8908 to track the need for the revert.