[BUG]IPC error when testing multiple-pipeline on MTL-RVP NOCODEC #8686

RanderWang · 2024-01-03T06:29:57Z

Kernel log:

[ 2924.250651] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-mtl 0000:00:1f.3: trigger cmd: 0 state: 2
[ 2924.261128] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx reply: 0x33000000|0x1: GLB_SET_PIPELINE_STATE
[ 2924.261147] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx done : 0x13000003|0x1: GLB_SET_PIPELINE_STATE [data size: 12]
[ 2924.261155] kernel: Message payload: 00000000: 00000002 00000002 00000003
[ 2924.261183] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx      : 0x13000002|0x1: GLB_SET_PIPELINE_STATE [data size: 12]
[ 2924.261543] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx reply: 0x330000a4|0x1: GLB_SET_PIPELINE_STATE
[ 2924.261547] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: FW reported error: 164 - Unknown
[ 2924.261626] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc error for msg 0x13000002|0x1
[ 2924.261675] kernel: Message payload: 00000000: 00000002 0000000a 00000009
[ 2924.261680] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: failed to set final state 2 for all pipelines

FW log:

[ 2924.260746] <inf> ipc: ipc_cmd: rx   : 0x13000002|0x1^M
[ 2924.260775] <inf> pipe: pipeline_trigger: pipe:10 0x0 pipe trigger cmd 0^M
[ 2924.260783] <wrn> pm_device: runtime_suspend: Unbalanced suspend^M
[ 2924.260788] <err> host_comp: host_common_trigger: comp:10 0xd0004 host_trigger(): dma stop failed: -120^M
[ 2924.260791] <err> pipe: pipeline_trigger_run: pipe:10 0x0 pipeline_trigger_run(): ret = -120, host->comp.id = 851972, cmd = 0^M
[ 2924.260803] <err> ipc: ipc4_pipeline_trigger: pipeline 10: trigger cmd 0 failed with: -120^M
[ 2924.260806] <err> ipc: ipc_compound_post_start: failed to process msg 19 status 164^M

The text was updated successfully, but these errors were encountered:

According to hardware spec, host dma needs some delay to complete stop. In the bug the host dma is disabled twice in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

RanderWang · 2024-01-03T08:10:00Z

solution:

drivers: dma: intel-adsp-hda: add delay to stop host dma zephyrproject-rtos/zephyr#67135

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled twice in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

marc-hb · 2024-01-12T01:07:33Z

Similar looking issue #8729 with a lot more details.

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

RanderWang · 2024-01-15T08:00:27Z

fixed

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. (cherry picked from commit e021ccf) Original-BugLink: thesofproject/sof#8686 Original-Signed-off-by: Rander Wang <rander.wang@intel.com> GitOrigin-RevId: e021ccf Change-Id: Ie878ee785583845e771da22a71ea09f278ed5c73 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/zephyr/+/5196152 Tested-by: ChromeOS Prod (Robot) <chromeos-ci-prod@chromeos-bot.iam.gserviceaccount.com> Reviewed-by: Tristan Honscheid <honscheid@google.com> Tested-by: Tristan Honscheid <honscheid@google.com> Commit-Queue: Tristan Honscheid <honscheid@google.com>

marc-hb · 2024-01-17T08:03:09Z

multiple-pause-resume is still very unreliable even with your fix @RanderWang , can you please take a look?

west.yml: update zephyr to main branch commit 53c121381f1a #8747 (comment)

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

kv2019i · 2024-02-01T12:17:07Z

@RanderWang @marc-hb How come all these bugs are closed even if the fix is not merged to SOF mainline? I don't see these errors in PR testing -- how can that be if the fix is not merged?

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>

marc-hb · 2024-02-01T17:20:54Z

How come all these bugs are closed even if the fix is not merged to SOF mainline?

Good question for @RanderWang who closed it as soon as the fix was merged in Zephyr.

In my book bugs are closed after at least one round of daily tests but opinions might differ?

Github's wrong opinion and really bad practice does not help: https://github.com/orgs/community/discussions/66741

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com>

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com> (cherry picked from commit e021ccf)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit fded11f)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug #8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com> (cherry picked from commit e021ccf)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit fded11f)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug #8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. (cherry picked from commit 00d4661) Original-Link: thesofproject/sof#8686 Original-Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> GitOrigin-RevId: 00d4661 Change-Id: Ifc46cd8ddd64c265f0e1ec00cae8a254b90aff1c Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/zephyr/+/5280273 Commit-Queue: Al Semjonovs <asemjonovs@google.com> Reviewed-by: Al Semjonovs <asemjonovs@google.com> Tested-by: Al Semjonovs <asemjonovs@google.com> Tested-by: ChromeOS Prod (Robot) <chromeos-ci-prod@chromeos-bot.iam.gserviceaccount.com>

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com> (cherry picked from commit e021ccf)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit fded11f)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit 00d4661)

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com> (cherry picked from commit e021ccf)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit 00d4661)

According to hardware spec, host dma needs some delay to completely stop. In the bug the host dma is disabled in different path in a few microseonds. The first setting disabled the host dma and called pm_device_runtime_put to power off it. The second setting found the host dma was still alive and calle pm_device_runtime_put again. This results to pm->usage checking failed. BugLink: thesofproject/sof#8686 Signed-off-by: Rander Wang <rander.wang@intel.com> (cherry picked from commit e021ccf)

Commit b2eaa6448076 ("drivers: dma: intel-adsp-hda: add delay to stop host dma") added a wait on GBUSY state to host DMA stop. This is problematic as in some case (like SOF chain-DMA usage), the host DMA side RUN bit is not cleared when intel_adsp_hda_dma_stop() is called. It is not possible to wait on GBUSY bit as there are valid cases where it can remain set. Address the original problem described in SOF bug zephyrproject-rtos#8686 and add a polling check for intel_adsp_hda_is_enabled(). As per the bug description, in some cases the GEN/FIFORDY bits are not cleared immediately and if a new call to intel_adsp_hda_dma_stop() is made, the PM refcounting will go haywire. Link: thesofproject/sof#8686 Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> (cherry picked from commit 00d4661)

kv2019i · 2024-03-07T20:22:01Z

Fixed and merged to both SOF main and stable-v2.9.

RanderWang added bug Something isn't working as expected MTL Applies to Meteor Lake platform labels Jan 3, 2024

RanderWang self-assigned this Jan 3, 2024

RanderWang mentioned this issue Jan 3, 2024

drivers: dma: intel-adsp-hda: add delay to stop host dma zephyrproject-rtos/zephyr#67135

Merged

marc-hb mentioned this issue Jan 12, 2024

[BUG] <wrn> runtime_suspend: Unbalanced suspend; <err> host_trigger(): dma stop failed: -120 in pause-resume test #8729

Closed

marc-hb added this to the v2.9 milestone Jan 12, 2024

RanderWang closed this as completed Jan 15, 2024

mengdonglin added IPC error IPC error is observed DMA stop failure labels Jan 24, 2024

kv2019i mentioned this issue Jan 26, 2024

ASoC: SOF: ipc4: change trigger order for chain DMA thesofproject/linux#4798

Closed

kv2019i mentioned this issue Feb 1, 2024

drivers: dma: intel-adsp-hda: modify stop dma logic zephyrproject-rtos/zephyr#68415

Merged

marc-hb reopened this Feb 1, 2024

kv2019i closed this as completed Mar 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]IPC error when testing multiple-pipeline on MTL-RVP NOCODEC #8686

[BUG]IPC error when testing multiple-pipeline on MTL-RVP NOCODEC #8686

RanderWang commented Jan 3, 2024 •

edited by marc-hb

Loading

RanderWang commented Jan 3, 2024 •

edited by marc-hb

Loading

marc-hb commented Jan 12, 2024

RanderWang commented Jan 15, 2024

marc-hb commented Jan 17, 2024

kv2019i commented Feb 1, 2024

marc-hb commented Feb 1, 2024

kv2019i commented Mar 7, 2024

[BUG]IPC error when testing multiple-pipeline on MTL-RVP NOCODEC #8686

[BUG]IPC error when testing multiple-pipeline on MTL-RVP NOCODEC #8686

Comments

RanderWang commented Jan 3, 2024 • edited by marc-hb Loading

RanderWang commented Jan 3, 2024 • edited by marc-hb Loading

marc-hb commented Jan 12, 2024

RanderWang commented Jan 15, 2024

marc-hb commented Jan 17, 2024

kv2019i commented Feb 1, 2024

marc-hb commented Feb 1, 2024

kv2019i commented Mar 7, 2024

RanderWang commented Jan 3, 2024 •

edited by marc-hb

Loading

RanderWang commented Jan 3, 2024 •

edited by marc-hb

Loading