[SMOKECO] Fails in TH waits for a report from DUT with a timeout of 300 seconds

[raju-apple]

From chip-certification-tool created by KishokG: CHIP-Specifications/chip-certification-tool#951

Describe the bug

UI test runner fails at all "TH waits for a report from DUT with a timeout of 300 seconds" by throwing below error

Test Failure: The test expects no error but the "FAILURE" error occured. Test Failure: The test expects a value but it does not exists in the response."

Test case affected :

• TC-SMOKECO-2.2
• TC-SMOKECO-2.3
• TC-SMOKECO-2.4
• TC-SMOKECO-2.5
• TC-SMOKECO-2.6

Steps to reproduce the behavior

1. From the Test-Harness user interface loaded the Smoke CO Alarm Cluster Test Plan (1).xml and General Diagnostics Cluster Test Plan.xml PICS file to select the test cases.
2. Selected the SMOKECO Automated and Semi Automated test case to executed.
3. Executed the below mentioned command to put DUT into a commissionable state, ./chip-all-clusters-app --enable-key 00112233445566778899aabbccddeeff
4. Click the Start button on Test-Harness user interface.

Expected behavior

TC should execute successfully without error

Log files

SMOKE_2023_08_29_18_44_15.log

PICS file

SMOKE_PICS.zip

Screenshots

Environment

Version: TH Fall2023
Sha: 58a51bef

Additional Information

No response

[raju-apple]

From the log we can find the following:
WARNING | 2023-08-29 13:20:05.100144 | Test Failure: The test expects a value, but it does not exists in the response."

I think It's been a problem at waitForReport since 2022... The response has been given, but TH is slow, so it missed the response.
Referencing Issue project-chip/connectedhomeip#19861

The countering PR for the Smoke CO has been given, but after talking about it, I revoked my work-around. project-chip/connectedhomeip#28044

To solve the issue now, we may need the PR from Vivien at project-chip/connectedhomeip#28257
@cjandhyala

Update - Changed WaitForReport at script to manual step, make sure we don't block the test...Once this issue is resolved at TH, will revoke the YMAL steps.

[raju-apple]

@ericzijian1994 looks like the YAML has the waitforReport step as manual for now , so the test should not be blocked right now , is there any other issue with these set of scripts ?

[raju-apple]

@ericzijian1994 looks like the YAML has the waitforReport step as manual for now , so the test should not be blocked right now , is there any other issue with these set of scripts ?

As far as I know... I don't think the script has running issues with the waitForReport after this turned to manual, but again, I wish waitForReport implementation could be fixed (Improved) so people can have those manual steps turn to automated.

[ericzijian1994]

@ericzijian1994 looks like the YAML has the waitforReport step as manual for now , so the test should not be blocked right now , is there any other issue with these set of scripts ?

From practice, I don't see the current approach to the script having issues will block people...

[ericzijian1994]

@ericzijian1994 looks like the YAML has the waitforReport step as manual for now , so the test should not be blocked right now , is there any other issue with these set of scripts ?

As far as I know... I don't think the script has running issues with the waitForReport after this turned to manual, but again, I wish waitForReport implementation could be fixed (Improved) so people can have those manual steps turn to automated.

May be it is possible to try to read the value before performing the wait-for-report step, and if the value is the same as the expected result, then there is no need to perform the wait. Or read the value at intervals while waiting to avoid missing reports. Equivalent to using notifications and polling at the same time. I don't know if such comments will be useful to improve the waitForReport command.

[fabiowmm]

Awaiting project-chip/connectedhomeip#28257

[fabiowmm]

SDK issue now fixed.