[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch #21

raju-apple · 2023-10-19T18:15:34Z

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#649

Description
[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch at step 2d

Additional Info :
TestHarness:
Version: v2.7Beta2
Sha: 1aff1e6

PFA log below
CC.log

raju-apple · 2023-10-19T18:15:35Z

Issue resolved in latest Image .
Additional Info:
Version: v2.7.1-official-TE
Sha: e22d9d6

raju-apple · 2023-10-19T18:15:36Z

@Rajashreekalmane @raju-apple , this is still happening on v2.8.1

raju-apple · 2023-10-19T18:15:37Z

@Rajashreekalmane @raju-apple , this is still happening on v2.8.1

Verified with TH Version: v2.8.1-officialSha: 1446f09 TC-CC-3.2 works as expected and ends with PASS.

DUT : All-clusters-app
Endpoint : 1
PICS used : TC_CC_PICS.zip

Snapt Shot of the particular step :

References :

Execution Log : UI_Test_Run_2023_05_23_12_38_32.log
Execution Report : Test Report.pdf

Please cross verify the log that I have attched also please attch your so that we will be able to RC the issue.

raju-apple · 2023-10-19T18:15:38Z

@Survensa thanks for checking this. I added timestamp to my DUT logs, please have a look at it. At line 2768, the DUT starts to transition downwards from 60 at a rate of 5steps/second. At line 2885 the read request comes in.

Time between start transition and read request is 10.6 seconds.
10.6 * 5 = 53
So the CurrentHue read will be 60 - 53 = 7, causing my DUT to fail.

I think when running this with the linux all-clusters-app on the same host as TH, the communication delay between the TH and all-clusters-app is smaller as compared to real products, that's why you are not able to reproduce it. I believe some other platforms are facing this issue as well.

DUT log: cc-3-2.txt

Related issues:
https://github.com/CHIP-Specifications/chip-test-plans/issues/2332
https://github.com/CHIP-Specifications/chip-certification-tool/issues/590

raju-apple · 2023-10-19T18:15:39Z

I just wanted to add our logs. We've seen that the TestHarness 2.8.1 is failing consistently for us, while running the same test case from ChipTool on a PC it is consistently passing. The difference appears to be the timing of sending the message after the 10 second delay. It seems the TestHarness is around 500ms slower.

TestHarness Log CC_3-2_2023_07_12_11_49_08.log

Here is an excerpt from the logs for our Test Harness. The message is sent at 1689126611.442990 and then the second message is sent at 1689126622.077342 which is a 10.634 second gap. (So, going from 60 hue to 7 hue over 10.634 seconds is the correct value).

CHIPTOOL   | 2023-07-12 01:50:11.632652 | [1689126611.442067][24:32] CHIP:TOO:  ***** Test Step 17 : TH sends MoveHue command to DUT with MoveMode=0x03 (down) and Rate=5 (units/s)
...
CHIPTOOL   | 2023-07-12 01:50:11.660656 | [1689126611.442990][24:32] CHIP:IN: (S) Sending msg 143287555 on secure session with LSID: 19050

CHIPTOOL   | 2023-07-12 01:50:22.112156 | [1689126622.076481][24:32] CHIP:TOO:  ***** Test Step 19 : TH reads CurrentHue attribute from DUT
...
CHIPTOOL   | 2023-07-12 01:50:22.526851 | [1689126622.077342][24:32] CHIP:DMG: SendReadRequest ReadClient[0xffff78016390]: Sending Read Request

Compared to the CHIP Tool on the PC. This takes 10.134 seconds to send the message. There I've seen between 8-10 which is within the tolerance.

[1689296342073] [68576:8663090] [TOO]  ***** Test Step 17 : TH sends MoveHue command to DUT with MoveMode=0x03 (down) and Rate=5 (units/s)
[1689296342074] [68576:8663090] [IN] (S) Sending msg 18120214 on secure session with LSID: 44065
[1689296352207] [68576:8663209] [TOO]  ***** Test Step 19 : TH reads CurrentHue attribute from DUT
[1689296352208] [68576:8663209] [IN] (S) Sending msg 18120216 on secure session with LSID: 44065

raju-apple · 2023-10-19T18:15:40Z

thanks for providing details. Root cause is a wrong interpretation of tolerance rules in the script - one example of https://github.com/CHIP-Specifications/chip-test-plans/issues/2332 .
While it would be good to speed up the TH, or compensate for the extra delay (if this is fixed, we can compensate and make the wait time 9.5 s instead of 10 s)
But the resolution for this particular issue is fixing the tolerances, see https://github.com/CHIP-Specifications/chip-test-plans/issues/2332#issuecomment-1635497808 for details

raju-apple · 2023-10-19T18:15:41Z

This issue has been addressed in this PR project-chip/connectedhomeip#28159 (commit 98781285915b198b4f01b32ba36c88a96983b2c4). The accepted range was expanded from 8-12 to 4-16. Not sure when that will make it's way into the Test Harness, but just wanted to share that the Test Case itself has been updated.

raju-apple · 2023-10-19T18:15:42Z

@stonehouse the latest TH will include the changes , please try it out and close the issue please

raju-apple · 2023-10-19T18:15:43Z

@raju-apple We've tested Test Harness beta (TH_fall2023_beta3) and found that the problem discussed appears to be worse in this version (when compared with 2.8.1). We're now seeing steps 2c, 2d, 2f, 2g and 3c all fail. Looking at the logs it appears the Test Harness is taking longer to send messages so steps that were previously passing are now exceeding the tolerance.

For example, step 2c starts at hue 200 and uses MoveHue to go up, it waits 10 seconds and expects a hue of 250. However, the tolerance is 215-255, which means there's only an effective 1 second tolerance (250-255). Testing on a Raspberry Pi 4 on an isolated network (we cooled the Pi down to ensure thermal throttling wasn't playing a role), the logs show it took 1.41 seconds for the Pi to send the message. So, if you consider starting at 200 hue and going up at 5 hue per second, then after 11.41 seconds a value of 1 seems correct.

From our testing, it looks like the Test Harness was taking around 500ms to send a message in 2.8.1 but is now taking over a second in the newest beta.

CC-3.2 Step 2c Failure

INFO       | 2023-08-08 08:11:38.921846 | Executing Test Step: Step 2b: TH sends MoveHue command to DUT with MoveMode=0x01 (up) and Rate=5 (units/s)
CHIPTOOL   | 2023-08-08 08:11:39.399382 | CHIP:TOO Command: colorcontrol move-hue 1 5 0 0 0x1950474687ba6a9e 1 
CHIPTOOL   | 2023-08-08 08:11:39.400909 | CHIP:TOO Sending command to node 0x1950474687ba6a9e
...
CHIPTOOL   | 2023-08-08 08:11:50.811706 | CHIP:EM <<< [E:45782i S:30820 M:35146545] (S) Msg TX to 1:1950474687BA6A9E [10A2] --- Type 0001:02 (IM:ReadRequest)
CHIPTOOL   | 2023-08-08 08:11:50.812566 | CHIP:IN (S) Sending msg 35146545 on secure session with LSID: 30820
...
CHIPTOOL   | 2023-08-08 08:11:50.879119 | CHIP:TOO Endpoint: 1 Cluster: 0x0000_0300 Attribute 0x0000_0000 DataVersion: 1636690129
CHIPTOOL   | 2023-08-08 08:11:50.879982 | CHIP:TOO   CurrentHue: 1
WARNING    | 2023-08-08 08:11:50.881543 | Test Failure: The response value (1) should be greater or equal to the constraint but 1 < 215.
INFO       | 2023-08-08 08:11:50.895509 | Test Step Completed [FAILED]: Step 2c: TH reads CurrentHue attribute from DUT

CC-3.2 Step 3c Failure

INFO       | 2023-08-08 08:12:19.368673 | Executing Test Step: Step 3b: TH sends MoveHue command to DUT with MoveMode=0x03 (down) and Rate=5 (units/s)
CHIPTOOL   | 2023-08-08 08:12:20.205705 | CHIP:TOO Command: colorcontrol move-hue 3 5 0 0 0x1950474687ba6a9e 1 
CHIPTOOL   | 2023-08-08 08:12:20.207495 | CHIP:TOO Sending command to node 0x1950474687ba6a9e
...
INFO       | 2023-08-08 08:12:30.658668 | Executing Test Step: Step 3c: TH reads CurrentHue attribute from DUT
CHIPTOOL   | 2023-08-08 08:12:31.313677 | CHIP:TOO Command: colorcontrol read current-hue 0x1950474687ba6a9e 1 
CHIPTOOL   | 2023-08-08 08:12:31.316085 | CHIP:TOO Sending command to node 0x1950474687ba6a9e

Here are logs from two separate test runs
m141_fab_CC_3-2_th-fall-beta3_2023_08_08_18_11_03.log
m141_fab_CC_3-2_th-fall-beta3_2023_08_08_18_07_06.log

raju-apple · 2023-10-19T18:15:44Z

@fabiowmm can you please take a look at this issue ?

raju-apple · 2023-10-19T18:15:45Z

Verified with TH Version: TH Fall2023 , Sha: 22538eb7 TC-CC-3.2 test case is still failing in step 2c, 2d with Cutrrent Hue attribute value mismatch.

DUT : All-clusters-app
Endpoint : 1
PICS used : Color Control Cluster Test Plan.zip

Snapt Shot of the particular steps :

References :
Execution Log : TC-CC-3.2.log

raju-apple · 2023-10-19T18:15:46Z

quoting from @stonehouse message above:

For example, step 2c starts at hue 200 and uses MoveHue to go up, it waits 10 seconds and expects a hue of 250. However, the tolerance is 215-255, which means there's only an effective 1 second tolerance (250-255).

This tolerance range is wrong!!
The tolerance should be symmetric around the expected value (250) and take into account that hue valiues will wraparound
So if a tolerance range of 35 is allowed (example)...

the minimum allowed value is 250-35 = 215
and the maximum allowed value is 250+35 = 285 which wraps around to 285-256=29

So the script should accept values in range 215..255 and 0..29 in this case.
(note that I'm using example value, please do the math for all applicable cases and update script and VS

fyi @raju-apple @Ashwinigrl @fabiowmm @Survensa @akrithika27

Ashwinigrl · 2023-12-13T11:58:29Z

@manjunath-grl Verified with TH Version: Version: main - dev Sha: f17efc4 TC-CC-3.2 test case is failing in step 2d, 2f and 2g with Cutrrent Hue attribute value mismatch.

DUT : All-clusters-app
Endpoint : 1
PICS used :
Color Control Cluster Test Plan .txt
On-Off Cluster Test Plan .txt
Screen Shot of the particular steps :

References :
Execution Log :
TC-CC-3.2.log

manjunath-grl · 2023-12-13T12:33:21Z

@Ashwinigrl Formerly, the tolerance was set as TargetValue ±15%, but it has now been updated to tolerance = (TargetValue - StartValue) ±15%. This modification is reflected in the PR 30951 please verify and close the ticket.

rquidute · 2024-01-16T16:21:15Z

Hi @Ashwinigrl is this issue still valid?

Rajashreekalmane · 2024-02-26T10:26:40Z

@rquidute Yes, Still we are facing this issue in RPI platform

Version: 2.10-beta3+spring2024
Sha: 1fd1429

PFA screenshot :

PFA execution Log :
UI_Test_Run_2024_02_26_10_21_37.log
TC-CC-3.2.txt

PICS Used :
On-Off Cluster Test Plan.txt
Color Control Cluster Test Plan.txt

* Update yaml_test_folder.py * Update test_sdk_yaml_collection.py * Update unittests.yml Don't run old code for unit tests

* Fix prblem with sdk_tests collection using hardcoded paths * Enable github workflows for main branch (#17) * Enable github workflows for main branch * Ignore Dockerfile * mypy: ignore python_testing in sdk_collection * minor linting issues fixed * ignore .devcontainer in spell checker * move .env file back in root * Update reference to fetch_sdk_tests_and_runner.sh * Use `pull_request_target` event to support forks * Update yaml_test_folder.py (#21) * Update yaml_test_folder.py * Update test_sdk_yaml_collection.py * Update unittests.yml Don't run old code for unit tests * [Fix] remove TestRunConfig usage from TH Backend (#581) (#6) * Add the repeat endpoint, deprecate the test_run_config endpoints and force the test run execution creation to use selected tests, and not test_run_config_id. * Deprecating unit tests related with test_run_config, creating new endpoint with required selected_tests parameter and ignoring the test_run_config reference, renaming the repeat endpoint method and adding comments. * Updating select_tests logic, moving to a utils file, and adding unit test for the repeat endpoint * Updating comments and adding missing attribution to None for test_run_config_id * Adding missing TODO comment. * Fixing selected_tests logic * Updating the 'repeat' unit test to use more suites to cover more cases. * Removing "/new" endpoint, updating logic of the default test run execution endpoint and updating unit tests, "repeat" endpoint and utils * Adding the new DB column "collection_id" to the TestSuiteExecution table. A DB revision file was created and the code was refactored to use this new info. * Removing unnecessary import * Set default log level to info, to eliminate chatter from python runner (#23) * Update SDK SHA (#25) * Added support to reset db for test (#28) * Missing submit button (#30) * Updates MessageTypeEnum to include options and message types * Update app/user_prompt_support/user_prompt_manager.py Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com> * Update app/constants/websockets_constants.py Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com> * Update app/user_prompt_support/user_prompt_manager.py --------- Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com> Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com> Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com> Co-authored-by: Carolina Lopes <116589288+ccruzagralopes@users.noreply.github.com> * Update version * Prompt request refactor (#31) * Refactor to use only 1 enum * Removed unsed import * Fix isort * Adds a new Message prompt request --------- Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org> * update backend version in FE (#32) * update backend version in FE * Update .version_information * Update .version_information * change ssh to https (#33) * Undo version information update --------- Co-authored-by: Mikael H. Moeller <mikaelhm@apple.com> Co-authored-by: Mikael Møller <mikaelhm@hey.com> Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com> Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com> Co-authored-by: Raul Marquez <130402456+raul-marquez-csa@users.noreply.github.com> Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com> Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org>

raju-apple added the v1.1 label Oct 19, 2023

raju-apple removed the v1.1 label Oct 31, 2023

fabiowmm assigned gladystonfranca Oct 31, 2023

fabiowmm added the Awaiting Dev Feedback label Oct 31, 2023

manjunath-grl mentioned this issue Dec 7, 2023

Fix YAML scripts Dec 07 project-chip/connectedhomeip#30869

Merged

raju-apple closed this as completed Jun 17, 2024

rquidute pushed a commit that referenced this issue Sep 23, 2024

Update yaml_test_folder.py (#21)

a68241b

* Update yaml_test_folder.py * Update test_sdk_yaml_collection.py * Update unittests.yml Don't run old code for unit tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch #21

[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch #21

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

Ashwinigrl commented Dec 13, 2023

manjunath-grl commented Dec 13, 2023 •

edited

Loading

rquidute commented Jan 16, 2024

Rajashreekalmane commented Feb 26, 2024 •

edited

Loading

[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch #21

[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch #21

Comments

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

Snapt Shot of the particular step :

References :

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

raju-apple commented Oct 19, 2023

Ashwinigrl commented Dec 13, 2023

manjunath-grl commented Dec 13, 2023 • edited Loading

rquidute commented Jan 16, 2024

Rajashreekalmane commented Feb 26, 2024 • edited Loading

manjunath-grl commented Dec 13, 2023 •

edited

Loading

Rajashreekalmane commented Feb 26, 2024 •

edited

Loading