Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature suggestion - Client test cases launch All-Clusters on the Test Harness #17

Closed
raju-apple opened this issue Oct 19, 2023 · 9 comments
Closed

Feature suggestion - Client test cases launch All-Clusters on the Test Harness #17

raju-apple opened this issue Oct 19, 2023 · 9 comments
Assignees
@fabiowmm
Labels
Feature v1.4 Issues found in 1.4 timeframe

Comments

@raju-apple
Copy link

From chip-certification-tool created by EF-JANC: CHIP-Specifications/chip-certification-tool#472

For DUT as Client test cases, when running the Manual_tests, would it be possible to start the All-Clusters-app as TH in the Test Harness so this can be logged directly with the test case, instead of us having to run 3 separate logging windows (TH + SSH To All-Clusters + SSH to DUT)
@raju-apple raju-apple added sve1 v1.4 Issues found in 1.4 timeframe labels Oct 19, 2023
@raju-apple
Copy link
Author

My initial concerns are -

  • We need to delete the temp files and run the cluster app between test runs.
  • Looks like there are a bunch of custom apps, should we always pick the all-clusters-app?

@raju-apple
Copy link
Author

Response and design proposal by @mikaelhm:
We might need a custom Test Suite for these so the app is just commisioned once. So we can avoid it for all test cases.
We could make a special "assisted manual test"

  • Starts app (agree it should be configurable)
  • streams logs without filtering to test engine logger
  • Runs normal manual test case prompt flow.

@raju-apple
Copy link
Author

Visualizing @mikaelhm 's design proposal
Screen Shot 2022-08-05 at 2 44 55 PM

@raju-apple
Copy link
Author

@sparampallirajag @mikaelhm Thanks for the feedback!

Please allow me to provide some feedback on the above :)

-Looks like there are a bunch of custom apps, should we always pick the all-clusters-app?
The testplan 'always' reffers to the All-Clusters-App as the TH so I would say yes as this would contain all clusters and should be the most advanced tool foor client tests.

The above approach of having it as 'manual' tests is fine - I just wanted to propose a more user friendly way of working.

And with regards to the deleting the tmp files this is correct, I have made a small script for this to ease the server tests on a SDK ;

sudo rm -rf /tmp/chip_* #remove the CHIP Temp files to allow for new commisioning
 
sleep .5 # Wait 0.5 second.
​
./apps/chip-all-clusters-app #Launch ACA

@raju-apple
Copy link
Author

Work in progress. This is not a blocker and will not be ready in a few days. So, removing the v1.x label.

@fabiowmm fabiowmm self-assigned this Oct 31, 2023
@raju-apple
Copy link
Author

@fabiowmm any resolution on this ?

rquidute pushed a commit that referenced this issue Sep 23, 2024
@mikaelhm
Enable github workflows for main branch (#17)
48ad49b 
* Enable github workflows for main branch

* Ignore Dockerfile

* mypy: ignore python_testing in sdk_collection

* minor linting issues fixed

* ignore .devcontainer in spell checker

* move .env file back in root

* Update reference to fetch_sdk_tests_and_runner.sh

* Use `pull_request_target` event to support forks
rquidute added a commit that referenced this issue Sep 23, 2024
@ccruzagralopes @mikaelhm
@antonio-amjr @rquidute @raul-marquez-csa @hiltonlima
Merge 2.10-beta1.1+spring2024 release into main (#34)
30f7d99 
* Fix prblem with sdk_tests collection using hardcoded paths

* Enable github workflows for main branch (#17)

* Enable github workflows for main branch

* Ignore Dockerfile

* mypy: ignore python_testing in sdk_collection

* minor linting issues fixed

* ignore .devcontainer in spell checker

* move .env file back in root

* Update reference to fetch_sdk_tests_and_runner.sh

* Use `pull_request_target` event to support forks

* Update yaml_test_folder.py (#21)

* Update yaml_test_folder.py

* Update test_sdk_yaml_collection.py

* Update unittests.yml

Don't run old code for unit tests

* [Fix] remove TestRunConfig usage from TH Backend (#581) (#6)

* Add the repeat endpoint, deprecate the test_run_config endpoints and

force the test run execution creation to use selected tests, and not test_run_config_id.

* Deprecating unit tests related with
test_run_config, creating new endpoint with required selected_tests
parameter and ignoring the test_run_config reference, renaming the
repeat endpoint method and adding comments.

* Updating select_tests logic, moving to a utils
file, and adding unit test for the repeat endpoint

* Updating comments and adding missing attribution to
None for test_run_config_id

* Adding missing TODO comment.

* Fixing selected_tests logic

* Updating the 'repeat' unit test to use more
suites to cover more cases.

* Removing "/new" endpoint, updating logic of the
default test run execution endpoint and updating unit tests, "repeat"
endpoint and utils

* Adding the new DB column "collection_id" to the
TestSuiteExecution table. A DB revision file was created and the code
was refactored to use this new info.

* Removing unnecessary import

* Set default log level to info, to eliminate chatter from python runner (#23)

* Update SDK SHA (#25)

* Added support to reset db for test (#28)

* Missing submit button (#30)

* Updates MessageTypeEnum to include options and message types

* Update app/user_prompt_support/user_prompt_manager.py

Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>

* Update app/constants/websockets_constants.py

Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>

* Update app/user_prompt_support/user_prompt_manager.py

---------

Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com>
Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>
Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>
Co-authored-by: Carolina Lopes <116589288+ccruzagralopes@users.noreply.github.com>

* Update version

* Prompt request refactor (#31)

* Refactor to use only 1 enum

* Removed unsed import

* Fix isort

* Adds a new Message prompt request

---------

Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org>

* update backend version in FE (#32)

* update backend version in FE

* Update .version_information

* Update .version_information

* change ssh to https (#33)

* Undo version information update

---------

Co-authored-by: Mikael H. Moeller <mikaelhm@apple.com>
Co-authored-by: Mikael Møller <mikaelhm@hey.com>
Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>
Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com>
Co-authored-by: Raul Marquez <130402456+raul-marquez-csa@users.noreply.github.com>
Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>
Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org>
@fabiowmm
Copy link
Contributor

@rquidute @ccruzagralopes isso aqui foi realizado pelos commits referenciados acima?

@ccruzagralopes
Copy link
Contributor

I believe this mention is a GitHub bug. The mentioned commit is actually from a PR in the backend repo that has the same issue number as this issue:

And just FIY: I believe you didn't notice that you wrote the message in Portuguese 😆

@fabiowmm
Copy link
Contributor

Deprioritized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fabiowmm fabiowmm

Labels
Feature v1.4 Issues found in 1.4 timeframe
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@raju-apple @fabiowmm @ccruzagralopes