New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Declarative PIXITs in python tests #395 #36817

Draft

asirko-soft wants to merge 10 commits into project-chip:master from asirko-soft:osirko-declarative_PIXITs_in_python_tests

+366 −5

asirko-soft commented Dec 12, 2024

project-chip/matter-test-scripts#395

Added a common mechanism for tests to declare their optional and required PIXIT values and override flags and ensure the required values are supplied. Add information about optional override flags with descriptions in the error outputs of tests.

Example:

def pixits_TC_ICDM_3_4(self) -> list[PIXITDefinition]:
        """Declare PIXITs required for this test"""
        return [
            PIXITDefinition(
                name="PIXIT.WAITTIME.REBOOT",
                pixit_type=PIXITType.INT,
                description="Time to wait for device reboot (in seconds)",
                required=True
            )
        ]

asirko-soft added 3 commits

December 11, 2024 15:04


          add dataclass to support PIXIT validations

fbf03c4


          Extend existing MatterTestConfig to handle PIXIT definitions

11d86ab


          move validation logic to a separate class

e96ec4b

semanticdiff-com bot commented Dec 12, 2024 •

edited

Loading

Review changes with

Changed Files

File	Status
src/python_testing/TestMatterTestingSupport.py	3% smaller
src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py	2% smaller

github-actions bot added tests matter-1.4-te2-script-change labels

tcarmelveilleux reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                  def validate_bool(value: Any) -> bool:
+                      if isinstance(value, str):
+                          value_lower = value.lower()
+                          if value_lower not in ('true', 'false', '1', '0', 'yes', 'no'):

Contributor

tcarmelveilleux Dec 12, 2024

only true false supported

Author

asirko-soft Dec 16, 2024

fixed

tcarmelveilleux reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                  @staticmethod
+                  def validate_float(value: Any) -> bool:
+                      float(value)

Contributor

tcarmelveilleux Dec 12, 2024

value is str for all validators
validate_xxx --> is_xxx_pixit_value_valid

Author

asirko-soft Dec 16, 2024

fixed

tcarmelveilleux reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                  @staticmethod
+                  def validate_string(value: Any) -> bool:
+                      str(value)

Contributor

tcarmelveilleux Dec 12, 2024

For all these you need to wrap the possibly-failing conversion in a try: except ValueError before true/false returned.

Also, when returning False, you are losing the reason for why it failed validation, which would have come out fo the exception.

Recommend moving all of these validators to raise exceptions rather than be bool return. In that case, it could remain validate_xxx

Author

asirko-soft Dec 16, 2024

fixed and converted bool returns to exceptions

tcarmelveilleux reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                      return True
+                  @classmethod
+                  def validate_value(cls, value: Any, pixit_def: PIXITDefinition) -> tuple[bool, Optional[str]]:

Contributor

tcarmelveilleux Dec 12, 2024

Avoid raw tuple returns. Raise an exception with message on failure

Author

asirko-soft Dec 16, 2024

Thank you for this, I was afraid that with using tuple there readability degrades a bit. Removed tuple return

tcarmelveilleux reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                      if missing or invalid:
+                          error = ""
+                          if missing:
+                              error += "\nMissing required PIXITs:\n" + "\n".join(missing)

Contributor

tcarmelveilleux Dec 12, 2024

Please avoid multi-line emssages

Author

asirko-soft Dec 16, 2024

fixed

cecille reviewed

View reviewed changes

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py

+                  default: Any = None
+                  @staticmethod
+                  def is_pixit(arg_name: str) -> bool:

Contributor

cecille Dec 12, 2024

actually, we probably want to validate all the flags. There are some flags that should be exposed that aren't called PIXIT.whatever

Author

asirko-soft Dec 16, 2024 •

edited

Loading

Do you suggest to rename all methods to use some more generic words?

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated Show resolved Hide resolved

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Outdated

+                  # Validate PIXITs before proceeding
+                  validator = PIXITValidator()
+                  valid, error = validator.validate_pixits(test_info[0].pixits, matter_test_config.global_test_params)

Contributor

cecille Dec 12, 2024

Does this validate only for the first test in the list?

might be better in the setup_test function for each test.

Author

asirko-soft Dec 16, 2024

Yeah, thank you for noticing this issue, I've moved validation to setup_test

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py

@@ @@ -612,6 +612,142 @@ def test_skipped(self, filename: str, name: str): @@
                       logging.info(f"Skipping test from {filename}: {name}")
+              class PIXITType(Enum):
+                  INT = "int"

Contributor

cecille Dec 12, 2024

This approach in general is OK, but I do want to consider this in the larger context of verifying that the test invocations are correct in a broader way. We can land this in stages, but it is important to consider how the other checks are going to be integrated here.

In particular, test authors need an easy way to declare certain other test requirements that also need verification. Ex. Endpoint project-chip/matter-test-scripts#450, which we should consider mandating with no default for tests that are run against a single endpoint, no --endpoint flag for tests that run against the entire node at once. Need a way to declare that a test requires a commissioned device, whether or not that commissioning happens as a part of the test invocation. Some tests require passcodes, discriminators etc, even for tests where the commissioning happens external to the test. Tests that DO use PICS (hopefully fewer and fewer) should have a PICS file specified (and this get extra difficult in the context of the test harness)

Anyway, worth thinking through how these extended checks would fit in with this design. I don't see anything in particular blocking these options right now.

It's also worth thinking through how to expose this properly to test authors. ie, is there a way that we can alert test authors that are using test flags that they should be providing these functions?

src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py

		@@ -612,6 +612,142 @@ def test_skipped(self, filename: str, name: str):
		logging.info(f"Skipping test from {filename}: {name}")


		class PIXITType(Enum):

Contributor

cecille Dec 12, 2024

I know I used PIXIT in the issue, but I think this applies generally to test flags. But I can't think of a great name for just "Flags" that doesn't sound too generic. Ideas welcome.

Author

asirko-soft Dec 16, 2024

To be honest, I also don't have many ideas. TestFlag? TestFlagType? TestParamType?

cecille reviewed

View reviewed changes

Contributor

cecille left a comment

Would you mind adding some tests here that the verifiers are correctly catching errors? We have some examples of how to test these frameworking bits in TestMatterTestingSupport.py et. al.

asirko-soft added 2 commits

December 13, 2024 15:14


          rename validators and adjust value type

93c6837


          change from boolean checks to handling of exceptions, fix issue where…

d3976d0

… validation would run only for first test

github-actions bot commented Dec 16, 2024

PR #36817: Size comparison from 75ab4c9 to d3976d0

Full report (1 build for stm32)

platform	target	config	section	`75ab4c9`	`d3976d0`	change	% change
stm32	light	STM32WB5MM-DK	FLASH	484720	484728	8	0.0
			RAM	144880	144880	0	0.0

asirko-soft and others added 5 commits

December 16, 2024 17:05


          don't validate pixits for commissioning test, fix typo, reformat

17c05f9


          commit review suggestion

13021d6

Co-authored-by: C Freeman <cecille@google.com>


          add tests for pixit validator

ef7d4bc


          apply linter suggestions

663add6


          apply Ruff suggestions

7d14dc9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

matter-1.4-te2-script-change tests