Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non-status test results #83

Open
spbnick opened this issue Jul 31, 2024 · 0 comments
Open

Support non-status test results #83

spbnick opened this issue Jul 31, 2024 · 0 comments

Comments

@spbnick
Copy link
Collaborator

spbnick commented Jul 31, 2024

Some tests need to report results past the simple PASS/FAIL status, and, more importantly, we need to be able to track them. Here are the ideas so far:

  • We use the "DONE" status to signify that the test doesn't have a simple PASS/FAIL status, and we need to look elsewhere for results.

  • We attach a set of metric fields to each test (result), one for each value type. I.e. an integer, a floating-point number, a string, etc. We call it "value", so it reads "test->value".

    This would be close to the "status" level of abstraction. "Metric" would be confusing, as it would be closer to "test metric" meaning, that is a measurement about the test itself.

    We create separate columns in the test table for each type to enable indexing and thus fast comparison. In JSON we only allow one of the value types at once. E.g.:

      "status": "DONE",
      "value": {"float": 3.145},
    

    Or:

      "status": "DONE",
      "value": {"int": 42},
    
  • If people want to have more than one value as the test output, they would need to report a (synthetic) subtest for each one. This way the system would be flexible enough to track each metric separately. If any tests require tracking of a combination of metrics, then they should boil them down to a single value on their own, and report it in a parent test of the combined metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant