Model Quality Metrics

[AttilaMihaly]

Quality metrics are part of the standard SDLC process. Those are great to measure code quality but since they are not specifically focused on business logic they are very generic and technical. I think there are a lot of additional metrics we can calculate that are useful for measuring business logic quality specifically. Here are a few that came into my mind:

Specification Accuracy

Measures how accurate the types are in your model.

Logic Completeness

• number of possible input values declared / number of handled input values
• Measures what percentage of input values are handled by the function.
• In a language with no runtime exceptions any percentage below 100% would translate to a compile error (exhaustiveness check) but it's worth having this in case we want to be more flexible about this. It also nicely highlights one of the benefits of modeling in Elm.

Domain Utilization

• number of individual data-points declared in the input / number of input data-points used in the function
• "Individual data-points" refers to a decomposition that treats every argument and every type within a product type (records, tuples) separately.
• This measures if a function declares exactly what it needs or expects the user to pass in a lot of data that is not needed.
• A low percentage here would mean that the function will be less reusable and more difficult to test.

Range Utilization

• number of possible output values declared / number of actual output values that the function returns
• Measures the accuracy of the declared return type in terms of what percentage of the possible values the function will actually return.
• A low percentage here usually means that a more accurate type could be used.
• For example the logic might return a finite set of specific string values but declare and declare the return type as string. In this case the percentage will converge to zero while replacing it with an enum may yield 100% in the best case.
• The same applies to representing booleans as integers.

Test Quality

Branch Coverage

• number of branches in the logic / number of branches covered by at least one test case
• This is the usual coverage metric used in mainstream development.
• It measures the quality of your tests based on how much of the logic they cover.
• It doesn't say anything about the relationship between the tests and the requirement.

Test Completeness

• number of possible input values declared / number of input values covered by at least one test case
• Measures the quality of your tests based on what percentage of the possible inputs have test cases.
• With input types (like enums) of finite cardinality it is possible to achieve 100% here but infinite values will produce a value that converges to 0.
• For functions that take infinite but return finite types we can use test cases with wildcards.
• For functions that take and return infinite types we would have to refine the metric based on property tests.