Discussion: measure the effectiveness and efficiency of our testing

[llxia]

As more and more new features are added, naturally we will have more and more tests. However, with the fast java release cadence, tight schedules, and limited resources (both human and machine), we should measure the effectiveness and efficiency of our testing. For example, do we have tests that are not valuable anymore? do we have test duplication tests? How to identify if we have any testing gaps? Do we have any tests that have never caught any bugs? Which tests are quick and effective that we should always run in PR builds?
This issue is opened for discussion regarding how to measure the effectiveness and efficiency of our testing.

Several ideas:

• scan git issues to get all defects. We can sort the data to find the most effective tests
• with the above data to create a heat map of problematic components/area
• user/client raised issues vs issues that are caught by AQA tests
• revive bug prediction

[smlambert]

I'll reference this issue in a proposed GSoC project here. Also noting this type of activity is explicitly called out in the AQAvit manifesto (specifically Section 2.3.3 Codecov & Other Metrics) linked from #965, where we also mention defect injection and comparative analysis mechanisms.