Produce more details about the applied fixes

[sinaa]

When running the fix_test_suite.pl script, although the script is effective at removing failing tests, currently little information is produced about how many cases were flaky (failed on the class they were generated on), or not compiling (on the fixed or buggy versions).

Therefore, having the following information as a csv would be very helpful to have:

• (project, bug_id, key)
• total_removed
• non_compiling_buggy
• non_compiling_fixed
• flaky

In the produced log files, if the reasoning for each test-case removal could also be stated, it'd be great for test-generation tool maintainers to know where to focus.

A finer-grained test-removal is also essential for the accuracy of this of course, as I am only referring to the number test-cases removed (related PR: #69 ).

[jose]

Hi @sinaa,

A few weeks ago I implemented exactly what you're looking for. Please have a look at branch numbers_from_fix_test_suite, and please let me know if it does not produce all info you want. I haven't got the time, but at some point that branch would be pull-request.

--
Cheers,
Jose

[jose]

As pull requests #69 and #86 have been merged, I'm closing this issue.