Test retry

[bootstraponline]

Fix #456

Read in JUnitReport.xml and remove duplicate results when flaky-test-attempts is > 0. For each test keep first result. If next result for the same test is successful, keep last successful result.

[codecov-io]

Codecov Report

Merging #522 into master will decrease coverage by 0.62%.
The diff coverage is 42.85%.

@@             Coverage Diff             @@
##             master    #522      +/-   ##
===========================================
- Coverage     80.33%   79.7%   -0.63%     
- Complexity      639     653      +14     
===========================================
  Files            74      75       +1     
  Lines          2029    2060      +31     
  Branches        259     273      +14     
===========================================
+ Hits           1630    1642      +12     
- Misses          251     258       +7     
- Partials        148     160      +12

[gtroshin]

Is there information about the same tests (with the same name) in JUnitReport.xml been run on different devices and how does it handled if for some device they failed for some all passed?

[bootstraponline]

Is there information about the same tests (with the same name) in JUnitReport.xml been run on different devices and how does it handled if for some device they failed for some all passed?

That's a good point. I'll push a fix for multi-device.

[valeraz]

From a quick look, this approach would probably break legitimate cases of duplicate test results in the final rollup (e.g. the use of repeatShards, unless you want to not allow the use of repeatShards with flakyTestAttempts). Deduping at the shard/device level seems like a safer approach.

[bootstraponline]

I pushed a new commit that names JUnitTestSuite after the device string (NexusLowRes-28-en-portrait). That should resolve concerns around the same tests being deduplicated across multiple devices.

[bootstraponline]

unless you want to not allow the use of repeatShards with flakyTestAttempts

I think you can use repeatShards with flakyTestAttempts, however flakyTestAttempts will activate deduplication. I agree about device level dedupe, that should work now.

[gtroshin]

I have a suite with several flaky tests and that new implementation showed it well! Everything is green.
Will you keep it in snapshot until the release or there are any plans to unmerge it?

[bootstraponline]

Will you keep it in snapshot until the release or there are any plans to unmerge it?

I merged the PR. It'll continue to be in the snapshots. Thanks for testing!