[HOLD for payment 2023-11-21] [HOLD for payment 2023-11-09] [HOLD for payment 2023-11-01] Improve the reliability of the E2E testing suite

[mountiny]

Problem

Currently, the E2E testing suite runs the tests on one device over and over. This leads to performance degradation as runs keep on coming since the devices on AWS do not do a complete clean reboot between the runs. This is making the results less accurate and hard to make any calls based upon.

Solution

Explore options how to structure and run these tests so they are more accurate.

[ospfranco]

.

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.90-2 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA]Separate e2e test runs #29656

If no regressions arise, payment will be issued on 2023-11-01. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @ospfranco does not require payment (Contractor)

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.94-2 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] [DEV] E2e/increase reliability #30484

If no regressions arise, payment will be issued on 2023-11-09. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @ospfranco does not require payment (Contractor)

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.98-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] Staggered approach to E2E tests #30793

If no regressions arise, payment will be issued on 2023-11-21. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @ospfranco does not require payment (Contractor)

[AndrewGable]

@hannojg is working on this!

[hannojg]

Yeah i think we nailed this now and the tests are stable 😇🎉