[HOLD for payment 2024-11-07] [HOLD for payment 2024-11-05] E2E: Don't fail whole e2e test suite when one test fails

[hannojg]

Currently in the e2e testing pipeline if one test fails the whole test suite is failed. Instead the test suite should always complete, and failed test cases should be reported individually.

This has been discussed in here (in the #e2e-announce channel).

Notes:

• for each completely failed test case we should send a individual slack notification
• failed tests should be marked in the report as failed
• if a test failed in the test suite we should comment the report to the PR against which the test suite was executed, noting that a certain test is broken
  • If a test is broken on main and merged PRs after that would have this test as failed. we should add a note in the github comment that its possible that the test has been broken in a previous PR and that the author can check the commits on main to see if this PR's commit was the first failed one. This can help avoid confusion for PR authors

Issue Owner

Current Issue Owner: @kirillzyusko

[hannojg]

cc @mountiny and @kirillzyusko and @chrispader (i remember one of the two was available to work on this ticket)

[melvin-bot]

Triggered auto assignment to @MitchExpensify (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details. Please add this Feature request to a GH project, as outlined in the SO.

[melvin-bot]

⚠️ It looks like this issue is labelled as a New Feature but not tied to any GitHub Project. Keep in mind that all new features should be tied to GitHub Projects in order to properly track external CAP software time ⚠️

[mountiny]

Thanks! Please comment on this issue whoever will work on this

[melvin-bot]

Triggered auto assignment to Design team member for new feature review - @dubielzyk-expensify (NewFeature)

[dubielzyk-expensify]

Assuming you don't need design help here?

[kirillzyusko]

I can handle this @mountiny 🙌

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.54-11 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] e2e: unify test metrics naming #51338

If no regressions arise, payment will be issued on 2024-11-05. 🎊

For reference, here are some details about the assignees on this issue:

• @kirillzyusko does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@kirillzyusko] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[mountiny]

No regression tests or payments required for this. Updating the e2e test suite

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.55-10 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] e2e: allow tests failure #51248

If no regressions arise, payment will be issued on 2024-11-07. 🎊

For reference, here are some details about the assignees on this issue:

• @kirillzyusko does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@kirillzyusko] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[mountiny]

I believe we can close this one, updating the e2e test suite, no regression tests required