[HOLD for payment 2024-06-28] Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm

[github-actions]

🚨 Failure Summary 🚨:

• 📋 Job Name: e2ePerformanceTests / Run E2E tests in AWS device farm
• 🔧 Failure in Workflow: Process new code merged to main
• 🔗 Triggered by PR: PR Link
• 👤 PR Author: @tienifr
• 🤝 Merged by: @dangrous
• 🐛 Error Message:
  warning: Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: realm/aws-devicefarm@7b9a912. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
  failure: Process completed with exit code 1.
  warning: No matching artifact of type LOG found for debug.log, skipping
  failure: Test run failed after 3145 seconds with: undefined. Timeout is set to 5400

⚠️ Action Required ⚠️:

🛠️ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Run E2E tests in AWS device farm.
This issue has been automatically created and labeled with Workflow Failure for investigation.

👀 Please look into the following:

1. Why the PR caused the job to fail?
2. Address any underlying issues.

🐛 We appreciate your help in squashing this bug!

[dangrous]

I'm 99% certain this has nothing to do with the PR - I think it's something on the AWS side... Looking at the test it looks like $WORKING_DIRECTORY maybe didn't turn into the value it was supposed to? There are a couple matching issues already open (see my links above) so I'm going to try to re run the job and see if that fixes it. Pretty sure it's just an occasional one time thing.

[kirillzyusko]

@dangrous I'm totally sure it's not because of your changes. Recently we added back one test (we thought it's stable), but turned out, that if we run 100+ iterations, then it may be broken.

I'm currently working on a fix 👀

[dangrous]

Hrm, @kirillzyusko it looks like it's still failing even now that your changes are merged. Does it still feel related? (or is there another way to test other than just rerunning the tests on that PR)?

[kirillzyusko]

@dangrous may I ask you to send a commit from main where it failed?

[dangrous]

oh maybe not from main? I just re-ran on the original PR that triggered this. Maybe it won't work ever, since that's not the updated code? https://github.com/Expensify/App/actions/runs/9568700663

[kirillzyusko]

Maybe it won't work ever, since that's not the updated code?

Right, I modified e2e code that runs on the device (i. e. it's included in main app .apk file), so in order to pass CI job you need to have this fresh code. If you are re-running old job it still will fail.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.0-9 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• fix: comment linking e2e test #43986

If no regressions arise, payment will be issued on 2024-06-28. 🎊

[dangrous]

Ah okay thanks for the explanation @kirillzyusko! I will go ahead and close this.