chore: adds quality gate for rerunning e2e spec files that are new or have been modified

[seaona]

Description

This PR adds a quality gate for new or modified e2e spec files. Whenever there is a PR which modifies or changes a test, this will be run more times, in order to prevent introducing a flakiness accidentally. It is done as follows:

• Identifies any new or modified e2e file from inside the test/ folder using git diff and using these 2 filters:
  • file.filename.startsWith('test/e2e/') &&
  • file.filename.endsWith('.spec.js') || file.filename.endsWith('.spec.ts')
• Copies the given specs x5 times in the list of testpaths to execute -> this number is arbitrary, we could modify it to any value we want. The reason for taking this approach instead of changing the retrial number is to benefit of the parallelization, as @HowardBraham pointed out in a comment.
• Since we already had a flag which could support the re-running successful tests, --retry-until-failure I just leveraged this into the for loop for each test, and if that testcase was identified as new/modified, the flag is added so the new tests fail fast without retrials

Incremental git fetch depth within shallow clone

We use git fetch with incremental depth as @danjm suggested. The ci environment uses a shallow clone, meaning we won't be able to succeed just by using git diff as it won't find the merge base. For fixing that, we start with a git fetch depth of 1, and keep incrementing the depth (1, 10, 100) it the error is no merge base up until 100. If the git diff still fails, we then do a full git fetch with the unshallow flag.

• [Update] This is the working result with the last commit https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/89269/workflows/103b78a8-8f0d-4323-96b0-9e235c4cbc81/jobs/3296802
  

New ci Job

The git diff is done in a new ci job which runs at the beginning in parallel of prep-deps.

Related issues

Fixes: #24009

Manual testing steps

1. Check ci runs (notice previous runs had failing and changed tests on purpose, in order to try the different scenarios described below)

Screenshots/Recordings

=============================================== [UPDATE with the new code changes]

• 🟢 Case 1: A test has changed -> it's rerun 1+5 times and it's successful (it will be run in different buckets)

new-tests-ci.mp4

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/89277/workflows/7fce0a2e-773f-46da-8ab9-1dbec7992b58/jobs/3297267/parallel-runs/10?filterBy=ALL

• 🟢 Case 2: A test has changed, but it has a mistake in the code (intentionally to simulate a flaky test) -> it fails immediately and there are no more retries. The rest of the tests, are retried if they failed as usual

  • Case for main test build test: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/89277/workflows/7fce0a2e-773f-46da-8ab9-1dbec7992b58/jobs/3297267/artifacts
  • Case for flask specific test: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/89277/workflows/7fce0a2e-773f-46da-8ab9-1dbec7992b58/jobs/3297277/artifacts

• 🟢 Case 3: A PR has no test spec files changed -> nothing different happens

• ci run: check current ci

Pre-merge author checklist

• I’ve followed MetaMask Coding Standards.
• I've completed the PR template to the best of my ability
• I’ve included tests if applicable
• I’ve documented my code using JSDoc format if applicable
• I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

• I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
• I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

[github-actions]

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.57%. Comparing base (4a9c480) to head (07e81d8).
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #24556   +/-   ##
========================================
  Coverage    69.57%   69.57%           
========================================
  Files         1360     1360           
  Lines        48172    48172           
  Branches     13296    13296           
========================================
  Hits         33513    33513           
  Misses       14659    14659           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[metamaskbot]

Builds ready [71bbf50]

• builds: chrome, firefox
• builds (beta): chrome
• builds (flask): chrome, firefox
• builds (MMI): chrome, firefox
• builds (test): chrome, firefox
• builds (test-flask): chrome, firefox
• build viz: Build System
• mv3: Background Module Init Stats
• mv3: UI Init Stats
• mv3: Module Load Stats
• mv3: Bundle Size Stats
• mv2: E2e Actions Stats
• code coverage: Report
• storybook: Storybook
• typescript migration: Dashboard
• all artifacts
• bundle viz:

  • background: 0, 1, 2, 3, 4, 5, 6
  • common: 0, 1, 2, 3, 4, 5, 6, 7, 8
  • content-script: 0
  • ui: 0, 1, 10, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (623 ± 464 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	60	133	88	21	10
		domContentLoaded	9	54	17	12	6
		load	48	2385	623	966	464
		domInteractive	9	54	17	12	6

Bundle size diffs

• background: 0 Bytes (0.00%)
• ui: 0 Bytes (0.00%)
• common: 0 Bytes (0.00%)

[Gudahtt]

LGTM!

[metamaskbot]

Builds ready [9b1c7cc]

• builds: chrome, firefox
• builds (beta): chrome
• builds (flask): chrome, firefox
• builds (MMI): chrome, firefox
• builds (test): chrome, firefox
• builds (test-flask): chrome, firefox
• build viz: Build System
• mv3: Background Module Init Stats
• mv3: UI Init Stats
• mv3: Module Load Stats
• mv3: Bundle Size Stats
• mv2: E2e Actions Stats
• code coverage: Report
• storybook: Storybook
• typescript migration: Dashboard
• all artifacts
• bundle viz:

  • background: 0, 1, 2, 3, 4, 5
  • common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
  • content-script: 0
  • offscreen: 0
  • ui: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (140 ± 187 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	62	94	80	10	5
		domContentLoaded	9	22	12	3	1
		load	40	1840	140	390	187
		domInteractive	9	22	12	3	1

Bundle size diffs

• background: 0 Bytes (0.00%)
• ui: 0 Bytes (0.00%)
• common: 0 Bytes (0.00%)

[Gudahtt]

LGTM!

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[metamaskbot]

Builds ready [07e81d8]

• builds: chrome, firefox
• builds (beta): chrome
• builds (flask): chrome, firefox
• builds (MMI): chrome, firefox
• builds (test): chrome, firefox
• builds (test-flask): chrome, firefox
• build viz: Build System
• mv3: Background Module Init Stats
• mv3: UI Init Stats
• mv3: Module Load Stats
• mv3: Bundle Size Stats
• mv2: E2e Actions Stats
• code coverage: Report
• storybook: Storybook
• typescript migration: Dashboard
• all artifacts
• bundle viz:

  • background: 0, 1, 2, 3, 4, 5
  • common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
  • content-script: 0
  • offscreen: 0
  • ui: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (186 ± 204 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	73	510	137	106	51
		domContentLoaded	10	96	30	20	10
		load	54	2018	186	425	204
		domInteractive	10	96	30	20	10

Bundle size diffs

• background: 0 Bytes (0.00%)
• ui: 0 Bytes (0.00%)
• common: 0 Bytes (0.00%)