test: flag Confirmations tests as Smoke tests

[seaona]

Description

In this PR we are flagging all the Confirmations tests as Smoke test. This includes the following spec files:

• advanced-gas (the only one that was previously flagged as Smoke)
• approve-default-erc
• eth-sign
• personal-sign
• send-erc20
• send-erc721
• send-eth
• typed-sign
• typed-sign-v3
• typed-sign-v4

Notice the amount of time added on the runs is not significant compared to the amount of scenarios we add, giving everyone more confidence on making changes in the Wallet. See numbers below.

Notice the tests are stable, being 100% Successful after several ci runs. This was achieved thanks to this fix by @vinistevam

Recent examples how Confirmations tests captured an error on main

• Here is an example of how running all the Confirmations tests, would have captured and prevent an error in main. Luckily we were running them locally and could spot the error and connect the dots with the slack thread.

• Here Signatures flow broken in main, that would have been captured by Confirmations tests if run as Smoke https://app.bitrise.io/build/fa65a941-6cd6-4b0c-9ece-151e76cdbb0d

Test Runs Data

Run #1 - Detox Build & Test

• ios: 16m 31s
• android: 36m 09s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/10633145-b793-4ac3-8d9b-316ca24586f8

Run #2 - Detox Build & Test

• ios: 18m 15s
• android: 34m 15s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/aad945da-60b3-49f2-ae89-18a6a59efe35

Run #3 - Detox Build & Test

• ios: 17m 30s
• android: 36m 12s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/a9bd1bd3-a23f-4535-9b21-8475ca790980

Run #4 - Detox Build & Test

• ios: 18m 47s
• android: 35m 58s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/3109490d-0355-4eb3-bbed-bedd644f26d9

Run #5 - Detox Build & Test

• ios: 16m 57s
• android: 32m 02s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/69a433eb-0bb0-4b67-95b2-c58145cc8963

Related issues

Fixes: #

Manual testing steps

Trigger ci test runs from Bitrise and see that all tests are passing.

Screenshots/Recordings

Before

After

Pre-merge author checklist

• I’ve followed MetaMask Coding Standards.
• I've clearly explained what problem this PR is solving and how it is solved.
• I've linked related issues
• I've included manual testing steps
• I've included screenshots/recordings if applicable
• I’ve included tests if applicable
• I’ve documented my code using JSDoc format if applicable
• I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.
• I’ve properly set the pull request status:
  • In case it's not yet "ready for review", I've set it to "draft".
  • In case it's "ready for review", I've changed it from "draft" to "non-draft".

Pre-merge reviewer checklist

• I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
• I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

[github-actions]

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (eb52137) 36.60% compared to head (9f996c5) 36.60%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7710   +/-   ##
=======================================
  Coverage   36.60%   36.60%           
=======================================
  Files        1092     1092           
  Lines       29178    29178           
  Branches     2678     2678           
=======================================
  Hits        10681    10681           
  Misses      17885    17885           
  Partials      612      612           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

E2E test started on Bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/e7e6164a-e4e8-4cce-8a08-5426e78286f3
You can also kick off another Bitrise E2E smoke test by removing and re-applying the (Run Smoke E2E) label

[cortisiko]

Can we remove the type-sign test from the smoke tag? It takes 5 mins for that one spec file to run compared to the other tests which take ~1 minute.

[seaona]

Can we remove the type-sign test from the smoke tag? It takes 5 mins for that one spec file to run compared to the other tests which take ~1 minute.

thanks for your comment @cortisiko . It's true that typed signature spec took a bit longer. We've re-arranged the tests so we can benefit from the parallelization and increase the speed for those.
Having Signatures included in the Smoke test pipeline is really important for Confirmations, so we'll like to re-evaluate it with the new timings (once we have the data).
Currently Signatures are broken in main, and that's something that could have been captured by these tests. We can re-run the jobs once that issue is fixed (now tests will fail) and post the new results here.

[cortisiko]

@seaona amazing, thanks for splitting the tests.

Currentlyhttps://github.com//issues/7920, and that's something that could have been captured by these tests.

Actually, the nightly regression test build (which includes these signature tests) caught the failing tests. They should be fixed in main now. Feel free to pull in main and rerun the tests.

[seaona]

After the test signatures change, all spec files are <= 2mins.

https://addons-testing.bitrise.io/builds/12a97b1c-bab3-4c01-9cbc-5b13e61eeb0a/testreport/2451de81-a95f-4afb-bfab-4e80fc52784b?status=passed

Run #7 - after Signatures Change - Detox Build & Test

• ios: currently failing due to issue with pods in main
• android: 30m 50s
• bitrise: https://app.bitrise.io/build/2cad43f1-9db3-4538-9eb5-8bd6cfe95dd0

Run #8 - after Signatures Change - Detox Build & Test

• ios: 17m 49s
• android: 36m 44s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/4d55eda4-31f5-440a-af3e-021df6df4594

Run #9 - after Signatures Change - Detox Build & Test

• ios: 17m 49s
• android: 36m 01s
• bitrise: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/e6908857-b75b-4c1c-8821-6f119863122a

[cortisiko]

PR looks good. Nit: Can you put the sign tests in a signatures folder?

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[seaona]

PR looks good. Nit: Can you put the sign tests in a signatures folder?

thank you @cortisiko done