Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: e2e replay test using Maestro (js) #4277

Merged
merged 11 commits into from
Nov 20, 2024
Merged

test: e2e replay test using Maestro (js) #4277

merged 11 commits into from
Nov 20, 2024

Conversation

vaind
Copy link
Collaborator

@vaind vaind commented Nov 14, 2024 •
edited
Loading

📢 Type of change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring

📜 Description

Adds integration tests for Replay by updating e2e tests to:

  • attach replays on errors (change the init config)
  • check that a replay is created on sentry.io

During development, I've found out it's possible to launch javascript from maestro directly, which reduces development iteration time (make a change in tests and see the result) from about 9 minutes to 20 seconds, because we don't actually need to rebuild the whole app when making a change to test assertions.

Note: android doesn't seem to capture replays in CI so it's disabled.
Note 2: metrics failure isn't related IMO

💡 Motivation and Context

Replaces #3993

💚 How did you test it?

📝 Checklist

  • I reviewed submitted code
  • I added tests to verify changes
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
  • All tests passing
  • No breaking changes

🔮 Next steps

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

Android (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 399.83 ms 469.78 ms 69.95 ms
Size 7.15 MiB 8.35 MiB 1.21 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
d197b5c+dirty 258.75 ms 313.61 ms 54.86 ms
8900e1a+dirty 371.40 ms 377.70 ms 6.31 ms
3ffcddd+dirty 244.82 ms 295.47 ms 50.65 ms
ad6c299+dirty 336.47 ms 362.89 ms 26.42 ms
8ae23a7+dirty 398.10 ms 411.48 ms 13.38 ms
a5d86e1+dirty 380.86 ms 466.42 ms 85.56 ms
d2c32bb+dirty 445.45 ms 497.85 ms 52.41 ms
0d3e677+dirty 384.24 ms 431.45 ms 47.21 ms
d7401ac+dirty 373.98 ms 394.08 ms 20.10 ms
27ef4ee+dirty 296.71 ms 351.00 ms 54.29 ms

App size

Revision Plain With Sentry Diff
d197b5c+dirty 7.15 MiB 8.09 MiB 962.72 KiB
8900e1a+dirty 7.15 MiB 8.03 MiB 901.79 KiB
3ffcddd+dirty 7.15 MiB 8.04 MiB 912.17 KiB
ad6c299+dirty 7.15 MiB 8.04 MiB 912.17 KiB
8ae23a7+dirty 7.15 MiB 8.35 MiB 1.20 MiB
a5d86e1+dirty 7.15 MiB 8.35 MiB 1.20 MiB
d2c32bb+dirty 7.15 MiB 8.35 MiB 1.20 MiB
0d3e677+dirty 7.15 MiB 8.35 MiB 1.20 MiB
d7401ac+dirty 7.15 MiB 8.04 MiB 910.85 KiB
27ef4ee+dirty 7.15 MiB 8.08 MiB 959.49 KiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

iOS (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1223.25 ms 1226.57 ms 3.32 ms
Size 2.36 MiB 3.10 MiB 753.35 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
c639edf+dirty 1236.18 ms 1235.04 ms -1.14 ms
86d6d2c+dirty 1267.55 ms 1286.21 ms 18.66 ms
2534337+dirty 1225.08 ms 1230.26 ms 5.17 ms
4a6664f+dirty 1209.49 ms 1208.63 ms -0.86 ms
d7401ac+dirty 1252.38 ms 1275.04 ms 22.66 ms
76d1baf+dirty 1244.10 ms 1268.52 ms 24.42 ms
9433f35+dirty 1246.94 ms 1271.45 ms 24.52 ms
0d3e677+dirty 1214.39 ms 1225.70 ms 11.31 ms
728164b+dirty 1256.10 ms 1259.08 ms 2.98 ms
abb7058+dirty 1255.42 ms 1268.86 ms 13.44 ms

App size

Revision Plain With Sentry Diff
c639edf+dirty 2.36 MiB 3.08 MiB 736.63 KiB
86d6d2c+dirty 2.36 MiB 2.82 MiB 462.82 KiB
2534337+dirty 2.36 MiB 2.88 MiB 525.47 KiB
4a6664f+dirty 2.36 MiB 3.04 MiB 696.39 KiB
d7401ac+dirty 2.36 MiB 2.83 MiB 481.14 KiB
76d1baf+dirty 2.36 MiB 2.82 MiB 469.45 KiB
9433f35+dirty 2.36 MiB 2.85 MiB 499.80 KiB
0d3e677+dirty 2.36 MiB 3.10 MiB 753.12 KiB
728164b+dirty 2.36 MiB 2.88 MiB 530.38 KiB
abb7058+dirty 2.36 MiB 2.87 MiB 520.42 KiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

iOS (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1227.94 ms 1229.04 ms 1.10 ms
Size 2.92 MiB 3.66 MiB 758.63 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
c639edf+dirty 1223.63 ms 1227.98 ms 4.35 ms
86d6d2c+dirty 1291.62 ms 1296.80 ms 5.18 ms
2534337+dirty 1220.87 ms 1221.47 ms 0.60 ms
4a6664f+dirty 1218.77 ms 1221.07 ms 2.30 ms
d7401ac+dirty 1288.10 ms 1289.54 ms 1.44 ms
76d1baf+dirty 1245.00 ms 1257.76 ms 12.76 ms
9433f35+dirty 1232.24 ms 1232.74 ms 0.50 ms
0d3e677+dirty 1239.02 ms 1241.22 ms 2.20 ms
728164b+dirty 1280.06 ms 1285.26 ms 5.20 ms
abb7058+dirty 1260.28 ms 1266.56 ms 6.28 ms

App size

Revision Plain With Sentry Diff
c639edf+dirty 2.92 MiB 3.64 MiB 742.55 KiB
86d6d2c+dirty 2.92 MiB 3.37 MiB 464.31 KiB
2534337+dirty 2.92 MiB 3.43 MiB 529.76 KiB
4a6664f+dirty 2.92 MiB 3.60 MiB 702.09 KiB
d7401ac+dirty 2.92 MiB 3.40 MiB 488.06 KiB
76d1baf+dirty 2.92 MiB 3.38 MiB 475.74 KiB
9433f35+dirty 2.92 MiB 3.41 MiB 503.55 KiB
0d3e677+dirty 2.92 MiB 3.66 MiB 758.42 KiB
728164b+dirty 2.92 MiB 3.44 MiB 533.26 KiB
abb7058+dirty 2.92 MiB 3.43 MiB 524.53 KiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 15, 2024 •
edited
Loading

Android (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 431.70 ms 433.49 ms 1.79 ms
Size 17.74 MiB 20.08 MiB 2.35 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
d197b5c+dirty 338.94 ms 354.87 ms 15.93 ms
0db0c72 372.12 ms 386.00 ms 13.88 ms
8900e1a+dirty 430.68 ms 456.13 ms 25.44 ms
5571a20 410.55 ms 441.06 ms 30.51 ms
80b2ce3 385.02 ms 387.36 ms 2.34 ms
a5d86e1 465.06 ms 457.83 ms -7.23 ms
7bc4d75 488.76 ms 473.28 ms -15.48 ms
34aba08 328.10 ms 342.84 ms 14.74 ms
52c0562 453.04 ms 434.71 ms -18.33 ms
148f924 492.65 ms 500.28 ms 7.63 ms

App size

Revision Plain With Sentry Diff
d197b5c+dirty 17.73 MiB 20.04 MiB 2.31 MiB
0db0c72 17.73 MiB 19.75 MiB 2.02 MiB
8900e1a+dirty 17.73 MiB 19.75 MiB 2.01 MiB
5571a20 17.73 MiB 19.93 MiB 2.19 MiB
80b2ce3 17.73 MiB 19.75 MiB 2.02 MiB
a5d86e1 17.74 MiB 20.08 MiB 2.34 MiB
7bc4d75 17.74 MiB 20.08 MiB 2.34 MiB
34aba08 17.73 MiB 19.80 MiB 2.07 MiB
52c0562 17.73 MiB 20.11 MiB 2.38 MiB
148f924 17.73 MiB 19.94 MiB 2.21 MiB

@vaind vaind force-pushed the test/e2e-replay-maestro-js branch from b1d5b9e to 425b974 Compare November 17, 2024 15:42
@vaind vaind marked this pull request as ready for review November 17, 2024 16:31
@vaind vaind mentioned this pull request Nov 17, 2024
Closed
9 tasks
krystofwoldrich
krystofwoldrich approved these changes Nov 20, 2024
View reviewed changes
Copy link
Member

@krystofwoldrich krystofwoldrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for exploring the graaljs,

this is good improvement to the maestro mobile tests.

🚀

@krystofwoldrich krystofwoldrich merged commit 4cfa8f3 into main Nov 20, 2024
62 checks passed
@krystofwoldrich krystofwoldrich deleted the test/e2e-replay-maestro-js branch November 20, 2024 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krystofwoldrich krystofwoldrich krystofwoldrich approved these changes

@lucas-zimerman lucas-zimerman Awaiting requested review from lucas-zimerman lucas-zimerman is a code owner

@antonis antonis Awaiting requested review from antonis antonis is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vaind @krystofwoldrich