Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests(mocha): fix snapshot expectations in retries #15735

Merged
merged 3 commits into from
Jan 8, 2024
Merged

Conversation

adamraine
Copy link
Member

@adamraine adamraine commented Jan 8, 2024

Noticed that snapshot tests were passing where they should be failing in CI (#15730 (comment))

If a test is retried, the snapshot expectation indices will start where the previous attempt left off. This leads to jest-snapshot treating the second attempt as a "new" snapshot which we automatically treat as passing because of the updateSnapshot: 'new' setting.

This PR uses updateSnapshot: 'none' by default so that CI will never accidentally pass. This means we have to use -u to set an initial snapshot for new tests, but that seems like a small price to pay.

However, mocha will only report the last failure attempt if a test is retried, which will now by a "snapshot does not exist" failure instead of a helpful snapshot diff. To fix this, this PR also resets the snapshot state between each test.

@adamraine adamraine requested a review from a team as a code owner January 8, 2024 18:17
@adamraine adamraine requested review from connorjclark and removed request for a team January 8, 2024 18:17
@@ -258,6 +258,7 @@ Array [
"largest-contentful-paint",
"largest-contentful-paint-element",
"layout-shift-elements",
"layout-shifts",
Copy link
Member Author

@adamraine adamraine Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These snapshot updates should have been caught in #15703

Confirmed they are caught here now:
https://github.com/GoogleChrome/lighthouse/actions/runs/7451657749/job/20273235889?pr=15735

Copy link
Collaborator

@connorjclark connorjclark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice find!

@adamraine adamraine merged commit 9c09193 into main Jan 8, 2024
27 checks passed
@adamraine adamraine deleted the fix-snapshot-retry branch January 8, 2024 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants