Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off flakey Firefox tests #473

Merged
merged 3 commits into from
May 2, 2024
Merged

Conversation

tunetheweb
Copy link
Member

Makes progress on #472 by commenting out the frequently failing tests.

Copy link
Member

@philipwalton philipwalton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's better to comment out particular failures or just disable FF tests until this is fixed? My concern is it will be very easy to forget they've been commented out.

@tunetheweb
Copy link
Member Author

Do you think it's better to comment out particular failures or just disable FF tests until this is fixed? My concern is it will be very easy to forget they've been commented out.

I did just comment out the asserts that failed (the rating checks). Annoyingly the LCP one that's failing is in the assertStandardReportsAreCorrect function used by a lot of the tests so it lost that assert for all of them but I preferred that to pulling that code out for that one test, or disabling the whole test.

I still think that's better to keep most of the Firefox tests working. I plan on leaving #472 open until this is fixed by Firefox as a reminder (hence why it's "makes progress on" rather than "fixes" in the initial comment which would have closed it).

Ultimately I'm hoping this is a short term issue and will be fixed on next version, but getting bored of seeing the failures and worried we will miss a REAL failure because of this.

@philipwalton
Copy link
Member

Gotcha. Keeping the issue open as a reminder to address it eventually SGTM.

@tunetheweb tunetheweb merged commit b9547e3 into v4 May 2, 2024
6 checks passed
@tunetheweb tunetheweb deleted the workaround-firefox-flakiness branch May 2, 2024 19:37
@tunetheweb tunetheweb mentioned this pull request Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants