Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: resolve flaky test (macos-14, 22, npm ci, 1) #5984

Closed
wants to merge 11 commits into from

Conversation

mrstork
Copy link
Contributor

@mrstork mrstork commented Dec 31, 2024

Summary

Resolves the ava timeouts that have been making our test suite flaky for quite some time now:

  • Failing tests were in (macos-14, 22, npm ci, 1)
  • Error message was a generic Timed out while running tests message from ava
  • Error was extremely rare when running tests locally
  • Doesn't happen on any other operating system

Predominately followed discussion in https://github.com/avajs/ava/discussions/3036 to get to a resolution

Copy link
Contributor

This pull request adds or modifies JavaScript (.js, .cjs, .mjs) files.
Consider converting them to TypeScript.

@mrstork mrstork changed the title test: compile code prior to running tests test: flaky tests - specify tsc as compiler for ava tests Dec 31, 2024
@mrstork mrstork force-pushed the investigate-flaky-tests branch from 9c6faa7 to 4227a74 Compare December 31, 2024 17:24
@mrstork mrstork changed the title test: flaky tests - specify tsc as compiler for ava tests test: wip - flaky tests investigation Dec 31, 2024
@mrstork mrstork force-pushed the investigate-flaky-tests branch 4 times, most recently from 84e934a to 081aca9 Compare December 31, 2024 18:49
@mrstork mrstork changed the title test: wip - flaky tests investigation test: resolve flaky test (macos-14, 22, npm ci, 1) Dec 31, 2024
@mrstork mrstork marked this pull request as ready for review December 31, 2024 19:05
@mrstork mrstork requested a review from a team as a code owner December 31, 2024 19:05
@mrstork mrstork marked this pull request as draft December 31, 2024 19:06
@mrstork mrstork force-pushed the investigate-flaky-tests branch from ceb8120 to 2bbffdf Compare December 31, 2024 19:46
@mrstork mrstork marked this pull request as ready for review December 31, 2024 19:51
@mrstork mrstork marked this pull request as draft December 31, 2024 19:54
@mrstork mrstork force-pushed the investigate-flaky-tests branch 2 times, most recently from c599b49 to f5e0315 Compare December 31, 2024 21:10
@mrstork mrstork force-pushed the investigate-flaky-tests branch from f5e0315 to 543ecc5 Compare December 31, 2024 21:54
repositoryRoot: packages/build/tests/install/fixtures/yarn_ci␊
testOpts:␊
pluginsListUrl: test␊
silentLingeringProcesses: true␊
useBinary: true␊
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useBinary was being passed in as a flag instead of altering test setup

@@ -368,7 +368,7 @@ test('Plugin errors that occur during the loading phase are piped to system logs
t.snapshot(normalizeOutput(output))
})

test('Plugins have a pre-populated Blobs context', async (t) => {
test.serial('Plugins have a pre-populated Blobs context', async (t) => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to parallelism, sometimes when calling getPort() by the time the port is used by the test it has been claimed by something else. Making the test run serially is a workaround for this.

@mrstork mrstork closed this Jan 2, 2025
@mrstork
Copy link
Contributor Author

mrstork commented Jan 2, 2025

Wasn't able to get to a resolution. I believe upgrading ava to v5 will help, but we're currently blocked on that due to needing to support node 14

@mrstork mrstork deleted the investigate-flaky-tests branch January 2, 2025 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant