Draft Statement of Work - Test flakiness lead

[mhdawson]

The test flakiness lead will be expected to:

• lead a test flakiness strategic initiative, rallying and supporting contributors who work to reduce flaky tests. This might include running regular test team meetings, documentation, tools, or whatever strategy works to achieve more than they can do on their own
• build tools and improve automation that allows the project to effectively manage flaky tests to reduce their impact on the CI
• Investigate and fix existing tests being marked as flaky in the status files

Duration

• 6 months

Success looks like

• reduced number of tests marked as flaky in the status files
• more CI runs that pass on the first run
• critical mass of contributors/collaborators dedicating time to addressing flaky tests that persists beyond the strategic initiative

[mhdawson]

@nodejs/tsc as discussed in the TSC meeting today a first cut at what a statement of work for a test flakiness lead might look like.

[mhdawson]

Adding to agenda so that we review/get feedback in a future meeting.

[joyeecheung]

I think we should explicitly add:

Investigate and fix existing tests being marked as flaky in the status files

Success looks like
...reduced number of tests marked as flaky in the status files

Otherwise this might just optimize towards marking all the tests as flaky and let them rot in the status files, which isn't ideal.

[mhdawson]

@joyeecheung updated, thanks for the suggestion.