Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix brittle flaky tests in core/test_io.py #243

Closed
wants to merge 1 commit into from

Conversation

blazyy
Copy link

@blazyy blazyy commented Oct 20, 2022

Four of the tests inside the fooof/tests/core/test_io.py module were found to be flaky (specifically, something called brittle flakiness), when using the pytest-flakefinder plugin. A flaky test is a test that both passes and fails despite no changes to the code or the test itself. Brittle flakiness (defined as per iFixFlakies) is when a test runs fine when run in conjunction with the other tests in the same module but fails when run on its own.

To reproduce:

pytest fooof/tests/core/test_io.py::test_load_json_str
pytest fooof/tests/core/test_io.py::test_load_json_fobj
pytest fooof/tests/core/test_io.py::test_load_jsonlines
pytest fooof/tests/core/test_io.py::test_load_file_contents

All other tests inside test_io.py ran fine when run on their own. But when the tests above were run on their own, they failed, due to some files not existing. When the entire module is run, this behavior is not seen. To fix, I just added the appropriate function calls which creates the JSON files required by the tests above.

I'm aware that the tests might not be run on their own, but this makes the entire module more robust so it's just a small fix for you to consider.

NOTE: This PR contains similar changes to my previously opened PR. The difference is that the tests in this PR were from a different module and were found at a later time.

@TomDonoghue
Copy link
Member

Hey @blazyy - I appreciate you looking through the module, but for this topic specifically I don't find it to be an issue that some of the tests is "flaky" - these tests are always run together, and the save / load tests do depend on each other, which makes each test more specific to each task. In general, if you have a suggested edit to a repo, it's often worth opening an issue to first check if the maintainers are interested in the update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants