[tests] Fix flaky tests #1466
Merged
[tests] Fix flaky tests #1466
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nan-li
force-pushed
the
tests_fix_flaky_tests
branch
7 times, most recently
from
ce22a6c
to
b70faf7
Compare
* Add new framework called `OneSignalOSCoreMocks ` to access the Operation Repo and reset it between tests. * Part of the cause for flaky tests is state carrying over in the Operation Repo singleton. For example, Deltas that have not been flushed yet from the preceding test are flushed in the next one.
* These tests check the mock client's number of requests executed, but it was counting the user setup request as well (such as Fetch Identity By Subscription). Add a waiting period for this before creating the Live Activities requests) * An alternative is to set the user and push subscription ID manually without triggering a request to be made.
nan-li
force-pushed
the
tests_fix_flaky_tests
branch
2 times, most recently
from
e16237e
to
9a28a65
Compare
* Instead of resetting after tests, reset before tests for cleanest starting state
nan-li
force-pushed
the
tests_fix_flaky_tests
branch
from
3a9f44a
to
46771e1
Compare
* Currently the Operation Repo is a singleton and shared between tests, so a pending flush could be on the DispatchQueue from a previous test and it will flush unexpectedly. * This test tests combining operations and intentionally increases the `pollIntervalMilliseconds` to `300`, but a previous flush could interrupt this.
* The UnitTestApp which runs all tests now has 2 test plans named "Full" and "Reduced" * The default is the Full test * We may want to disable certain long-running tests from running in the CI under the Reduced plan * There are no tests disabled in this commit, only adding 2 identical test plans
* Under the Reduced Test Plan, concurrency tests are disabled. * These tests spam calls and can cause carry over to following tests
* Fix bug that made the helper method to clear user defaults between tests return early.
nan-li
force-pushed
the
tests_fix_flaky_tests
branch
from
aaf9ba1
to
f79af94
Compare
|
NVM, that commit made it worse, removed it. |
nan-li
force-pushed
the
tests_fix_flaky_tests
branch
from
9ddc5f1
to
f79af94
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
One Line Summary
Fix flaky tests so the CI can be meaningful.
Details
Add
OneSignalOSCoreMocksmoduleIntroduce 2 separate Test Plans
UnitTestApp_TestPlan_Fullwhich is the Default test plan and can be run in Xcode, and aUnitTestApp_TestPlan_Reducedwith concurrency tests disabled that is run by the CI script.Other Misc Changes
Motivation
Fix flaky tests so the CI can be meaningful.
Scope
Tests
Testing
Unit testing
Manual testing
Minimal
Affected code checklist
Checklist
Overview
Testing
Final pass
This change is