-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix incorrect test termination condition in TestCommand subclasses. #7428
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The logic in NextTest was doing the following: 1. Start the test at mTestIndex. 2. Increment mTestIndex. 3. If mTestCount == mTestIndex (i.e. no more tests to run), start shutdown (by calling SetCommandExitStatus, which unblocks the shutdown sequence).. This failed when running the last test: we would start the test, increment mTestIndex, then hit that mTestCount condition and immediately start shutdown, before waiting for the last test to complete. The fix is to test for the "no more tests to run" condition on entry to NextTest. This way the last test runs compeletely, and when it's done and calls NextTest we go ahead and shut down.
pullapprove
bot
requested review from
andy31415,
chrisdecenzo,
Damian-Nordic,
hawk248,
jepenven-silabs,
msandstedt and
woody-apple
June 7, 2021 17:57
woody-apple
approved these changes
Jun 7, 2021
msandstedt
approved these changes
Jun 7, 2021
bzbarsky-apple
added a commit
to bzbarsky-apple/connectedhomeip
that referenced
this pull request
Jun 7, 2021
This change depends on project-chip#7428 and project-chip#7430 which fix the problems properly. Fixes project-chip#7408 Fixes project-chip#7409 Fixes project-chip#7410
andy31415
approved these changes
Jun 8, 2021
bzbarsky-apple
added a commit
to bzbarsky-apple/connectedhomeip
that referenced
this pull request
Jun 8, 2021
This change depends on project-chip#7428 and project-chip#7430 which fix the problems properly. Fixes project-chip#7408 Fixes project-chip#7409 Fixes project-chip#7410
bzbarsky-apple
added a commit
that referenced
this pull request
Jun 9, 2021
nikita-s-wrk
pushed a commit
to nikita-s-wrk/connectedhomeip
that referenced
this pull request
Sep 23, 2021
…roject-chip#7428) The logic in NextTest was doing the following: 1. Start the test at mTestIndex. 2. Increment mTestIndex. 3. If mTestCount == mTestIndex (i.e. no more tests to run), start shutdown (by calling SetCommandExitStatus, which unblocks the shutdown sequence).. This failed when running the last test: we would start the test, increment mTestIndex, then hit that mTestCount condition and immediately start shutdown, before waiting for the last test to complete. The fix is to test for the "no more tests to run" condition on entry to NextTest. This way the last test runs compeletely, and when it's done and calls NextTest we go ahead and shut down.
nikita-s-wrk
pushed a commit
to nikita-s-wrk/connectedhomeip
that referenced
this pull request
Sep 23, 2021
…roject-chip#7434) This change depends on project-chip#7428 and project-chip#7430 which fix the problems properly. Fixes project-chip#7408 Fixes project-chip#7409 Fixes project-chip#7410
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The logic in NextTest was doing the following:
calling SetCommandExitStatus, which unblocks the shutdown sequence)..
This failed when running the last test: we would start the test,
increment mTestIndex, then hit that mTestCount condition and
immediately start shutdown, before waiting for the last test to
complete.
The fix is to test for the "no more tests to run" condition on entry
to NextTest. This way the last test runs compeletely, and when it's
done and calls NextTest we go ahead and shut down.
Problem
Last test in a TestCommand was not being allowed to actually complete.
Change overview
Wait for it to complete.
Testing
Unfortunately, nothing in our CI right now catches the "test did not actually complete" situation. I was considering adding some more machinery around this, but wasn't quite sure how best to do that.
I did test that removing some of our raciness workarounds causes
scripts/tests/test_suites.sh
to crash ~10% of the time without this fix even if I ensure that shutdown is somewhat synchronized with message processing, because of the following sequence of events:With this fix those crashes disappeared.
Any suggestions on better tests here would be much appreciated, of course, but it's a bit hard to test bugs in the test harness... ;)