ZTS fails to detect killed processes. #9003

AttilaFueloep · 2019-07-08T00:48:44Z

Motivation and Context

If a process gets killed by a signal ksh(1) returns 256+signum. log_mustnot was expecting a return value of 128+signum, failing to detect killed processes. This resulted in test passing where they should not.

Description

How Has This Been Tested?

Verified that tests that cause segfaults passed without this change and failed, as they should, with this change.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the ZFS on Linux code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
All new and existing tests passed.
All commit messages are properly formatted and contain Signed-off-by.

AttilaFueloep · 2019-07-08T14:34:37Z

The tests failing is expected. Once #9001 is merged they should pass again.

By the way, how can I see which tests failed, there is no obvious link on the "Details" page.

behlendorf · 2019-07-08T17:16:41Z

Thanks for running this down. I'd wondered about this in the past but never investigated further. According the ksh man pages for both illumos and Linux, 256+signum is the expected for abnormal termination. So I'm not sure why this was ever set to 128.

       If it terminates abnormally, its value is 256+signum. The name of the signal
       corresponding  to  the exit  status can be obtained by way of the -l option of
       the kill built-in utility.

By the way, how can I see which tests failed, there is no obvious link on the "Details" page.

You're going to want to click on the summary link for the ZTS results. This is a summarized version of the full log which lists which tests failed and which (if any) of those failures were unexpected. It will also include full extracts for those specific test failures. For example:

Tests with results other than PASS that are unexpected:
    FAIL cli_root/zfs_send/zfs_send-b (expected PASS)
    FAIL cli_root/zfs_send/zfs_send_004_neg (expected PASS)
    FAIL slog/slog_replay_volume (expected PASS)

Once #9001 is merged they should pass again.

That makes sense. Once #9001 is merged will want to get this change rebased to verify all is well.

behlendorf · 2019-07-08T17:22:40Z

tests/test-runner/include/logapi.shlib

@@ -199,11 +199,11 @@ function log_neg_expect
 		print -u2 $($out)
 		_printerror "$@" "unexpectedly exited $status (File not found)"
 	# bus error - core dump


nit: let's make it explicit where 263 (and 267) come from in the comments. Perhaps something like this.

# bus error - core dump (256+signal, SIGBUS=7)

AttilaFueloep · 2019-07-08T19:10:38Z

So I'm not sure why this was ever set to 128.

Maybe this was run by sh or bash formerly. They use 128+signum.

Thanks for the link! I will address your review and rebase once #9001 is merged.

behlendorf · 2019-07-08T22:29:36Z

#9001 was merged so feel free to rebase this when convenient.

AttilaFueloep · 2019-07-08T22:51:23Z

Squashed and rebased. Let's see how the tests go.

richardelling · 2019-07-08T23:10:33Z

For the archives, ksh was the de-facto shell at Sun at the time. bash was not guaranteed to be available and sh is simply too limited, left there for backwards compatibility.

That said, it was well known that 256 + errno is expected, thanks for tracking this down.

log_neg_expect was using the wrong exit status to detect if a process got killed by SIGSEGV or SIGBUS, resulting in false positives. Signed-off-by: Attila Fülöp <attila@fueloep.org>

AttilaFueloep · 2019-07-09T10:44:05Z

Sorry, I've to force push again to fix the commit message. There is a 73 characters wide line and a typo.

codecov · 2019-07-09T10:46:09Z

Codecov Report

Merging #9003 into master will increase coverage by 0.21%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9003      +/-   ##
==========================================
+ Coverage    78.5%   78.72%   +0.21%     
==========================================
  Files         401      401              
  Lines      120145   120145              
==========================================
+ Hits        94322    94580     +258     
+ Misses      25823    25565     -258

Flag	Coverage Δ
#kernel	`79.49% <ø> (+0.1%)`	⬆️
#user	`66.57% <ø> (+0.34%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d20b76...7a8b9a2. Read the comment docs.

behlendorf · 2019-07-09T16:42:27Z

The test run encountered a surprising number of of rare test cases failures (4 total, one per bot). While unlikely, we have seen each of the failures at least once before. I've resubmitted those builds to repeat the testing.

AttilaFueloep · 2019-07-10T16:06:17Z

With the exception of the kernel.org build failure, which I think is unrelated, everything else seems to be fine now.

behlendorf · 2019-07-10T18:43:32Z

@AttilaFueloep looks good, I'll get this merged. The kernel.org failures are due to the 5.3 merge window now being open, so we'll need to see what's changed and sort that out.

log_neg_expect was using the wrong exit status to detect if a process got killed by SIGSEGV or SIGBUS, resulting in false positives. Reviewed-by: loli10K <ezomori.nozomu@gmail.com> Reviewed by: John Kennedy <john.kennedy@delphix.com> Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Attila Fülöp <attila@fueloep.org> Closes openzfs#9003

log_neg_expect was using the wrong exit status to detect if a process got killed by SIGSEGV or SIGBUS, resulting in false positives. Reviewed-by: loli10K <ezomori.nozomu@gmail.com> Reviewed by: John Kennedy <john.kennedy@delphix.com> Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Attila Fülöp <attila@fueloep.org> Closes #9003

behlendorf requested a review from jwk404 July 8, 2019 17:16

behlendorf added the Status: Code Review Needed Ready for review and testing label Jul 8, 2019

behlendorf reviewed Jul 8, 2019

View reviewed changes

jwk404 approved these changes Jul 8, 2019

View reviewed changes

AttilaFueloep force-pushed the fix-tests branch from c6f4ce6 to 03e4ec9 Compare July 8, 2019 22:49

behlendorf approved these changes Jul 9, 2019

View reviewed changes

Fix ZTS killed processes detection.

7a8b9a2

log_neg_expect was using the wrong exit status to detect if a process got killed by SIGSEGV or SIGBUS, resulting in false positives. Signed-off-by: Attila Fülöp <attila@fueloep.org>

AttilaFueloep force-pushed the fix-tests branch from 03e4ec9 to 7a8b9a2 Compare July 9, 2019 10:45

behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Jul 9, 2019

loli10K approved these changes Jul 9, 2019

View reviewed changes

behlendorf merged commit c3fba90 into openzfs:master Jul 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZTS fails to detect killed processes. #9003

ZTS fails to detect killed processes. #9003

AttilaFueloep commented Jul 8, 2019

AttilaFueloep commented Jul 8, 2019

behlendorf commented Jul 8, 2019

behlendorf Jul 8, 2019

AttilaFueloep Jul 8, 2019

AttilaFueloep commented Jul 8, 2019

behlendorf commented Jul 8, 2019

AttilaFueloep commented Jul 8, 2019 •

edited

Loading

richardelling commented Jul 8, 2019

AttilaFueloep commented Jul 9, 2019

codecov bot commented Jul 9, 2019 •

edited

Loading

behlendorf commented Jul 9, 2019

AttilaFueloep commented Jul 10, 2019

behlendorf commented Jul 10, 2019

ZTS fails to detect killed processes. #9003

ZTS fails to detect killed processes. #9003

Conversation

AttilaFueloep commented Jul 8, 2019

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

AttilaFueloep commented Jul 8, 2019

behlendorf commented Jul 8, 2019

behlendorf Jul 8, 2019

Choose a reason for hiding this comment

AttilaFueloep Jul 8, 2019

Choose a reason for hiding this comment

AttilaFueloep commented Jul 8, 2019

behlendorf commented Jul 8, 2019

AttilaFueloep commented Jul 8, 2019 • edited Loading

richardelling commented Jul 8, 2019

AttilaFueloep commented Jul 9, 2019

codecov bot commented Jul 9, 2019 • edited Loading

Codecov Report

behlendorf commented Jul 9, 2019

AttilaFueloep commented Jul 10, 2019

behlendorf commented Jul 10, 2019

AttilaFueloep commented Jul 8, 2019 •

edited

Loading

codecov bot commented Jul 9, 2019 •

edited

Loading