Enable mono runtime to handle SIGTERM like CoreCLR #82806 #82813

nealef · 2023-03-01T05:33:53Z

Provide fix for #81093 - "Mono does not emit ProcessExit event on SIGTERM"

src/mono/mono/mini/mini-posix.c
- Add signal handler for SIGTERM
src/mono/mono/mini/mini-windows.c
- Add signal handler for SIGTERM
src/mono/mono/mini/mini-runtime.c
- Add mono_sigterm_signal_handler to process SIGTERM that will set a global variable to be monitored by the GC finalizer thread
src/mono/mono/mini/mini-runtime.h
- Define prototype for mono_sigterm_signal_handler()
src/mono/mono/metadata/gc.c
- Monitor for sigterm and kick off the shutdown process when encountered by calling mono_runtime_try_shutdown().
- Exit with either the user set exitcode (System.Environment.ExitCode) or SIGTERM + 128.

…on SIGTERM" * src/mono/mono/mini/mini-posix.c - Add signal handler for SIGTERM * src/mono/mono/mini/mini-windows.c - Add signal handler for SIGTERM * src/mono/mono/mini/mini-runtime.c - Add mono_sigterm_signal_handler to process SIGTERM that will set a global variable to be monitored by the GC finalizer thread * src/mono/mono/mini/mini-runtime.h - Define prototype for mono_sigterm_signal_handler() * src/mono/mono/metadata/gc.c - Monitor for sigterm and kick off the shutdown process when encountered by calling mono_runtime_try_shutdown(). - Exit with either the user set exitcode (System.Environment.ExitCode) or SIGTERM + 128.

vargaz · 2023-03-01T10:23:52Z

Users expect the process to terminate when it's sent a SIGTERM, and adding a handler for that could cause the process to get stuck if the managed code gets stuck etc.

nealef · 2023-03-02T02:11:03Z

Users expect the process to terminate when it's sent a SIGTERM, and adding a handler for that could cause the process to get stuck if the managed code gets stuck etc.

What does coreCLR do in this situation? It fields the SIGTERM and invokes managed code. How does it prevent getting stuck?

tmds · 2023-03-13T21:23:02Z

What does coreCLR do in this situation? It fields the SIGTERM and invokes managed code. How does it prevent getting stuck?

It doesn't.

The user code may prevent the application from terminating, for example by blocking the ProcessExit event.

SIGTERM is a friendly request for an app to terminate.

If it doesn't terminate in a timely fashion, the user can send it SIGKILL.

That's what a control process, like systemd will do.

@nealef can you remove the following line, which skips the SigTermExitCode test on Mono:

runtime/src/libraries/System.Runtime/tests/System/ExitCodeTests.Unix.cs

Line 19 in db21974

    
           [ActiveIssue("https://github.com/dotnet/runtime/issues/31656", TestRuntimes.Mono)]

- Re-enable ExitCodeTests for mono

tmds · 2023-03-14T04:13:48Z

The SigTermExitCode test is failing, because Environment.ExitCode changes aren't getting picked up.

I think the fix may be something like: call mono_environment_exitcode_set(128+SIGTERM) at the start of mono_sigterm_signal_handler, and call exit(mono_environment_exitcode_get() after mono_runtime_try_shutdown.

- Set a default exit code before starting the SIGTERM processing

src/mono/mono/mini/mini-runtime.c

src/mono/mono/metadata/gc.c

- Set a default exit code before setting the term_signaled variable that gets checked in gc

- Simplify use of exit code now that a default is being set

nealef · 2023-03-14T04:32:40Z

Fixed

tmds · 2023-03-14T04:34:41Z

Thanks.

Let's see if the test passes for all cases with these changes.

tmds · 2023-03-14T06:31:29Z

The test is passing for all cases. So it has the right behavior now concerning the ExitCode.

I'm not familiar enough with mono to approve the implementation.

lambdageek

LGTM. couple minor nits.

src/mono/mono/mini/mini-runtime.c

src/mono/mono/mini/mini-windows.c

src/mono/mono/metadata/gc.c

lambdageek · 2023-03-14T14:12:04Z

Not sure what's up with (Build mono minijit Pri0 Runtime Tests Run windows x64 release) - might be related?

ping @lateralusX

- Rename term_signaled to match mono style - Remove volatile attribute - Move testing of shutdown until after the sem wait * src/mono/mono/mini/mini-runtime.c - Rename term_signaled to mono_term_signaled * src/mono/mono/mini/mini-windows.c - Use the correct signal for handler

lateralusX · 2023-03-15T09:20:46Z

src/mono/mono/mini/mini-windows.c

@@ -189,6 +189,7 @@ mono_runtime_install_handlers (void)
 	win32_seh_set_handler(SIGFPE, mono_sigfpe_signal_handler);
 	win32_seh_set_handler(SIGILL, mono_crashing_signal_handler);
 	win32_seh_set_handler(SIGSEGV, mono_sigsegv_signal_handler);
+	win32_seh_set_handler(SIGTERM, mono_sigterm_signal_handler);


If this is going to work on Windows we also need to handle that signal in win32_seh_set_handler as well as react to the exception that will be generated and passed to seh_vectored_exception_handler.

Adding the case SIGTERM and the corresponding term_handler variable in exceptions_[x86|amd64] appears straightforward but how are they then used? I see there is exception handling for EXCEPTION_ILLEGAL_INSTRUCTION etc. where the handler is picked up and called, but how and where would this particular signal be fielded?

@lateralusX @lambdageek Looking for further guidance on the outstanding Windows' issue.

On windows there is no direct mapping with "SIGTERM", the closest we have is the probably SetConsoleCtrlHandler and mapping different scenarios like done by:

runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/PosixSignalRegistration.Windows.cs

Line 70 in 2327a36

private static Interop.BOOL HandlerRoutine(int dwCtrlType)

This is how its handled by CoreCLR:

runtime/src/coreclr/vm/ceemain.cpp

Line 605 in ffffc4b

::SetConsoleCtrlHandler(DbgCtrlCHandler, TRUE/*add*/);

So for Mono Windows handling of "SIGTERM" it probably need to setup a console ctrl handler and then react on the event as part of that handler and can't be handled through existing vectorized exception handling logic.

lateralusX · 2023-04-11T15:00:22Z

src/mono/mono/metadata/gc.c

@@ -892,6 +896,12 @@ finalizer_thread (gpointer unused)
 		}
 		wait = TRUE;

+		/* Just in case we've received a SIGTERM */
+		if (mono_term_signaled) {


Since this is updated from a different thread without any locking, maybe we would need a read memory barrier to make sure we see updated value, especially when wait == FALSE,. There might be one hidden in the mono_coop_sem_timedwait, looks like at least sem_trywait seems to issue memory barriers, and other implementations probably do as well (like WaitForSingleObjectEx), so maybe not needed in the end, but we depend on implementation details. I guess we can leave it as is for now, just wanted to make a comment/note around potential but adding a read barrier in that code path shouldn't be too dramatic.

I think that at one point, this PR had mono_term_signaled as volatile.

Maybe that (or something else) is needed to ensure these lines don't reorder, causing the wrong exit code to be picked up.

This is how the exit code is set:

mono_environment_exitcode_set(128+SIGTERM); /* Set default exit code */ mono_term_signaled = TRUE;

We can always do acquire/release semantics on mono_term_signaled and we would make sure this is not reordered by compiler or CPU. Just using volatile on mono_term_signaled will only prevent compiler to reorder load/store but CPU is still free to do load/store reorder.

nealef · 2024-03-21T04:29:24Z

Closing this as a new PR was generated that includes Windows has been created: #100056

nealef requested review from vargaz, lambdageek, SamMonoRT and thaystg as code owners March 1, 2023 05:33

ghost added the community-contribution Indicates that the PR has been added by a community member label Mar 1, 2023

dotnet-issue-labeler bot added the area-VM-meta-mono label Mar 1, 2023

build-analysis bot mentioned this pull request Mar 1, 2023

Tracking issue for CI build timeouts #76454

Closed

This was referenced Mar 1, 2023

System.Security.Cryptography.X509Certificates.Tests.ChainTests.BuildInvalidSignatureTwice failure #82837

Open

Some System.Buffers.Binary.Tests.ReverseEndiannessUnitTests tests broken on main #82868

Closed

* src/libraries/System.Runtime/tests/System/ExitCodeTests.Unix.cs

9ac0c3b

- Re-enable ExitCodeTests for mono

* src/mono/mono/mini/mini-runtime.c

98eeadf

- Set a default exit code before starting the SIGTERM processing

tmds reviewed Mar 14, 2023

View reviewed changes

src/mono/mono/mini/mini-runtime.c Show resolved Hide resolved

tmds reviewed Mar 14, 2023

View reviewed changes

src/mono/mono/metadata/gc.c Outdated Show resolved Hide resolved

nealef added 2 commits March 14, 2023 04:29

* src/mono/mono/mini/mini-runtime.c

38b5e87

- Set a default exit code before setting the term_signaled variable that gets checked in gc

* src/mono/mono/metadata/gc.c

3adb5d6

- Simplify use of exit code now that a default is being set

build-analysis bot mentioned this pull request Mar 14, 2023

IOException running NuGet-Migrations during tests in dotnet CLI first run #80619

Closed

lambdageek approved these changes Mar 14, 2023

View reviewed changes

src/mono/mono/mini/mini-runtime.c Outdated Show resolved Hide resolved

src/mono/mono/mini/mini-windows.c Outdated Show resolved Hide resolved

src/mono/mono/metadata/gc.c Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Mar 15, 2023

[release/6.0] Doublelinklist GC failures on Mono #83245

Closed

lateralusX reviewed Mar 15, 2023

View reviewed changes

lateralusX reviewed Apr 11, 2023

View reviewed changes

lambdageek self-assigned this Jan 22, 2024

nealef mentioned this pull request Mar 21, 2024

Provide fix for #81093 - "Mono does not emit ProcessExit event on SIGTERM" #100056

Merged

nealef closed this Mar 21, 2024

github-actions bot locked and limited conversation to collaborators Apr 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable mono runtime to handle SIGTERM like CoreCLR #82806 #82813

Enable mono runtime to handle SIGTERM like CoreCLR #82806 #82813

nealef commented Mar 1, 2023

vargaz commented Mar 1, 2023

nealef commented Mar 2, 2023

tmds commented Mar 13, 2023

tmds commented Mar 14, 2023

nealef commented Mar 14, 2023 via email

tmds commented Mar 14, 2023

tmds commented Mar 14, 2023

lambdageek left a comment

lambdageek commented Mar 14, 2023

lateralusX Mar 15, 2023 •

edited

Loading

nealef Mar 25, 2023

nealef Apr 6, 2023

lateralusX Apr 11, 2023 •

edited

Loading

lateralusX Apr 11, 2023 •

edited

Loading

tmds Apr 15, 2023

lateralusX Apr 17, 2023 •

edited

Loading

nealef commented Mar 21, 2024

Enable mono runtime to handle SIGTERM like CoreCLR #82806 #82813

Enable mono runtime to handle SIGTERM like CoreCLR #82806 #82813

Conversation

nealef commented Mar 1, 2023

vargaz commented Mar 1, 2023

nealef commented Mar 2, 2023

tmds commented Mar 13, 2023

tmds commented Mar 14, 2023

nealef commented Mar 14, 2023 via email

tmds commented Mar 14, 2023

tmds commented Mar 14, 2023

lambdageek left a comment

Choose a reason for hiding this comment

lambdageek commented Mar 14, 2023

lateralusX Mar 15, 2023 • edited Loading

Choose a reason for hiding this comment

nealef Mar 25, 2023

Choose a reason for hiding this comment

nealef Apr 6, 2023

Choose a reason for hiding this comment

lateralusX Apr 11, 2023 • edited Loading

Choose a reason for hiding this comment

lateralusX Apr 11, 2023 • edited Loading

Choose a reason for hiding this comment

tmds Apr 15, 2023

Choose a reason for hiding this comment

lateralusX Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

nealef commented Mar 21, 2024

lateralusX Mar 15, 2023 •

edited

Loading

lateralusX Apr 11, 2023 •

edited

Loading

lateralusX Apr 11, 2023 •

edited

Loading

lateralusX Apr 17, 2023 •

edited

Loading