Cooperative signal handling

[maxfischer2781]

• I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.

---

This PR adjust the signal handling of uvicorn.server.Server to integrate with asyncio applications by not suppressing shutdown signals. Major changes include:

• Original signal handlers are restored before Server.serve finishes
• When terminated by a signal, the appropriate original signal handler is invoked if possible

In the usual case, this allows CTRL+C to end both uvicorn and the containing async application. Closes #1579.

---

There is a caveat for tests in that signal handling on Windows is not as precise as on Unix (see e.g. Stackoverflow: How to handle a signal.SIGINT on a Windows OS machine?). For testing, I could not setup a situation where the tests receive a signal without the test environment also being interrupted by it.

[JoanFM]

Hey @maxfischer2781 ,

I wonder if your PR would make this PR #1768 unnecessary?

[JoanFM]

Hey @maxfischer2781 ,

I wonder if your PR would make this PR #1768 unnecessary?

Yes, I think ur implementation is better and involves less magic

[JoanFM]

It may be nice to add a test covering SIGTERM signal handling?

[maxfischer2781]

Hey @maxfischer2781 ,

I wonder if your PR would make this PR #1768 unnecessary?

@JoanFM Possibly, but the two changes have slightly different behaviour. This PR will trigger the original handler after the univocrn shutdown procedure. #1768 will trigger the original handler before the uvicorn shutdown procedure.

As a default, I think triggering the original handlers late is better – many handlers shutdown quickly (namely, the default handlers) and would skip the uvicorn shutdown entirely. It might be worth having a method to chain handlers explicitly, though.

[JoanFM]

Yes, plus your PR does not use any private function and would work with more loops.

One question I have, will this PR respect the signals passed to an event loop as:

loop.add_signal_handler(...)

instead of

signal.signal(...)

?

[maxfischer2781]

One question I have, will this PR respect the signals passed to an event loop as:

loop.add_signal_handler(...)

instead of

signal.signal(...)

?

Sadly it won't. There is no clean way to query the loop for registered handlers – handlers are stored in a private attribute of the loop and methods allow only to set or delete handlers by signal number.

[JoanFM]

One question I have, will this PR respect the signals passed to an event loop as:

loop.add_signal_handler(...)

instead of

signal.signal(...)

?

Sadly it won't. There is no clean way to query the loop for registered handlers – handlers are stored in a private attribute of the loop and methods allow only to set or delete handlers by signal number.

Okey, but signal.signal should work the same except if I need to interact with event loop itself.

Any idea when this might be merged?

[Kludex]

Any idea when this might be merged?

Is it me, or this question sounds a bit presumptious? 😅

Thanks for the PR @maxfischer2781 , and sorry the delay. This requires a bit of my focus, as there are many ideias related around.

I'll check the whole signal thingy for 0.23.0. I'm about to release 0.22.0.

No estimation can be provided, since I work on uvicorn on my free time.

[JoanFM]

Any idea when this might be merged?

Is it me, or this question sounds a bit presumptious? 😅

Thanks for the PR @maxfischer2781 , and sorry the delay. This requires a bit of my focus, as there are many ideias related around.

I'll check the whole signal thingy for 0.23.0. I'm about to release 0.22.0.

No estimation can be provided, since I work on uvicorn on my free time.

Sorry @Kludex, No bad intentions in this question. I really appreciate the work you guys do and understand it is not easy to dedicate time.

I just do not know how this specific project worked.

Thanks for everything

[Kludex]

Does this solve #1301 as well?

Ok, review done. Happy new year! 😎

[Kludex]

Relevant: #1708

[maxfischer2781]

@Kludex I've addressed/implemented all suggestions now. Please take another look once you find the time.

I don't think an approach as suggested in #1708 makes sense in uvicorn: Practically, signal handling is a two-step approach of 1) handling the signal and setting the shutdown flag and 2) actually shutting down the server. That means one cannot run a "regular" signal handler before or after the uvicorn signal handler (1), since one needs to wait for the server to shut down (2) first.

[Kludex]

I'll check this better later on.

[maxfischer2781]

@Kludex Do you still see any required changes? As far as I can tell I should have everything covered.

[DrInfiniteExplorer]

@Kludex It would be super to see this merged! 🙏

[br3ndonland]

@Kludex Do you still see any required changes? As far as I can tell I should have everything covered.

@maxfischer2781 thanks for your work on this. Since you opened the PR, we've added a PR template with a checklist. For your PR, I think it would currently look like this:

• I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.

So it seems like what's missing is some documentation of the server's signal handling behavior. We have a docs page on server behavior that could work for this (it's here in the repo code). If you could add a section there and clearly explain how signal handling will work after this PR, I think it would help everyone understand this better.

After some docs are added to explain the signal handling behavior, I'm happy to give this PR a review as well.

[maxfischer2781]

@br3ndonland Thanks for the reply, I'll prepare proper docs. Can you give me some input on things I would change for making this "official" via documentation:

• I think Deployment > Running programmatically would be a more suitable place. The exact behaviour is only relevant for people that may fiddle with signals, to everyone else it'll "just work" now.
  • Would this location be fine?
• If this gets documented as something programmers can rely on, I would prefer to make it portable and consistent. Right now the Windows and Unix versions behave slightly differently: the Unix version uses the asyncio signal interface but does not use its capabilities. This makes recovering some signal handlers impossible, but technically some user code may rely on the asyncio capabilites.
  • Would consistently using the plain signal package be fine? This is technically a breaking change, though not for documented behaviour.

[br3ndonland]

I think Deployment > Running programmatically would be a more suitable place.

Sure! This PR is about running the server programmatically, so either the server behavior page or the deployment docs on running programmatically could work. Thanks for the suggestion.

If this gets documented as something programmers can rely on, I would prefer to make it portable and consistent. Right now the Windows and Unix versions behave slightly differently: the Unix version uses the asyncio signal interface but does not use its capabilities. This makes recovering some signal handlers impossible, but technically some user code may rely on the asyncio capabilites.

Would consistently using the plain signal package be fine? This is technically a breaking change, though not for documented behaviour.

You make a good point here. I agree that the implementation should be as portable and consistent as possible, but given the long discussion in this PR and in #1579, it might be most pragmatic to keep the existing behavior as-is and just document it. I'm happy to take a look at the code either way.

[formbook]

Has it been decided to leave the broken behaviour?

[maxfischer2781]

I'm still committed to finishing this but seeing how long this dragged on my initial time window for working on it is gone. It may take about 1-4 weeks until I have time to work on this again.

[Kludex]

Thanks for not giving up @maxfischer2781 🙏

I'll make a release shortly.

[vytas7]

In Falcon's test suite, we used to check the return code for functional tests of ASGI servers. Since we terminate the server via SIGTERM, Uvicorn now returns -15 which is a proper way in Unix. However, when running Actions on Windows, we are getting codes like 3221225786 == 2**31 + 2**30 + 314. Is this expected on Windows, and how to interpret these values? @Kludex

[maxfischer2781]

3221225786 is the unsigned interpretation of Windows' 0xC000013A / STATUS_CONTROL_C_EXIT. This is a magic value just like signal 15 on UNIX.

[vytas7]

I see, thanks @maxfischer2781!

[shadchin]

Please tell me, is this the expected behavior after this pull request?

main.py

async def app(scope, receive, send):
    assert scope['type'] == 'http'

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            [b'content-type', b'text/plain'],
        ],
    })
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })

uvicorn 0.28.0

$ uvicorn main:app
...
$ kill -15 <proc_id>
$ echo $?
0

uvicorn 0.29.0

$ uvicorn main:app
...
$ kill -15 <proc_id>
$ echo $?
143

non zero return code

[maxfischer2781]

Yes, this is expected. Notice that 143 is 128+15, bash's way of saying the program was killed by signal 15 - i.e. exactly what you requested with kill -15 <proc_id>.

[Kludex]

@maxfischer2781 Are you able to run tests/supervisors/test_signal.py successfully locally? We may have broken the --timeout-graceful-shutdown flag - I didn't have time to check properly yet.

[shadchin]

Thanks for the answer

I was a little confused, because I was used to SIGTERM being a regular program termination mechanism and expecting a return code of 0, that everything is fine, now we get a non-zero return code, maybe this will break someone's work under supervisors

[maxfischer2781]

@Kludex The tests are working for me locally, but that's MacOS. Windows might be behaving differently again sigh. If you have any leads that there is a problem, can you open a ticket and ping me? I'll have a look then.

[Kludex]

I use MacOS as well. Did you run only that file?

[maxfischer2781]

Now I did. I can confirm that running the tests in isolation makes them fail due to an unhandled KeyboardInterrupt.

What's weird is that the full test suite runs the test_signal.py before test_server.py, i.e. before I muck around with the exception handlers. I need to dig around a bit what to see exactly is failing here.

---

Edit: Just running both test_signal.py and test_server.py isn't enough to hide the failure. There must be another test that interferes here.

---

Edit: Running any of the other supervisor tests is sufficient to hide the failure. I'm guessing this is because they all install custom signal handlers but do not remove them.

As far as I can tell there are three related "issues" here:

1. The supervisors don't play nice to running in the same process, since they all modify the signal handlers without restoring them.
2. The various tests and utilities aren't prepared to contain signals/exceptions, as that wasn't necessary previously.
3. My change re-throwing SIGINT upsets both of these, since 1. means tests may silence signals but 2. means tests are not prepared for signals.

My guess is that this may be hiding a veritable error but I need to double-check that. I'll start by fixing 2 and working up from there, even if it just means we can rely on the tests again.

[bekoev]

Hi Max @maxfischer2781 , I would greatly appreciate your advices on the following.

Context:

• Win 11, WSL 2, Ubuntu 22.04
• main.py

  from fastapi import FastAPI
  import uvicorn
  
  app = FastAPI()
  
  if __name__ == "__main__":
      uvicorn.run(
          "main:app"
      )

• Command set 1: uvicorn main:app then CTRL+C
• Command set 2: python main.py then CTRL+C
• Observation: prior to v. 0.29 in both cases the output corresponds to a graceful shutdown. As of v. 0.29, the command set 2 results in a traceback with asyncio.exceptions.CancelledError and KeyboardInterrupt.

Questions:

1. Is there a (potential) issue in recent versions of uvicorn or the new behaviour is harmless?
2. Is a fix planned or underway?

Thanks in advance!

[maxfischer2781]

@bekoev The behaviour is harmless and intentional, but it's been raised as annoying and misleading multiple times now. So this will likely be made less noisy in the future.