Use native coroutines instead of tornado coroutines

[SylvainCorlay]

• This fixes Python contextvars don't persist across cells in notebook ipython#11565 because tornado's wrapping of asyncio is a bit aggressive with creating Task objects which causes them to have a different contextvars.Context.
• However, this is a major change. I think this deserves a careful review - and be considered a backward incompatible change.
• If/when this goes in, we should also consider completely getting rid of tornado in ipykernel.

cc @afshin @minrk @Carreau.

Note: tests are passing except for the in-process kernel tests, for which we will need to make more functions async.

[minrk]

Makes sense. One minor suggestion that we preserve the maybe_future functionality to allow non-async methods, since most are in fact not async. That's probably the biggest breaking change here.

Since asyncio doesn't have its own maybe_future we can be more simplistic and just use:

result = maybe_async_api()
if inspect.isawaitable(result):
    result = await result

The main difference this has is that concurrent.futures.Future is not awaitable. Everything tornado returns is, though (>=5.0, at lest). I think that's okay, though.

[SylvainCorlay]

I could fix the in-process kernel tests.

[SylvainCorlay]

Also, should I remove other uses of tornado in the code base in this PR?

[minrk]

should I remove other uses of tornado in the code base in this PR?

I think it's sensible to do it in stages, since this is a concrete discrete change.

It looks like the main thing left is our use of queues. However, asyncio's queue is not threadsafe, unlike tornado's, and the whole point of our use of the queues is inter-thread communication. So I think it makes sense to deal with that in its own PR, and keep this one to just async def. Edit: tornado's queues are also not threadsafe. They are only called via add_callback. But I can imagine there being subtle threadsafety issues, since we do use them across threads.

[Carreau]

That looks great; with this, capturing of IO and some of the cleanup that has been done recently it would make a compelling release. We may even use the occasion to drop a couple deprecated things.

[SylvainCorlay]

That looks great; with this, capturing of IO and some of the cleanup that has been done recently it would make a compelling release. We may even use the occasion to drop a couple deprecated things.

👍 And debugger support!

[blink1073]

👍🏼

[SylvainCorlay]

It looks like the main thing left is our use of queues. However, asyncio's queue is not threadsafe, unlike tornado's, and the whole point of our use of the queues is inter-thread communication. So I think it makes sense to deal with that in its own PR, and keep this one to just async def. Edit: tornado's queues are also not threadsafe. They are only called via add_callback. But I can imagine there being subtle threadsafety issues, since we do use them across threads.

Tornado queues are not thread safe, but it does not mean you can't use them with threads. It only means you can't use their APIs from a different thread where the ioloop runs - except for add_callback which is documented as being threadsafe.

[minrk]

👍 in general. I'd only add that I think we shouldn't deprecate synchronous handlers, so I think we should skip the deprecation message, and shouldn't convert existing do_ methods and message handlers to async if they aren't already. There isn't really a benefit to that that I can see, and it does have a (tiny) performance cost.

[SylvainCorlay]

I addressed all of @minrk comments.

The windows python 3.7 test failure is the same as in master and is probably due to the release of importlib-metadata a few days a go. This is fixed in #649.

[blink1073]

Kicking CI

[minrk]

Nice!

[SylvainCorlay]

Many thanks to @minrk for the careful review.

[blink1073]

Wow, good work y'all. 🎉

[gsakkis]

I just upgraded ipykernel to 5.5.5 for this fix but it doesn't seem to be working for me:

Am I missing something?

EDIT: it turns out this PR is not backported to 5.x releases; it works fine after pip install ipykernel==6.0.0b0

[SylvainCorlay]

Yep, will be in 6.0.