New kernels REST API

[davidbrochart]

Problem

In jupyter-server/jupyverse#191 we are experimenting with a new kernels REST API for code execution when RTC is enabled: instead of letting the front-end deal with the low-level kernel protocol, it just sends an execution request that consists of:

• the shared notebook document path,
• the cell index in that notebook,
• the kernel ID to use for the execution.

The execution is handled in the back-end, which also updates the notebook cell outputs. All front-ends see their notebook update through Yjs. I think this execution mode could be very promising because:

• it saves a round-trip between the back-end and the front-end,
• it doesn't propagate the complexity of the kernel protocol to the front-end,
• the front-end can disconnect and get all the missing document updates when it reconnects.

Unfortunately this doesn't work with ipywidgets, since they make use of the kernel protocol.

Proposed Solution

I am not sure how this could be solved unless we drastically change ipywidgets' architecture. But as I know there have been discussions about using Yjs in ipywidgets, maybe it could be possible.
The way I see it is that the widget state should somehow be encoded in the shared document using Y data structures. This probably means using Ypy/Yjs instead of traitlets/BackboneJS.

[echarles]

This probably means using Ypy/Yjs instead of traitlets/BackboneJS.

This is probably what it means. Does it also mean that all existing ipywidgets will not work in that model? Would it be possible to create an adapter between both worlds to offer a migration path?

[williamstein]

I am not sure how this could be solved unless we drastically change ipywidgets' architecture. But as I know there have been discussions about using Yjs in ipywidgets, maybe it could be possible.

For what it is worth, I've mostly solved the analogue of this problem in CoCalc, which fully uses RTC and has the execution model you describe. This has been a long and painful journey involving studying a lot of ipywidgets code, etc., and it's still not completely done ( sagemathinc/cocalc#6011 is the one thing I know of that is left, and that's probably one of the easiest things from this point of view). Anyway, I'm happy to discuss my experience... I gave this little talk in the weekly meeting: https://www.youtube.com/watch?v=t4h5QrBKjyY

[williamstein]

This probably means using Ypy/Yjs instead of traitlets/BackboneJS.

This is probably what it means. Does it also mean that all existing ipywidgets will not work in that model? Would it be possible to create an adapter between both worlds to offer a migration path?

In CoCalc, which uses realtime sync + ipywidgets, all existing ipywidgets do work, except for some custom widgets that I don't yet support (as mentioned above), but that's just a matter of time. The way it works is that I use CoCalc's realtime sync to synchronize the state of the BackboneJS models, so I don't have to modify any of the existing ipywidgets codebase at all.

[davidbrochart]

The way I see it is that the widget state should somehow be encoded in the shared document using Y data structures. This probably means using Ypy/Yjs instead of traitlets/BackboneJS.

FYI efforts in that direction has started in ypywidgets, where a widget's model is a Y document.