-
|
I have a widget like this import duckdb
import anywidget
import pyarrow as pa
class CustomWidget(anywidget.AnyWidget):
_esm = """
import * as arrow from 'https://cdn.jsdelivr.net/npm/apache-arrow@11.0.0/+esm'
export function render(view) {
view.el.classList.add("custom-widget");
view.model.on("msg:custom", msg => {
console.log("received message", msg);
if (msg.type === "result") {
const table = arrow.tableFromIPC(msg.result);
console.log("table", table);
}
});
const btn = Object.assign(document.createElement("button"), { innerText: "Send message" });
btn.addEventListener("click", () => {
const msg = {type: "arrow", sql: "SELECT count(*) FROM df"};
console.log("sending message", msg);
view.model.send(msg);
});
view.el.appendChild(btn);
}
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.on_msg(self._handle_custom_msg)
def _handle_custom_msg(self, data, buffers):
print(f"{data=}, {buffers=}")
if data["type"] == "arrow":
result = duckdb.query(data["sql"]).arrow()
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, result.schema) as writer:
writer.write(result)
buf = sink.getvalue()
msg = {"type": "result", "result": buf.to_pybytes()}
print(f"{msg=}")
self.send(msg)
widget = CustomWidget()
widget
I would expect the client to get binary data but instead, the messages are strings and so the client doesn't parse the Arrow IPC correctly.
|
Beta Was this translation helpful? Give feedback.
Replies: 8 comments
-
|
I don't know whether this is because of jupyter-widgets/ipywidgets#3689. |
Beta Was this translation helpful? Give feedback.
-
|
The buffer in our message we send is binary |
Beta Was this translation helpful? Give feedback.
-
|
I see, let me have a look. The issue you linked (jupyter-widgets/ipywidgets#3689) is related to sending binary data from JS to Python (so my hunch is that this is a separate issue). |
Beta Was this translation helpful? Give feedback.
-
|
Ok, so in order to send binary data for custom messages, you must use the self.send(msg, buffers=[msg.pop("result")])Then on the JS side, the handler for view.model.on("msg:custom", (msg, buffers) => {
let dataView = buffers[0]; // DataView
let buffer = dataView.buffer; // ArrayBuffer
})(This is all probably some information that should find its way into our docs. My first time learning some of this, and not very well documented in the Jupyter Widgets itself. Sorry for the confusion/frustration.) |
Beta Was this translation helpful? Give feedback.
-
|
Oop, I also just figured out you can send binary data from the client to the with let bytes = new TextEncoder().encode("Hello, world");
let msg = { type: "encoded-text" }
let callbacks = undefined; // not sure what this is or does
let buffers = [new DataView(bytes.buffer)]
view.model.send(msg, undefined, buffers);and then in Python callback @widget.on_msg
def _handle_custom_message(instance, msg, buffers):
if msg["type"] == "encoded-text":
mview = buffers[0] # always a memoryview
print(buffers[0].tobytes()) # b'Hello, world' |
Beta Was this translation helpful? Give feedback.
-
Yes, it's not well documented. I thought I read that binary data gets automatically extracted but that doesn't seem to be the case. I really, really, appreciate that you found the solution! |
Beta Was this translation helpful? Give feedback.
-
|
(i think) binary data gets automatically extracted for synced traitlets but apparently not for custom messages. |
Beta Was this translation helpful? Give feedback.
-
indeed. |
Beta Was this translation helpful? Give feedback.
Ok, so in order to send binary data for custom messages, you must use the
buffersargument of theself.send.Then on the JS side, the handler for
msg:customevent receives amsg(result ofJSON.parse) andbuffers(your binary data serialized as a JSDataViewinstance):(This is all probably some information that should find its way into our docs. My first time learning some of this, and not very well documented in the Jupyter Widgets itself. Sorry for the confusion/frustration.)