Output.clear_output() in a thread does not block until the output is cleared

[eriknw]

Description

I create an Output object on the main thread and display something on it. Then, in another thread, I try to clear the output and display other things on it. There is a surprising race condition that causes some of the things in the new thread to not be displayed. Specifically, the call to out.clear_output() does not clear the output right away, so subsequent calls to the Output object will get dropped once clear_output finally takes effect.

Reproduce

From IPython notebook:

import string
import threading
from ipywidgets import Output
from IPython.display import display
import time

out = Output()
out.append_stdout('hello\n')
display(out)

def bad_clear(out):
    out.clear_output()
    for c in string.ascii_letters:
        out.append_stdout(c)
        time.sleep(.01)

thread = threading.Thread(target=bad_clear, args=[out])
thread.start()

For me, the output from the first cell can be jklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ. abcdefghi got cleared!

To work around this, I currently do the following:

def good_clear(out):
    out.clear_output()
    while out.outputs:
        time.sleep(0.01)  # NEW
    for c in string.ascii_letters:
        out.append_stdout(c)
        time.sleep(.01)

thread = threading.Thread(target=good_clear, args=[out])
thread.start()

Expected behavior

I expect the output of out to be cleared in bad_clear immediately after out.clear_output() is called. If it's too troublesome to do this, then I at least expect a note in the docstring and the example added here: #1794

Context

Previous relevant issues: #1794 #1722 #1752

• ipywidgets version: 7.6.3

[eriknw]

Relatedly, the workaround I posted does not work reliably in JupyterLab (version 3.0.16). Here is a reproducer:

import string
import threading
from ipywidgets import Output
from IPython.display import display
import time

out1 = Output()
out1.append_stdout('hello\n')
display(out1)

out2 = Output()
out2.append_stdout('hello\n')
display(out2)

def run(out):
    out.clear_output()
    while out.outputs:  # <-- one of the threads may get stuck in this loop
        time.sleep(.01)
    for c in string.ascii_letters:
        out.append_stdout(c)
        time.sleep(.1)

thread1 = threading.Thread(target=run, args=[out1])
thread2 = threading.Thread(target=run, args=[out2])
thread1.start()
thread2.start()

For me, the typical outcome when running this is that only one of the outputs gets cleared and is then able to be updated. The other output stays as "hello", and never leaves the while out.outputs: loop.

[raziqraif]

From this page, it seems like it is unsafe to use context managers with output widgets when multiple threads are involved.
https://ipywidgets.readthedocs.io/en/latest/examples/Output%20Widget.html#Interacting-with-output-widgets-from-background-threads

While your code does not explicitly use context managers, apparently the clear_output() method does that under the hood.

A workaround that you can use is to clear your output with out.outputs = ()

E.g.

def run(out):
    out.outputs = ()
    for c in string.ascii_letters:
        out.append_stdout(c)

[pnsvk]

Thank you @raziqraif ..
This workaround did the job for me..

[maartenbreddels]

Note that due to ipython/ipykernel#1135 we should be able to change how this works in the future.

[pwuertz]

I'm also seeing race-conditions with Output.clear_output(), even without creating multiple threads!
This simple sequence of commands in Jupyter Lab already fails for me:

out = widgets.Output()
display(out)
out.append_stdout("Hello")
out.clear_output()
out.append_stdout("World")

What I'm seeing is a short flash of HelloWorld that vanishes immediately, instead of the output settling on World as one might expect.

Using out.outputs = () instead of out.clear_output() works however.

So this issue seems to affect clear_output() in general, not just multi-threading use cases.

jupyterlab 4.2.4, ipywidgets 8.1.3