Rendering stuck on cell that reads data

[Dr4cky]

Hi there,

I am having an issue and I have no idea why it is not working. The rendering gets stuck on the following cell:

path = Path(r"data.csv")
df = pd.read_csv(path, index_col=0, delimiter=';')

It doesn't show an error, it just keeps loading forever "Executing 2 out of 6".
If I just simply run this cell in my notebook, it runs instantly. Any idea what is going wrong?

Thanks!

[ggogate]

Anything in the browser Console?
Also can you try loading a different, smaller sample CSV which is comma separated.

[Dr4cky]

Anything in the browser Console? Also can you try loading a different, smaller sample CSV which is comma separated.

I just found out that it is not actually that cell that causes the problem, but the cell after:

returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

A portion of the error that I see in the console is as follows:

id
raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id)
tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: c5416394-05a1-485e-a9bb-595a9d5e2387)
404 GET /api/kernels/c5416394-05a1-485e-a9bb-595a9d5e2387?1702219619084 (::1) 4.17ms

[ggogate]

Try commenting out the figure part and see if the HTML displays okay. Then try moving the fig_ts part above the 'with'.

[Dr4cky]

I did a reinstall and the current error that Voila gives me is that it cannot find the module plotly, even though it is installed in my environment and I can the code in the notebook just fine. Is it calling another kernel?

[ggogate]

Try starting Jupyter/voila with --enable_nbextensions=True

[Dr4cky]

Gave me the same errors unfortunately, so plotly module not found and kernel does not exist.

[trungleduc]

Hi, could post the full reproduction code? It is easier for us if you follow the issue template.

[ggogate]

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors.
Maybe try installing previous stable versions of libraries.
If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

[Dr4cky]

My code is as follows:

import numpy as np
import pandas as pd
from pathlib import Path
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import plotly.graph_objects as go

#%%
def show_ts(df):
    fig = go.Figure()
    for col in df.columns:
        fig.add_trace(
            go.Scatter(
                name=col,
                mode="markers+lines",
                x=df.index,
                y=df[col],
                hovertemplate=("Return: %{y:.2%}<extra></extra>"),
            )
        )
    fig.update_layout(
        title={"text": "Returns", "y": 0.9, "x": 0.5, "xanchor": "center", "yanchor": "top"},
        font_family="Times New Roman",
        title_font_family="Times New Roman",
        legend_title_font_color="green",
    )
    fig.layout.yaxis.tickformat = ",.0%"
    fig.update_xaxes(title_font_family="Arial")
    fig.update_layout(
        # height=600,
        paper_bgcolor="rgba(0,0,0,0)",
        plot_bgcolor="rgba(0,0,0,0)",
        legend=dict(font_family="Courier New", font_color="white"),
    )
    fig.show()

#%%
data = np.random.random((10, 3))
df = pd.DataFrame(data, columns=['A', 'B', 'C'])


returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

tab = widgets.Tab([returns_plot])
tab.set_title(0, 'Data')
header_one = widgets.HTML(value="<h4>Task One</h4>", layout=widgets.Layout(width='300px'))
display(widgets.VBox([header_one, tab]))

It is stuck on 2/4. The message I see in console is:

[Voila] WARNING | Clearing invalid/expired login cookie username-localhost-8869
[Voila] WARNING | Notebook dashboard2.ipynb is not trusted
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

[Dr4cky]

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors. Maybe try installing previous stable versions of libraries. If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

My requirements is as follows:

ipykernel==6.27.1
ipywidgets==8.1.1
jupyterlab==4.0.9
nbformat==5.9.2
numpy==1.26.2
openpyxl==3.1.2
pandas==2.1.4
plotly==5.18.0
pre-commit==3.5.0
voila==0.5.5

Running Python version 3.12.1

[ggogate]

I was able to reproduce the issue with the same Python and library versions.
I was also able to run voila by making following changes -

1. Removing the imports to interact - #from ipywidgets import interact, interactive, fixed, interact_manual
2. Change the fig.show() to return a fig. Alternately add the with-display in the show_ts()
   #fig.show()
   return fig
3. Add a display("") before the with and move the show_ts above the with
   #%%
   data = np.random.random((10, 3))
   df = pd.DataFrame(data, columns=['A', 'B', 'C'])
   display("Before")
   fig_ts = show_ts(df)
   #display("After")
   returns_plot = widgets.Output()
   with returns_plot:
   text = widgets.HTML('Time series data.')
   display(fig_ts)
   #interact(fig_ts)
   display(text)

Alternately, move the display part inside the show_ts() function.

Attaching both versions of notebooks -
Voila-Test.zip

[Dr4cky]

Thanks for your help! Unfortunately, Voila fails to run for both of your notebooks. It seems like display(fig_ts) also does not work.

[Dr4cky]

I tried Voila version 0.4.0, after which the code does work. Why would that be?

[afonit]

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

[Dr4cky]

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

Thanks! I gave up trying to figure it out and just used an older version of Voila (0.4.0). Hopefully they will fix it, though,

[afonit]

I have been trying to find a minimal reproducible example to aid in the troubleshooting/solution.

What adds to the difficulty - is it can be inconsistent, for example with the all current packages in a clean enviornment:

import ipywidgets as widgets
import pandas as pd
from IPython.display import display, HTML, Markdown, Javascript

data = {'a': [1, 2, 3],
        'b': ['one', 'two', 'three'],
       }

df = pd.DataFrame(data)

dropdown = widgets.Dropdown(
    description='select one',
    options=[1, 2, 3],
    value=1,
)

base_out = widgets.Output()

with base_out:
    base_out.clear_output()
    display(Markdown("""## This is in the output"""))
    display(df)

base_out

That will render fine every time.

but if I break it up into cells with some extra getting displayed eg:

Then it will frequently get stuck on:

and if I do a CTRL+SHIFT+r then the page will refresh and load the page completely.

With nothing in the logs other than:

Jan 03 14:51:08 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:53 test-env-149200 voila[18148]: [Voila] Kernel started: 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Debugger warning: It seems that frozen modules are being used, which may
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - to python to disable frozen modules.
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:56 test-env-149200 voila[18148]: [Voila] Connecting to kernel 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3.

[trungleduc]

Thanks for the reproduction, I will take a look at it.

[afonit]

@trungleduc, please let us/me know if there are any additional steps we could take to aid in nailing this down.

[trungleduc]

I'm able to reproduce the issue with a more compact snippet

import ipywidgets as widgets
import os
returns_plot = widgets.Output()
with returns_plot:
    display(widgets.HTML("Hello"))
    display(widgets.IntSlider())
returns_plot

with these dependencies

voila=0.5.5
ipywidgets=8.1.1
ipykernel=6.29.0
jupyter_client=8.6.0
nbclient=0.7.4                    

Voila will hang randomly after some refreshes. I think the issue is related to #1234 and #1253. Voila hangs because in this await call, nbclient does not receive the execute_reply message sent from the kernel side.

Moreover, from this comment (#1234 (comment)), it looks like this block is actually the culprit since if I remove it, Voila works as expected after multiple refreshes.

  if isinstance(stream, zmq.asyncio.Socket):
      assert stream is not None
      stream = zmq.Socket.shadow(stream.underlying)

Ping @davidbrochart @blink1073 if you have idea how to fix this.

[blink1073]

Yes, the asyncio logic is definitely convoluted, I'm working on a fix in jupyter/jupyter_client#997, but I haven't had much bandwidth recently.

[blink1073]

I don't think the issue is with that code block per se, because the effect is that it is not actually sending, because send_multipart returns an awaitable for an async socket.

[ClaytonAstrom]

Is there a better parent issue to track the discussions on this? Either on jupyter_server or something?

[cosmicfarmers]

For anyone who tries to resolve this issue.
In my case, preheating kernel solved problem, although it is not an ultimate solution; it didn't make the problem when preheating kernels.

[ClaytonAstrom]

I ended up having to back port some voila 0.5 changes to a forked version of 0.4.4, using jupyter-server 1.18.1. That worked for me.

[afonit]

Has anyone found a consistent workaround or solution for this? I know someone said the pre heated kernels worked for them as a workaround. We tried that and did not see a resolution to the issue.

[trungleduc]

@afonit do you have this issue without using the Output widget?

[afonit]

@trungleduc when I had just done a new vm (ubuntu 20.04) today and did everything fresh, it was the same issue as before, as you noted in:
#1428 (comment)

I am trying with these libraries:

voila=0.5.7
ipywidgets=8.1.3
ipykernel=6.29.4
jupyter_client=8.6.2
nbclient=0.10.0

We were using some very old packages that worked really well together but have not had to reimage and upgrade.

Right now it is being intermittent in working.

I did notice I am getting some messages like in: #1458

and

Jun 07 21:22:09 prod-705475 voila[61365]:     {%- block body -%}
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/share/jupyter/nbconvert/templates/base/null.j2", line 29, in block 'body'
Jun 07 21:22:09 prod-705475 voila[61365]:     {%- block body_loop -%}
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/share/jupyter/voila/templates/lab/index.html.j2", line 86, in block 'body_loop'
Jun 07 21:22:09 prod-705475 voila[61365]:     {% set page_config_full = page_config.copy() %}
Jun 07 21:22:09 prod-705475 voila[61365]:     ^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/lib/python3.12/site-packages/jinja2/environment.py", line 487, in getattr
Jun 07 21:22:09 prod-705475 voila[61365]:     return getattr(obj, attribute)
Jun 07 21:22:09 prod-705475 voila[61365]:            ^^^^^^^^^^^^^^^^^^^^^^^
Jun 07 21:22:09 prod-705475 voila[61365]: jinja2.exceptions.UndefinedError: 'page_config' is undefined

To answer your output question, yes if I do some very basic notebooks with no outputs just markdown and a widget - they render fine in Voila.

[trungleduc]

I would avoid using the Output widget, and just render the normal widgets or group them in the HBox or VBox widget for now. Calling display inside an output context might be the root of the issue.

[afonit]

I would avoid using the Output widget, and just render the normal widgets or group them in the HBox or VBox widget for now. Calling display inside an output context might be the root of the issue.

We will try to do that - thank you. We have those heavily embedded id display's, we will try to simplify them and make it work - almost all our notebooks are widget/display heavy.

Thanks for your time and replies.

[jvailius]

I have the same issue as soon as I add a widget from ipywidgets (v8): Rendering progress bar is stuck a x out of y steps.

[trungleduc]

Hi @jvailius, could you provide your code to reproduce the issue? Does it involve using the Output widget?

[jvailius]

Hi @trungleduc - this is somewhat embarrassing because I can't reproduce it. I can not say why it works now with an interactive widget. Although from time to time it is stuck again - but rarely. Either is is a change to my code or some dependency that changed while trying different voila/widgets versions. Sorry for the noise. :/

[trungleduc]

@jvailius Actually this issue is quite hard to reproduce, it also happened to me, so I want to limit the possible culprits. The most possible reason is the Output widget.

[jvailius]

The most possible reason is the Output widget.

Forgot to mention that I was not using Output but it may have been related to fig.show() in plotly.

[konosubakonoakua]

same here

[arthurBarthe]

Hi,

Don't know if this might be helpful in solving the issue, but a fix for me has been to replace display (within out context voila was hanging) by append_display_data, as per the code below which I believe is similar.

anim = video_plot_3d(sample, get_title=lambda i: f'time: {i}', cmap='BrBG')
out = widgets.Output(layout={'border': '1px solid black'})
plt.close()
out.append_display_data(HTML(anim.to_html5_video()))
out

Apologies if this is unrelated.

[trungleduc]

Hi all, we released a new rendering method in Voila 0.5.8 (https://voila.readthedocs.io/en/stable/customize.html#changing-dashboard-rendering-technique). You can activate it by using the flag --progressive_rendering=True

Could you try this mode to see if the issue is still persistent?

[christakahashi]

I have the same problem. i'm using plt.plot() from matplotlib, and theFloatLogSlider, and IntSlider widgets, and interact. without --progressive_rendering=True I get the same stuck at 2 of 3. with that flag it makes it to 3 of 3 and then starts to load something. but gets stuck on a screen that looks like this:

I tried various versions and 0.4.5 works for me but later versions do not (i tried latest and 0.4.6, witch do not work. all minors between 0.4.0 and 0.4.5 work for me)
(EDIT: i was shifting the versions around in conda and it depends on what order i change versions. e.g. if i downgrade to 0.4.4 from 0.5.8 it doesn't work but if I go from 0.4.0 to 0.4.4 it does work, so maybe something changed in a dependency).

Trying to cutdown to a minimal example with 0.5.8 i get this, which seems to still break it

from ipywidgets import interact
# %%
def foo(lam:float ):
    print(lam)
interact(foo,lam=(0.1,10,.1) )

with this conda environment.yaml

name: test
channels:
  - conda-forge
dependencies:
  - python=3.11
  - numpy
  - scipy
  - matplotlib
  - jupyter 
  - voila=0.5.8