Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I am training a deep learning model in Jupyter using Keras. When I close out of the window and open it again, the window no longer updates my training progress in real time. #6103

Closed
Tylersuard opened this issue Jul 7, 2021 · 2 comments

Comments

@Tylersuard
Copy link

Describe the bug
I enter my code into a jupyter notebook and run it. The training runs fine, and the output of the cell updates in real time to show the training's progress. However, I have to keep that window open, sometimes for weeks! If I close that window and open it again, the notebook will still be running and the model will still be training, but the output of that cell will no longer be updating. This causes problems especially on cloud platforms while running long experiments. I have to send the output of the training cell to a .csv file and then download that file every time I want to check on the progress of my model.

Expected behavior
A clear and concise description of what you expected to happen.
When I exit out of the window, I expect to be able to get back in, and have the training of my model update in real time.

Additional context
I am using Jupyter notebooks in a cloud machine learning environment. I have contacted the cloud service provider and they say this is an issue with Jupyter notebooks.

@kevin-bates
Copy link
Member

kevin-bates commented Jul 7, 2021

Hi @Tylersuard, this is a long-standing issue within Notebook and (now) Jupyter Server/Lab and referenced in several issues. However, the real-time collaboration efforts should introduce the ability to retain a kernel's state/output on the server, thereby providing the ability to capture previously lost output. (See jupyterlab/jupyterlab#5382 (comment))

This issue will not be resolved within this repository but we're hopeful we'll be able to address this in Jupyter Server/Lab with RTC in the near future.

Closing as a duplicate of #641 (which has been recently updated with a similar comment).

@GF-Huang
Copy link

GF-Huang commented Aug 5, 2021

Any news?

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants