Jupyter 2023.3.1201040234 extremely high memory usage causes disconnect with remote SSH workstation

[mrburrito]

I run VSCode on a Windows machine using the Remote SSH extension to work with code on a Linux server with 32GB of RAM. When I updated the Python extension this morning to 2023.6.0, I started seeing continual disconnects from my remote code every time I tried to save a file. I tried removing the remote server (~/.vscode-server) as that was a suggested fix. This appeared to work until I realized none of my extensions were installed. As soon as I installed extensions, the problem reappeared. Every time I would try to save a Python file, I'd see a popup saying "Saving File: Running Python formatter..." which would spin forever until the remote connection timed out and could not be re-established.

I started investigating on the remote server and, as soon as I hit save, memory usage shot up from 2GB to 31GB.

I downgraded the Python extensions to:

• Python: 2023.4.1
• Pylance: 2023.3.40
• Jupyter: 2023.3.1000892223

With that configuration, memory usage shoots up to 8GB then settles back down to 5GB and everything appears to function as it did before.

I'm going to investigate further to see if I can narrow down exactly which of those three extensions is causing the issue, but it appeared I needed to back all of them down to fix the problem.

[mrburrito]

I updated Jupyter to latest (2023.3.1201040234) and could not replicate the problem, which didn't surprise me since I'm not using notebooks.

The problem does appear to be with the Python extension. As soon as I updated that to 2023.6.0, leaving Pylance at the downgraded version, saving a file shot memory usage over 30GB and killed my SSH sessions - both the VSCode Remote session and my separate SSH terminal where I've been troubleshooting. It appears to be while the extensions are activating.

Also, it seems to take an extremely long time to save a file while this is happening. I would expect file save to take milliseconds and, even with the downgraded extension, it's a good 30-40 seconds before my changes are saved. This seems to be new behavior and I'm not sure if it's the Python extension or a change in how VSCode handles extension loading with the 1.77 update. Once the extension has loaded, performance is as expected.

[mrburrito]

Additional details:

• VSCode 1.77.3
• Local Host: Windows 10 Enterprise for Virtual Desktops 19044.2728
• Remote Host: CentOS 7.9.2009

[karrtikr]

I updated Jupyter to latest (2023.3.1201040234) and could not replicate the problem, which didn't surprise me since I'm not using notebooks.

Since you're not using notebooks, can you try uninstalling the Jupyter extension and see if the problem reoccurs? Use the latest version for other extensions: 2023.6.0

[mrburrito]

I updated Python and Pylance to latest and was able to replicate the problem when Jupyter was enabled. Disabling Jupyter resolved it, so it appears to be some interaction between the latest Python and Jupyter that causes the issue.

Summary:

Python	Pylance	Jupyter	Problem?
2023.6.1	2023.4.20	2023.3.1201040234	Yes
2023.6.1	2023.4.20	disabled	No
2023.4.1	2023.3.40	2023.3.1000892223	No

Is there a reason Jupyter is bundled with the Python extension? I would think it makes sense for Jupyter to require Python, but not the other way around. That doesn't solve this problem for people that are working with Jupyter notebooks though.

[karrtikr]

@mrburrito Thanks for the info, we're dropping dependency on Jupyter with this release: microsoft/vscode-python#18073.

Transferring this issue to Jupyter.

[mrburrito]

Thanks @karrtikr!

[debonte]

@mrburrito, can you provide a Pylance log of the issue as described here?

Is the issue specific to a particular codebase? Or does it repro even for super simple scenarios?

[mrburrito]

@debonte sorry for the delayed response. I'll work on getting the trace today and try different scenarios. I am trying to open a monorepo with ~40 different virtual environments, which could be part of the problem. I'll test with a single-environment project and see what memory usage looks like.

[mrburrito]

I'm still working on the pylance log, but not sure if I'll be able to provide it. I need to scrub it to make sure we're not leaking any company IP and its 100k+ lines.

The issue does not seem to occur for a simple use case where I have one project with a single interpreter configured. It appears to be a problem when using a workspace with multiple projects, all with their own interpreters. I'll try to set up some quick tests for smaller workspaces to see if there's a break point.

[mrburrito]

Error occurs in a Workspace with 20 projects

WIth only 7 projects in the Workspace, there appears to be a spike in memory usage, but it does eventually complete initialization and resume normal operations. It still took ~60 seconds to save a one line change to a Python file.

Pylance logs shows a lot of "Long operation" logs for:

• analyzing: /path/to/file.py
• checking: /path/to/file.py
• getSemanticTokens full at /path/to/file.py
• index execution environment /path/to/project/in/workspace
• index packages /path/to/project/in/workspace
• parsing: /alt/.vscode-server/extensions/ms-python.vscode-pylance-2023.5.10/dist/typeshed-fallback/stdlib/builtins.pyi

That last one seems to run once for each project in the workspace, even though it's indexing the exact same file. Can that just be somehow baked in to the extension or deconflicted so it only needs to be parsed once?

[debonte]

@mrburrito, are you still seeing this issue? If so, and censoring the Pylance log is too much hassle, would you be willing to provide a log to me directly? I can provide my email address.

[github-actions]

This issue has been closed automatically because it needs more information and has not had recent activity. If the issue still persists, please reopen with the information requested. Thanks.