Go to Symbol in Workspace fails with Heap out of memory and Exclusion patterns are not respected #228

hmoravec · 2020-08-10T23:40:56Z

Environment data

Language Server version: Pylance 2020.8.0
OS and version: Ubuntu 18.04, Dev container with python:3.7.8-slim-buster base image
Python version: Python 3.7.8

Expected behaviour

Go to Symbol in Workspace shows symbols in workspace upon typing.
Files satisfying patterns in Python > Workspace symbols: Exclusion Patterns are ignored in Go to Symbol in Workspace.

Actual behaviour

When typing text to Go to Symbol in Workspace prompt, Pylance starts parsing files and eventually fails with JavaScript heap out of memory error.
Pylance parses site packages in /usr/local/lib/python3.7/site-packages folder when there is **/site-packages/** in Python > Workspace symbols: Exclusion Patterns.

Logs

[Info  - 1:11:54 AM] Pylance language server 2020.8.0 (pyright 4b16a648) starting
[Info  - 1:11:54 AM] Server root directory: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server
[Error - 1:11:54 AM] stubPath /etc/apps/vsa-api/typings is not a valid directory.
(node:914) UnhandledPromiseRejectionWarning: Error: request cancelled
    at Object.t.throwIfCancellationRequested (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:390884)
    at h._addSymbolInformationForScope (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497682)
    at h.visitModule (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497291)
    at h.visitNode (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:114990)
    at h.walk (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:112052)
    at h.findSymbols (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497229)
    at Function.addSymbolsForDocument (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:499589)
    at t.SourceFile.addSymbolsForDocument (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:169618)
    at /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:130603
    at Object.runWithCancellationToken (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:327589)
(node:914) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:914) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:914) UnhandledPromiseRejectionWarning: Error: request cancelled
    at Object.t.throwIfCancellationRequested (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:390884)
    at B (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:188169)
    at J (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:189179)
    at /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:230595
    at /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:230618
    at Array.forEach (<anonymous>)
    at De (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:230400)
    at /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:227219
    at Object.l [as doForSubtypes] (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:343083)
    at Fe (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:225466)
(node:914) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:914) UnhandledPromiseRejectionWarning: Error: request cancelled
    at Object.t.throwIfCancellationRequested (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:390884)
    at h._addSymbolInformationForScope (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497682)
    at h.visitModule (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497291)
    at h.visitNode (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:114990)
    at h.walk (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:112052)
    at h.findSymbols (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:497229)
    at Function.addSymbolsForDocument (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:499589)
    at t.SourceFile.addSymbolsForDocument (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:169618)
    at /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:130603
    at Object.runWithCancellationToken (/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:1:327589)
(node:914) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)

<--- Last few GCs --->

[914:0x42a6380]    41356 ms: Mark-sweep 2032.9 (2049.8) -> 2032.2 (2050.3) MB, 1407.4 / 0.0 ms  (average mu = 0.087, current mu = 0.011) allocation failure scavenge might not succeed
[914:0x42a6380]    43189 ms: Mark-sweep 2033.1 (2050.3) -> 2032.5 (2050.8) MB, 1812.8 / 0.0 ms  (average mu = 0.044, current mu = 0.011) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1a93942]
    1: StubFrame [pc: 0x1a957b4]
    2: StubFrame [pc: 0x1a66310]
Security context: 0x0f2f7651a2f1 <JSObject>
    3: replace [0xf2f765096f1](this=0x26494d7a06f1 <String[101]: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/typeshed-fallback/stdlib>,0x0cac02449021 <JSRegExp <String[6]: [\\\/]>>,0x0f2f76573331 <String[#1]: />)
    4: g [0x2534f2dac7f9] [/home/vsa/.vscode-server/e...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200811.011239.914.0.001.json
Node.js report completed
 1: 0x98edf0 node::Abort() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 2: 0x98fe56 node::OnFatalError(char const*, char const*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 3: 0xb1552e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 4: 0xb158a9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 5: 0xf20105  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 6: 0xf2aa6b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 7: 0xf2b787 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 8: 0xf2e225 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 9: 0xef9250 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
10: 0x11cb0ce v8::internal::Runtime_AllocateInNewSpace(int, unsigned long*, v8::internal::Isolate*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
11: 0x1a93942  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
[Info  - 1:12:39 AM] Connection to server got closed. Server will restart.
[Error - 1:12:39 AM] Request workspace/symbol failed.
Error: Connection got disposed.
	at Object.dispose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:46:82357)
	at Object.dispose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:285821)
	at v.handleConnectionClosed (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:286033)
	at v.handleConnectionClosed (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:46:33937)
	at t (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:284142)
	at invoke (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:437980)
	at s.fire (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:438741)
	at Q (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:46:71922)
	at invoke (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:437980)
	at s.fire (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:438741)
	at l.fireClose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:46:62375)
	at ChildProcess.<anonymous> (/home/vsa/.vscode-server/extensions/ms-python.python-2020.7.96456/out/client/extension.js:1:224392)
	at ChildProcess.emit (events.js:205:15)
	at maybeClose (internal/child_process.js:1021:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
[Info  - 1:12:39 AM] Pylance language server 2020.8.0 (pyright 4b16a648) starting
[Info  - 1:12:39 AM] Server root directory: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server
[Error - 1:12:39 AM] stubPath /etc/apps/vsa-api/typings is not a valid directory.

The text was updated successfully, but these errors were encountered:

jakebailey · 2020-08-10T23:50:19Z

Workspace symbols are currently being improved (#34), performance included. Note that if you mean files.exclude, this is a VS Code option to hide things in the UI (see #123) and we don't read it.

What does this "Python > Workspace symbols: Exclusion Patterns" setting look like in the settings JSON file and what is its description? I'm pretty sure that this setting is only intended to control the Python extension's ctags integration, and isn't related to the language server.

jakebailey · 2020-08-10T23:52:43Z

https://github.com/microsoft/vscode-python/blob/55b4573830f28f5c121a186230cd1265c7fca010/package.json#L3193-L3203

Says: "Pattern used to exclude files and folders from ctags See http://ctags.sourceforge.net/ctags.html."

Pylance is a workspace symbol provider, but doesn't use ctags, so this option would have no effect.

Looking more closely at your comment about it looking into site-packages, this is definitely a case of #34, as our current workspace symbol searching is overzealous and goes into site-packages and bundled stubs.

hmoravec · 2020-08-11T09:28:37Z

And is there any option to exclude files from Go to Symbol in Workspace? I tried to put pyrightconfig.json into the root with

{
    "exclude": [
        "/usr/local/lib/python3.7/site-packages"
    ]
}

as suggested here but Pylance still parsed site packages. But it is probably expected as the Pyright docs mentions

Note that files in the exclude paths may still be included in the analysis if they are referenced (imported) by source files that are not excluded.

But the main problem is still the OOM error. I don't consider parsing site packages overzealous. E.g. PyCharm parses them too and it is useful to be able to search symbols also in site packages. I'd like to exclude them only temporarily so it is working at least for my project files until the memory issues are resolved.

jakebailey · 2020-08-11T17:12:52Z

Parsing isn't overzealous, no, it's running various parts of the analysis that could be, and we need to a lot in order to correctly determine the types of some symbols. Right now, we effectively query the same analysis as everything else to get symbols and their types.

If possible, it would be helpful to enable trace logging to get an idea if there's a specific file which is causing the overflow, as there could be a bug.

Note that this is still workspace symbol search, which has a specific meaning within the LSP and is used by various clients to do things like test discovery and such, so we don't want to overload it to mean something it shouldn't.

hmoravec · 2020-08-11T17:56:48Z

It failed with OOM during parsing plotly 3.7.1 package but not on the same file. When I removed import of plotly, it finished parsing after cca 30s. When I added again import of plotly, it failed with OOM again.

But every time I use the workspace symbol search, there is cca 3s delay. E.g. I open symbol search and I type a few characters, then there is 3s delay, it shows symbols, then I continue with typing, again 3s delay, then it updates the symbols.

Also it would be useful to be able to select which type of symbols should be searched, e.g. only classes, methods and functions. Similarly as it can be set e.g. in Outline settings.

jakebailey · 2020-08-11T18:05:50Z

Did you run with trace logging such that we could see when it's running out of memory? Workspace symbol hanging I expect, but regular analysis I do not.

"python.analysis.logLevel": "Trace"

hmoravec · 2020-08-11T18:34:45Z

This time it failed on werkzeug:

[FG] parsing: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/typeshed-fallback/third_party/2and3/werkzeug/__init__.pyi (1257ms)

<--- Last few GCs --->

[674:0x2ef8380]  2722674 ms: Mark-sweep 2030.7 (2049.4) -> 2030.0 (2049.9) MB, 1306.7 / 0.0 ms  (average mu = 0.109, current mu = 0.015) allocation failure scavenge might not succeed
[674:0x2ef8380]  2723937 ms: Mark-sweep 2030.9 (2049.9) -> 2030.2 (2049.4) MB, 1207.6 / 0.0 ms  (average mu = 0.078, current mu = 0.043) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1a93942]
Security context: 0x0aaa1c69a2f1 <JSObject>
    1: visitAssignment [0xc2da3c974e1] [/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:~1] [pc=0x6e34cc19d9e](this=0x3a743a68c919 <JSObject>,0x15c47a67ca59 <Object map = 0x37748c62e1c9>)
    2: visitNode [0x1c996935cb99] [/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.0/server/pyright.bundle.js:...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x98edf0 node::Abort() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 2: 0x98fe56 node::OnFatalError(char const*, char const*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 3: 0xb1552e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 4: 0xb158a9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 5: 0xf20105  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 6: 0xf2aa6b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 7: 0xf2b787 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 8: 0xf2e225 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 9: 0xef7f9b  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
10: 0xefd514 v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
11: 0xe82bba v8::internal::TranslationBuffer::CreateByteArray(v8::internal::Factory*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
12: 0xc255d7 v8::internal::compiler::CodeGenerator::GenerateDeoptimizationData() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
13: 0xc25cd2 v8::internal::compiler::CodeGenerator::FinalizeCode() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
14: 0xd9ffef v8::internal::compiler::PipelineImpl::FinalizeCode(bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
15: 0xda0ff6 v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl(v8::internal::Isolate*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
16: 0xc16d50 v8::internal::Compiler::FinalizeOptimizedCompilationJob(v8::internal::OptimizedCompilationJob*, v8::internal::Isolate*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
17: 0xc1090b v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
18: 0xec78be v8::internal::StackGuard::HandleInterrupts() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
19: 0x11cb005 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
20: 0x1a93942  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]

erictraut · 2020-08-11T18:54:35Z

We have mechanism in place that watches the heap size and empties our internal type cache if we are at risk of overflowing. However, this mechanism was not being invoked during a workspace symbol search, which explains the problem you're seeing. I've introduced a fix, and it will be in the next release. Thanks again for the bug report.

jakebailey · 2020-08-11T19:40:20Z

I'm going to mark this as fixed in the next release; the overflow shouldn't happen anymore and further improvements to the workspace symbols are being tracked in #34 (including making it faster and not over-scan).

jakebailey · 2020-08-13T20:45:30Z

This issue has been fixed in version 2020.8.1, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#202081-13-august-2020

hmoravec · 2020-08-13T20:57:52Z

Unfortunately, I got the same error also with 2020.8.1:

[FG] parsing: /usr/local/lib/python3.7/site-packages/jupyter_client/localinterfaces.py (1640ms)

<--- Last few GCs --->

[1745:0x43b6380]   277105 ms: Mark-sweep 2030.8 (2050.3) -> 2030.1 (2050.8) MB, 1804.8 / 0.0 ms  (average mu = 0.102, current mu = 0.019) allocation failure scavenge might not succeed
[1745:0x43b6380]   278745 ms: Mark-sweep 2031.1 (2050.8) -> 2030.4 (2050.8) MB, 1615.2 / 0.0 ms  (average mu = 0.058, current mu = 0.015) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1a93942]
    1: StubFrame [pc: 0x1ae9cd8]
Security context: 0x3ea15857e571 <JSObject>
    2: addDeclaration [0x2722e29a2709] [/home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.1/server/pyright.bundle.js:1] [bytecode=0x3be55d544961 offset=0](this=0x2759d3434f79 <r map = 0x314fb61b3269>,0x2759d3435021 <Object map = 0x314fb61b4c09>)
    3: _addInferredTypeAssignmentForVariable [0x39d90f2a74c1] [/home...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200813.225535.1745.0.001.json
Node.js report completed
 1: 0x98edf0 node::Abort() [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 2: 0x98fe56 node::OnFatalError(char const*, char const*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 3: 0xb1552e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 4: 0xb158a9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 5: 0xf20105  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 6: 0xf2aa6b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 7: 0xf2b787 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 8: 0xf2e225 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
 9: 0xef9250 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
10: 0x11cb0ce v8::internal::Runtime_AllocateInNewSpace(int, unsigned long*, v8::internal::Isolate*) [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
11: 0x1a93942  [/home/vsa/.vscode-server/bin/91899dcef7b8110878ea59626991a18c8a6a1b3e/node]
[Info  - 10:55:35 PM] Connection to server got closed. Server will restart.
[Error - 10:55:35 PM] Request workspace/symbol failed.
Error: Connection got disposed.
	at Object.dispose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:46:83265)
	at Object.dispose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:300448)
	at v.handleConnectionClosed (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:300660)
	at v.handleConnectionClosed (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:46:34845)
	at t (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:298769)
	at invoke (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:463922)
	at o.fire (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:464683)
	at Q (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:46:72830)
	at invoke (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:463922)
	at o.fire (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:464683)
	at l.fireClose (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:46:63283)
	at ChildProcess.<anonymous> (/home/vsa/.vscode-server/extensions/ms-python.python-2020.8.101144/out/client/extension.js:1:238481)
	at ChildProcess.emit (events.js:205:15)
	at maybeClose (internal/child_process.js:1021:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
[Info  - 10:55:36 PM] Pylance language server 2020.8.1 (pyright 0916fb9e) starting

jakebailey · 2020-08-13T21:07:17Z

Hm, that's shame. It's curious that the parse for that localinterfaces.py file is taking almost 2 seconds, however. How big is that file? Wonder this is related to our parse cache not being cleared.

erictraut · 2020-08-13T21:21:10Z

Apologies for not addressing your issue with this week's release. I think I understand why my initial fix didn't address it, and I've taken another run at the problem. I've added the cache high watermark detection in a few more places to cover the case that you are hitting. I'm not able to repro your problem, so I can't verify it directly, but I think this will prevent it.

hmoravec · 2020-08-13T21:34:00Z

localinterfeces.py has only cca 200 lines. The parsing of all files was fast for cca 20 seconds, only at the end there was a few files that took longer:

...
[FG] parsing: /usr/local/lib/python3.7/site-packages/jupyter_client/blocking/__init__.py (1ms)
[FG] binding: /usr/local/lib/python3.7/site-packages/jupyter_client/blocking/__init__.py (0ms)
[FG] parsing: /usr/local/lib/python3.7/site-packages/jupyter_client/asynchronous/__init__.py (0ms)
[FG] binding: /usr/local/lib/python3.7/site-packages/jupyter_client/asynchronous/__init__.py (1ms)
[FG] parsing: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.1/server/typeshed-fallback/stdlib/3/getpass.pyi (1ms)
[FG] binding: /home/vsa/.vscode-server/extensions/ms-python.vscode-pylance-2020.8.1/server/typeshed-fallback/stdlib/3/getpass.pyi (0ms)
[FG] parsing: /usr/local/lib/python3.7/site-packages/zmq/__init__.py (23ms)
[FG] binding: /usr/local/lib/python3.7/site-packages/zmq/__init__.py ...
[FG]   parsing: /usr/local/lib/python3.7/site-packages/zmq/backend/__init__.py (6ms)
[FG]   binding: /usr/local/lib/python3.7/site-packages/zmq/backend/__init__.py (5ms)
[FG]   parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/__init__.py (37ms)
[FG]   binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/__init__.py ...
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/error.py (21ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/error.py (2ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/context.py (37ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/context.py (8ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/tracker.py (4ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/tracker.py (1ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/socket.py (915ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/socket.py (13ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/constants.py (10ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/constants.py (1097ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/frame.py (11ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/frame.py (1ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/poll.py (1829ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/poll.py (1ms)
[FG]     parsing: /usr/local/lib/python3.7/site-packages/zmq/sugar/version.py (2ms)
[FG]     binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/version.py (0ms)
[FG]   binding: /usr/local/lib/python3.7/site-packages/zmq/sugar/__init__.py (3959ms)
[FG] binding: /usr/local/lib/python3.7/site-packages/zmq/__init__.py (4012ms)
[FG] parsing: /usr/local/lib/python3.7/site-packages/jupyter_client/localinterfaces.py (1640ms

E.g. zmq/sugar/poll.py which took 1829 ms to parse has also only cca 100 lines.

hmoravec · 2020-08-13T22:22:08Z

To replicate the issue it should be sufficient to use many packages in the environment. Here is example that also fails with OOM.

Dockerfile

FROM python:3.7.8-slim-buster
COPY requirements.txt .
RUN pip install -r requirements.txt

requirements.txt

connexion[swagger-ui]==2.2.0
matplotlib==3.0.3
numpy==1.16.2
pandas==0.24.2
Flask==1.0.2
Pillow==7.1.0
mongoengine==0.20.0
PyYAML==5.1
tqdm==4.31.1
Flask-Cors==3.0.7
statsmodels==0.9.0
scipy==1.2.1
xlrd==1.2.0
plotly==3.7.1
pebble==4.3.10
json-merge-patch==0.2
requests==2.22.0
PyJWT==1.7.1
Werkzeug==0.16.1
prometheus_client==0.8.0
ipython[all]==7.8.0
jupyter-core==4.6.3
pytest==5.3.5
pytest-mypy==0.4.2
coverage==5.0.3
coverage2clover==2.0.1
pylint==2.4.4
pylint2junit==1.0.1
pydocstyle==3.0.0
requests_mock==1.6.0
black==19.10b0

replicate_oom.py

import connexion
import matplotlib
import numpy
import pandas
import flask
import PIL
import mongoengine
import yaml
import tqdm
import statsmodels
import scipy
import xlrd
import plotly
import pebble
import json_merge_patch
import requests
import jwt
import prometheus_client
import pytest


def hello():
    print("Go to symbol 'hello' in workspace. It fails with OOM after cca 30 seconds.")

erictraut · 2020-08-14T22:42:03Z

Thanks for the example. I'm not able to repro an out-of-heap-space error with the latest code.

jakebailey · 2020-08-20T23:19:43Z

This issue has been fixed in version 2020.8.2, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#202082-20-august-2020

github-actions bot added the triage label Aug 10, 2020

jakebailey added the waiting for user response Requires more information from user label Aug 10, 2020

github-actions bot removed the triage label Aug 10, 2020

jakebailey added bug Something isn't working fixed in next version (main) A fix has been implemented and will appear in an upcoming version and removed waiting for user response Requires more information from user labels Aug 11, 2020

jakebailey closed this as completed Aug 13, 2020

jakebailey reopened this Aug 13, 2020

jakebailey removed the fixed in next version (main) A fix has been implemented and will appear in an upcoming version label Aug 13, 2020

erictraut added the fixed in next version (main) A fix has been implemented and will appear in an upcoming version label Aug 15, 2020

jakebailey mentioned this issue Aug 17, 2020

"heap limit Allocation failed" when background indexing entire workspace #254

Closed

jakebailey closed this as completed Aug 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Go to Symbol in Workspace fails with Heap out of memory and Exclusion patterns are not respected #228

Go to Symbol in Workspace fails with Heap out of memory and Exclusion patterns are not respected #228

hmoravec commented Aug 10, 2020

jakebailey commented Aug 10, 2020 •

edited

Loading

jakebailey commented Aug 10, 2020

hmoravec commented Aug 11, 2020 •

edited

Loading

jakebailey commented Aug 11, 2020

hmoravec commented Aug 11, 2020 •

edited

Loading

jakebailey commented Aug 11, 2020

hmoravec commented Aug 11, 2020

erictraut commented Aug 11, 2020

jakebailey commented Aug 11, 2020

jakebailey commented Aug 13, 2020

hmoravec commented Aug 13, 2020

jakebailey commented Aug 13, 2020

erictraut commented Aug 13, 2020

hmoravec commented Aug 13, 2020

hmoravec commented Aug 13, 2020

erictraut commented Aug 14, 2020

jakebailey commented Aug 20, 2020

Go to Symbol in Workspace fails with Heap out of memory and Exclusion patterns are not respected #228

Go to Symbol in Workspace fails with Heap out of memory and Exclusion patterns are not respected #228

Comments

hmoravec commented Aug 10, 2020

Environment data

Expected behaviour

Actual behaviour

Logs

jakebailey commented Aug 10, 2020 • edited Loading

jakebailey commented Aug 10, 2020

hmoravec commented Aug 11, 2020 • edited Loading

jakebailey commented Aug 11, 2020

hmoravec commented Aug 11, 2020 • edited Loading

jakebailey commented Aug 11, 2020

hmoravec commented Aug 11, 2020

erictraut commented Aug 11, 2020

jakebailey commented Aug 11, 2020

jakebailey commented Aug 13, 2020

hmoravec commented Aug 13, 2020

jakebailey commented Aug 13, 2020

erictraut commented Aug 13, 2020

hmoravec commented Aug 13, 2020

hmoravec commented Aug 13, 2020

erictraut commented Aug 14, 2020

jakebailey commented Aug 20, 2020

jakebailey commented Aug 10, 2020 •

edited

Loading

hmoravec commented Aug 11, 2020 •

edited

Loading

hmoravec commented Aug 11, 2020 •

edited

Loading