Very large stub files are not processed by Pylance.

[Ryan-DowlingSoka]

Environment data

• Language Server version: 2021.6.3
• OS and version: win32 x64
• Python version (and distribution if applicable, e.g. Anaconda): 3.7.7
• python.analysis.indexing: undefined
• python.analysis.typeCheckingMode: off

Expected behaviour

Intellisense works even for very large stub files.

Actual behaviour

While closed, the very large stub file is not processed by Pylance, resulting in no intellisense.

Logs

Python Language Server Log

[Info  - 10:58:31 AM] Pylance language server 2021.6.3 (pyright e66600b2) starting
[Info  - 10:58:31 AM] Server root directory: c:\Users\ryandow\.vscode\extensions\ms-python.vscode-pylance-2021.6.3\dist
[Error - 10:58:37 AM] File length of "[REDACTED].py" is 20948900 which exceeds the maximum supported file size of 16777216
[Error - 10:58:37 AM] File length of "[REDACTED].py" is 20948900 which exceeds the maximum supported file size of 16777216

---

We work with a third party piece of software which creates a single stub file for all the python bindings it exposes. I've filed a bug with them that the stub file is too large, but at the same time, I am coming at it from this angle that this file limit might be an arbitrary one?

When the stub file is open in VsCode intellisense works, implying to me this is an arbitrary limit. Perhaps this could be changed in configuration?

[erictraut]

You have a stub file that is 20MB in size? A stub file is meant to describe the interface surface for a library. Is the interface to this library really that large?

We recently added this limit for practical reasons. It is somewhat arbitrary, but if we attempt to process bigger files, at some point we will hit memory or stack limits and you will experience crashes.

[Ryan-DowlingSoka]

The stub file in question is auto-generated by the software as almost a wrapper for the entire software package. The interface surface is indeed gigantic. It isn't under my domain, so unfortunately I can't re-engineer it myself to be split up, but I have put in a support ticket to have them investigate.

It also isn't written ideally (it isn't a .pyi for example, and it doesn't use typehinting explicitly like it should, which I've also requested in a ticket to them) and currently is mainly used for auto-complete, and doc strings.

But yes, the file is 416056 lines long. My hope is the vendor can refactor the code generation to use multiple files and better type-hinting, but in the meantime I don't have a ton of say on that. :P Which is why i'm coming at it from both sides here.

[erictraut]

I think we can safely increase the maximum source file size from 16MB to 32MB. I just made that change, and it will be in the next release of pylance. I don't think we should increase it any further though, so please continue to work with the vendor to produce stub files that are not so unruly.

[Ryan-DowlingSoka]

Thanks, I appreciate it. I will continue to put pressure there.

Cheers

[jakebailey]

Leaving open until the fix is released.

[jakebailey]

This issue has been fixed in version 2021.7.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202170-6-july-2021

[matthijsjanssens]

I'm experiencing the same issue when using the unreal.py autocompletion file generated by unreal engine 4 (36MB in our case...). It would be nice if I could increase the limit locally to 64MB using a workspace setting. If I open unreal.py in editor it works fine for the rest of the session.

[jakebailey]

If you open the file in the editor, then it'll be sent to us over the LSP (which will take a little time, but eventually finish), skipping the checks added above, so that's why that it works.

I tried finding a value for what VS Code considers to be "large", and they appear to start treating files differently at 50 MB. That's when they start showing warnings like "this file cannot be formatted because it is too large", skipping syncing between hosts, hiding diffs, and other such limitations.

https://github.com/microsoft/vscode/blob/1e750a7514f365585d8dab1a7a82e0938481ea2f/src/vs/editor/common/model/textModel.ts#L194

@erictraut How do you feel about aligning our maximum to match VS Code's? That'd help the above 36 MB example and be consistent in behavior up until VS Code starts warning the user in the UI.

I can't recall exactly what the huge file size was that prompted our hard limit, though. I think it was much bigger.

[erictraut]

Yeah, aligning those limits makes sense to me.

For those who are generating such large stub files, I would strongly encourage you to explore other options. Stub files that are tens of megabytes in size will not result in a good experience for anyone. Consider emitting a set of stub files whose file and directory structure mirrors the layout of the modules they are describing.

[matthijsjanssens]

Thanks @jakebailey and @erictraut!
I'll look into the unreal engine source code to see if I can make any changes for it to generate .pyi files instead of one massive unreal.py file for autocompletion.

[Ryan-DowlingSoka]

For reference, when this initially happened I contacted epic and got these jiras added:

https://issues.unrealengine.com/issue/UE-119549
https://issues.unrealengine.com/issue/UE-119557

If you have access to the UE5 Early Access you can see the original support ticket here:
https://ue5help.epicgames.com/s/question/0D54z000073SKPqCAO/python-intellisense-pylance-failing-due-to-unrealpy-being-larger-than-16777216