Auto-imports only work for code from currently open files when indexing is enabled

[klbostee]

Environment data

• Language Server version: 2021.3.1

Expected behaviour

I would expect pylance to be able to auto-import any symbol from my codebase, not only from currently open files.

Actual behaviour

As mentioned in the changelog for 2021.3.1, auto-imports now only work for code from currently open files when indexing is enabled. The release notes also said "please file an issue if you have enabled indexing and this affects your workflow", which is what I'm doing right now :)

Additional information

This does indeed affect our workflow quite a bit. Could it at least be made configurable maybe?

[jakebailey]

Thanks for this; it's hard to find people who are actually using our insiders features...

Before we disabled it (sorry!), did you experience any sort of performance troubles with indexing? Or was it not a problem and the feature was worth the extra CPU time spent working? Did completions show up in a reasonable amount of time?

I'm very curious about your codebase and how it helped/hurt.

[klbostee]

So we're looking into moving our Python dev team over to VS Code with Pylance (one of the main reasons being that dev containers are really appealing to us, for what it's worth). This team relies heavily on auto-imports, symbol search and reference search, and enabling indexing seems to be the only way to make those (mostly) work for our large codebase.

We do see performance problems with indexing, as already discussed in #997 also. More specifically, completions and auto-imports can definitely be slow at times. The performance issues are enough of a turn-off to keep the team from switching over permanently at this point in fact, but not having the full functionality makes things even worse for us I'm afraid.

I guess I was hoping that VS Code with Pylance could still be an option for us further down the road, as indexing performance improves over time. This change is more of a step backwards that makes that all less feasible however. We'd be happy to test things out and/or provide info/feedback if that can help to keep the dream alive though :)

[heejaechang]

@klbostee how about providing the ability through code action? (the light bulb), we are trying to reduce things that affect typing such as completion but fine some feature taking more time on user explicit command such as code action.

for example, we already suggest "add import" for mis-spelled symbol in code action but not in completion or we include lower case variables in __ all __ in code action but not in completion.

[klbostee]

Providing it (only) through a code action sounds like a good potential solution to me @heejaechang. I'd have to ask the others in the team, but for me it was typically already very similar to this in practice anyway. The red curly line would appear because the symbol isn't defined, and then I'd trigger an auto-complete by hitting ctrl+space and pick the auto-import. So the only difference would be having to trigger code actions instead of auto-complete.

Some of our devs are pretty religious about being able to access everything via the keyboard, but I just checked and there seems to be a keyboard shortcut for code actions too (cmd+.) so that shouldn't be an issue.

[jakebailey]

The next release (later today) will re-eanble it for code actions (quick fixes), but not for completions. We'll see how far that get us, and use this thread for further feedback.

[b99andla]

I have experienced the auto imports getting worse during the last few updates to Pylance. I am unsure as what I need to do now to get this working again.

I have installed the latest version of Pylance, I have added "python.analysis.indexing": true as well but it is still not working to use the

A recent example:
I am working in a project where I would like to send mail. We have the function send_mail in three different places in this project, all of them outside my current workspace.

IntelliJ shows these imports as:

While the latest pylance (2021.3.3) shows no valid imports at all:

What do we need to do to get code action imports working in this case?

[jakebailey]

Did send_mail work in a previous version of Pylance, or is this just an example to compare to PyCharm? We have some internal limits to prevent over-indexing, and I think send_mail may be too nested in Django for us to have indexed it at startup.

[erictraut]

We should fully index all files within the project though. I'm guessing that common.utils.mail is within the workspace, Whereas django.core.mail and wagtail.admin.mail are imported from outside the workspace?

[b99andla]

I am sorry, I was under the impression that this has been working. But I may have just made this up... I am actually unsure now.

Is there any way to make this work with the current release of pylance?

[jakebailey]

Right now, we have some hardcoded limits in the indexer as we work on ways to improve its performance. Most packages are limited to the top level, but django is restricted to two levels, but django.core.mail is three and is outside those limits.

We've recently identified some more perf improvements, so we'll see what we can do, but at the moment there isn't a way for you to tweak this externally.

[jakebailey]

Today's release (2021.5.3) includes stricter filtering on auto-imports which should mitigate some of the performance problems we have with offering far too many symbols. If this goes well, it's likely we can look into re-enabling auto-imports that make use of the user code index, but we'll see (I don't want to jinx it).

[ThiefMaster]

Do you have any details on how the logic changed and are there any settings to tune the behavior?

[jakebailey]

The change is that auto-import completions at least need to start with the same character as what has already been typed. This is something that is currently being put into TypeScript's auto-import system, as they have similar issues to us with offering so many completions that performance is greatly affected. Quick fixes remain unchanged.

There are no settings to tune this behavior. Is there a reason you're looking for a way to change it? Happy to take feedback as to how this may break; the intent is to filter out the bulk of the completions nobody is accepting.

[ThiefMaster]

Is there a reason you're looking for a way to change it?

Just curiosity in case I noticed something strange so I can try things out before posting something here.
But once thing that may be useful though is a setting to make it case sensitive. For example if I use session I know I do not want a Session class, so getting those suggestions is a bit pointless IMO. But I'm sure some people may like getting them...

[ThiefMaster]

I also noticed that for a multi-root workspace I often don't get proper suggestions from the code in another root. This is quite annoying when you have a core project and a plugin repo, and usually import stuff from the core in plugin code.

The repos are linked above so if you want to investigate this further feel free to try with them (I have one root for the core project and one for the folder containing all the plugins). Especially the Event import is interesting because it's a HUGELY common name, so getting matches from my own code first would be really helpful there. It looks like it ONLY searches within that particular root, and not in other ones at all:

In terms of sorting, it's also a mix between thirdparty (site-packages) and custom code. Having imports from own code come first would be nice...

[jakebailey]

Multi-root is its own problem, because the roots do not interact with each other (and in fact get their own distinct analysis). Having a multi-root workspace does not imply that one workspace can import from another as though the other roots are on sys.path.

IIRC you need to set extraPaths to get them to recognize each other.

[ThiefMaster]

I tried setting "python.analysis.extraPaths": ["/home/adrian/dev/indico/py3/src"], but it didn't help. FWIW, the package is installed with pip install -e in the virtualenv, so from what I understood in another issue it should not be needed to explicitly set it as an extraPath?

Anyway, in the language server output I see this, so the src dir is clearly in the path - I do not get completions for imports coming from there though.

Search paths for /home/adrian/dev/indico/py3/plugins/public
  /home/adrian/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.5.4-pre.1/dist/typeshed-fallback/stdlib
  /home/adrian/dev/indico/py3/plugins/public
  /home/adrian/dev/indico/py3/plugins/public/typings
  /home/adrian/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.5.4-pre.1/dist/typeshed-fallback/stubs/...
  /home/adrian/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.5.4-pre.1/dist/bundled/stubs
  /home/adrian/.pyenv/versions/3.9.5/lib/python3.9
  /home/adrian/.pyenv/versions/3.9.5/lib/python3.9/lib-dynload
  /home/adrian/dev/indico/py3/env/lib/python3.9/site-packages
  /home/adrian/dev/indico/py3/plugins/public/livesync
  [...]
  /home/adrian/dev/indico/py3/plugins/public/citadel
  /home/adrian/dev/indico/py3/plugins/public/themes_legacy
  /home/adrian/dev/indico/py3/src

If I use CTRL+T I do find those symbols though, so they are clearly being indexed.

[heejaechang]

we have added an option to include more symbols from third party libs if user wants, see this #2312 (comment)

I also opened this issue to track UX on multi root workspace - #2601

[jli]

Just chiming in that I feel like auto-import and finding references has gotten worse quite recently, maybe in the past 1-2 weeks. When I "Find all references", it seems like Pylance is missing references more and more. This is all for code within my workspace, not external libraries.

VSCode: Version: 1.71.0 (Universal)
Pylance: v2022.9.20

Sorry this isn't a more useful report :/

[adamalpi]

Same here. I updated vscode, and suddenly auto-import does not recognize the local package anymore.
VSCode: Version: 1.71.2
Pylance: v2022.9.21

[debonte]

@jli, @adamalpi, can you see if setting "python.analysis.indexing":true helps? If not, please file a new issue rather than commenting on this old issue.

[solarw]

@debonte
similar issue for me "python.analysis.indexing": true, set

updated ubuntu and all the software
and autocomplete works only if py file is opened.
works well for custom modules installed, not for my own code

[debonte]

@solarw, please file a new issue.