Package installs not detected for interpreters / environments outside the workspace

[blois]

Environment data

• Language Server version: 2021.2.0 (pyright e278f28d)
• OS and version: OSX 10.15.7
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8, Conda 4.8.3
• VSCode version: 1.54.0-insider

Expected behaviour

Start with a clean environment
Open a .py with

import jmespath

Correctly observe an error for the missing import.
Outside of the editor, install with pip install jmespath.

Expect the error to be resolved with no further action needed.

Actual behaviour

The error persists until the language server is restarted.

Logs

I was curious about where this was falling down because I run into it a lot. What I see is:

• On my system the install is going into /otp/miniconda3/envs/analysis/lib/python3.8/site-packages
• I am correctly seeing that directory in the log message Adding fs watcher for library directories:
• When the file change actually occurs, the filename parameter is a relative name.
• The relative path goes to https://github.com/microsoft/pyright/blob/b7373a2e92bc535e3e6a9a8d369b26c9ae7ad555/packages/pyright-internal/src/analyzer/service.ts#L1143
• Where isIgnored is ignoring it because the path is relative? (based on comment at https://github.com/microsoft/pyright/blob/b7373a2e92bc535e3e6a9a8d369b26c9ae7ad555/packages/pyright-internal/src/common/fileSystem.ts#L97)

I don't really know the ins and outs of node file watchers, but at a glance I'm not sure when absolute paths are expected?

Omitted as the the logging I was able to get didn't seem useful based on analysis above.

Code Snippet / Additional information

This could be something with my filesystem I suppose? But I'm unable to get any other behavior anywhere else.

[jakebailey]

We added that hack in to mirror the same hack in VS Code's own file watchers (linked in that above code); there's some goofiness in the way some OSs report their events and the "pattern" is that relative paths are bad events that need to be ignored.

How have you found that the event was a relative path? The event log comes after the isIgnored check, so I wouldn't have expected it to be printed...

[blois]

I switched over to following the instructions from https://github.com/microsoft/pyright/blob/master/docs/build-debug.md#building-the-vs-code-extension and manually added logging (was a fun adventure). I think this is an appropriate approximation? The core issue was the same between the two, though it's entirely possible that the cause is different.

[jakebailey]

Thanks. We've also had some internal reports about the library watching not working, so this is an interesting hint.

[blois]

I'm curious if this is a difference between the Node & the Chokidar file system watchers.

[jakebailey]

It's possible this regressed (or never worked) as we mainly test with virtual environments within the workspace, as opposed to the conditions where packages are installed in global (or pipenv/conda) locations outside the workspace.

I do recall there was a reason why we prefer the built-in node watcher over chokidar for this purpose, but I can't recall why off the top of my head.

[erictraut]

The node and chokidar file system watchers unfortunately each have different quirks, limitations, and bugs (some of them resulting in crashes). And those differ by platform. We've done our best to select the best options and configure them in the best way, but there's no great answer here.

For the Pyright VS Code extension, I exposed a "Restart language server" command, and I use it to work around cases like this where the file system watcher doesn't work. We may want to expose the same command in Pylance.

[jakebailey]

We already do have one, I implemented it a few months ago.

I think I want to reexamine the file watching code; pyright originally did use chokidar but it was changed when we were starting on Pylance. That and the FS code is flipped around. I have a branch I'm working on to pick up Monday.

[jakebailey]

This does come down to a difference in the file watcher interfaces; I did some live checking with @rchiodo and switching it to Chokidar does make this work again. I'll do some testing to verify this doesn't regress anything.

[jakebailey]

This issue has been fixed in version 2021.2.1, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202121-10-february-2021

[jakebailey]

Reopening; we unfortunately had to revert the file watcher change as it caused too many issues on macOS (and potentially others).

For now, a workaround is to use the "Restart language server" command, which should be as efficient as letting the file watcher reload things, but obviously isn't automatic. (Better than a full reload.)

[Fatsie]

As I reported in #1357 I have seen same problems but for me it was on Ubuntu 20.04, Pylance 2021.5.4

[heejaechang]

dup this to #2505