Unable to find modules in .venv (reportMissingImports)

[davidolrik]

Ok, this is a strange one...

In recent versions of pylance, import detection for modules in .venv for some of my projects stopped working.
I was running the latest release version, and then tried upgrading to both insiders weekly and daily.

And when import detection doesn't work, autocompletion etc. also stops working.

Imports of builtin modules works, and import of my own stuff also works - it is only modules in .venv that can't be found.

Environment data

• Language Server version: Pylance language server 2021.5.5-pre.1 (pyright fd334302)
• OS and version: macOS 11.4 (20F71)
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.9.4
• Location of projects: APFS Volume • APFS (Case-sensitive, Encrypted)

Expected behaviour

When a modules has been installed in .venv, and the .venv has been selected in vscode pylance offers autocompletion etc.

Actual behaviour

All modules imported from .venv can't be found

Logs

No errors was seen in the logs, even with Trace on.

Additional information

In the environment data above I have highlighted the fact that my projects live on a case sensitive volume, and this seems to be the source of the problem.

I tried creating a small self contained project to show the bug, but couldn't reproduce it - at least not until I moved the project into the same folder as one of projects that have the problem.

So I tried various combinations of directory structures until I realized that case mattered.

These are the folders I tried:

• xxx/pylance-bug works
• xxx/xxx/pylance-bug works
• xxx/xxx/xxx/pylance-bug works
• xxx/xxx/Xxx/pylance-bug fails

And the final test I did was having this folder xxx/xxx/Xxx/pylance-bug open in vscode, seeing the error and then creating a symlink from xxx/xxx/Xxx to xxx/xxx/xxx and reloading the window after which it started working and imports could once again be found.

So I'm guessing that something is lower casing a path somewhere, which is a problem on case sensitive file systems.

[jakebailey]

This is an unfortunate side effect of another bugfix we made a couple of weeks ago; all paths we use are typically normalized, but we had forgotten to do the same to paths returned by the interpreter. On Windows and macOS, we normally detect that the OS is case-insensitive via a check in a tempdir (make a file with capitals, then see if you can access it via lowercase, etc), then normalize paths such that we can properly compare them. This clearly fails when an OS is mixing case-insensitive filesystems with case-sensitive ones, and our code isn't really equipped to handle that case.

@heejaechang

[heejaechang]

let me take a look and get back to you.

[davidolrik]

Yeah, the temp file thing won't work on my system as all my projects are on a separate volume which is case-sensitive in order to make things more Unix like.

Why the need to normalize them? - Couldn't we just assume the paths from the os are the correct ones and use those?

I believe that both macOS and Windows are case-insensitive but case-preserving(not totally sure about Windows though), but if that is the case then we should be able to treat all file systems as case-sensitive.

[jakebailey]

We can get paths from different sources (VS Code, Python, the Node FS API, file watchers), all of which may give different names to the same file, and since we have to work with paths as strings (and then use them in various maps), we lose the ability to compare them cleanly.

For example, VS Code chooses to canonicalize Windows drive letters as lowercase, so those need to be fixed up.

But, it could be the case that we can audit our conversions to ensure things are case preserving and handle the VS Code case, provided everything else is consistent.

[jakebailey]

The next release should do a better job using the real path for these interpreter paths and hopefully mitigate this issue.

[jakebailey]

This issue has been fixed in version 2021.6.1, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202161-9-june-2021