Watch mode repeatedly prints Error returned from file system watcher. when virtual env is inside the directory being analyzed

[adampauls]

Describe the bug
When running pyright -w on the command line, it correctly reports some errors then infinitely loops with Error returned from file system watcher. I'm on MacOS 13.4.1 (22F82). Happens both in iTerm2 and in the terminal within PyCharm. Perhaps it's at least worth exposing the error here? I did so manually locally and see Error: EMFILE: too many open files, watch. It also only happens on larger projects, so this comment might be relevant.

UPDATE: I verified that if I manually change this line to

const excludes: string[] = ["**/node_modules/**", "**/__pycache__/**", "**/.*/**"];

then the problem is resolved. I see that plumbing through configOptions.exclude to the file system watcher constructor is not trivial, but hopefully someone more familiar with the code can do it quickly.

VS Code extension or command-line
pyright 1.1.316 from the command line.

Additional context
Only instance of the error message I could find on a Google search was this bug.

[adampauls]

Is there any chance that the pyright excludes mentioned here are not piped through to the file watcher here? That would make sense, since we have a .venv folder inside the folder that we are running pyright on.

[erictraut]

Piping through the excludes is not a proper solution. Excluded files and directories may still be imported by included files, so they should be retained as part of the files system watcher. The problem here is that MacOS doesn't support recursive file system watchers like Linux and Windows do.

[adampauls]

Upon further investigation, it looks like the problem could be that canUse() is returning undefined here. I set export CHOKIDAR_PRINT_FSEVENTS_REQUIRE_ERROR=1 and see that I'm getting

Error: Cannot find module 'fsevents'
Require stack:
- /Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js
- /Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at 4943 (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js:1:84)
    at i (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js:1:816)
    at 5047 (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/vendor.js:2:464020)
    at i (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js:1:816)
    at 8590 (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/vendor.js:2:451691)
    at i (/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js:1:816) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/dist/pyright.js',
    '/Users/adpauls/.cache/pyright-python/1.1.316/node_modules/pyright/index.js'
  ]
}

I don't have much TS experience so I don't know what's going on here. I do see that fsevents is an optional dependency from Chokidar's point of view. Is it perhaps pyright's responsibility to make sure that library is installed on MacOS?

[adampauls]

Ah, this is an issue in the Python wrapper right? I will report this issue over there.

[erictraut]

I was incorrect above when I said that "MacOS doesn't support recursive file system watchers". It does support recursive file watching via the "fsevents" API. Chokidar uses this API on MacOS to implement efficient recursive file watching events.

I think you may be correct that chokidar is failing to use "fsevents" on your system and is falling back on a technique that doesn't scale well to large file hierarchies.

Why do you think this is related to the Python wrapper? Does the problem go away if you install pyright using npm (npm install pyright)?

[adampauls]

I didn't try installing through npm, but I did say in the other issue that running npm install fsevents in the directory that the Python wrapper installs pyright to fixed the issue. I will double check that installing via npm works as well, but I expect it will.

[erictraut]

The python wrapper simply calls npm install pyright, so I would be surprised if you saw different behaviors between the two. The npm install pyright should install fsevents automatically. I'll need to investigate further to determine why it might fail to install.

[adampauls]

You are right, npm install pyright does not fetch fsevents. I don't have much experience with npm and dependency resolution. What I assume is the published package does not have any dependencies, so internal dependencies like chokidar are presumably compiled in? And perhaps whatever machine is doing that is not on MacOS and so the optionalDependency on fsevents doesn't get installed?

[erictraut]

This is addressed in pyright 1.1.318, which I just published. Thanks @adampauls for your help in identifying a fix!

[adampauls]

Confirmed it is now fixed with me for the released version, thanks!