API: Allow to use the file watcher for aribitrary folders

[innerlee]

vscode 0.10.8, windows 10.

like in the video, branch status updated only when clicking git panel.
if auto tracking is not possible, how about update status when clicking the sync button in the status bar?

[MystK]

Hello. This is not a bug. It is not updating for you because you did not enter your Git credentials. After entering your git credentials, the branch automatically changes.

[innerlee]

Hi @MystK, all operations are local. The credential error in the video is not my point, just a side effect of showing that, after pressing the sync button, the local git status is still not updated. Before that, I pressed the "master" button, this interaction also did not update the status. Until at last, I opened the git panel, new branch finally shows.
So, I expect that, clicking the "master" button or the sync button will refresh status first and show the new created/deleted branch.

[MystK]

Thanks for the clarification. My apologies. I've tried to reproduce your issue but cannot. I've clones your environment and steps but my branch still switches to the testing branch without having to click the Git panel.

[innerlee]

@MystK, I just find that this happens only when using vscode open a subfolder of a git repository. working on the repository root folder works normal.

[joaomoreno]

This is due to a lack of file system events within the .git folder, when scoping VS Code to a sub directory of the repository.

@bpasero Let's talk about this when you're back.

[felixfbecker]

It works for me, it just takes a while...

[rhires]

Actually...it used to track, but it doesn't anymore. That's what I'm seeing on Windows7 anyway with the latest update.

[bpasero]

@joaomoreno thats a tough one, would you want explicit API to listen to a .git folder in the parent chain? I think if git does not find a .git folder to listen on, it should fallback to some polling mechanism instead.

[joaomoreno]

Well ideally any contribution in Code should be able to tell the filesystem events contribution hey, I wanna listen to all file change events at this given root. The watcher should then accommodate that request.

Moving this to feature request and renaming title.

[bpasero]

PR is up at #139881

@joaomoreno just to clarify, the need from the Git extension is to be able to watch some files inside the .git folder for changes? Is that a recursive watcher need or just flat?

[eamodio]

For GitLens (and I'm pretty sure the Git extensions needs) it would need to be recursive, so that we can watch for specific changes within the .git folder.

Here are the watcher patterns GitLens uses today:
https://github.com/gitkraken/vscode-gitlens/blob/76e9abdceb70632bc4472aacb347bc4e994e742f/src/git/models/repository.ts#L243-L258

[Jolg42]

For Prisma's extension we are currently using chokidar and it's recursive
https://github.com/prisma/language-tools/blob/main/packages/vscode/src/plugins/prisma-language-server/index.ts#L65-L82

[bpasero]

I just looked at the Git extension and it does not require recursive watching:

vscode/extensions/git/src/watch.ts

Line 16 in a6177d9

const dotGitWatcher = fs.watch(location);

The patterns from GitLens seem fine to me and would work with what I am building in the PR.

@Jolg42 you seem to be listening to changes just to a single file but you need to scan any node_modules folder within the workspace, not just top level, is that right?

[Jolg42]

@bpasero The most common use case would be indeed watching 1 file that could be in any node_modules folder. It might also be that we watch 2 files in 2 different node_modules folders.

[karrtikr]

Just FYI for Python extension, we currently use chokidar to watch for python binaries in environment folders outside workspace upto depth 2: https://github.com/microsoft/vscode-python/blob/456ac068e95ba600079ddcdc0cc7551d1659a699/src/client/common/platform/fileSystemWatcher.ts#L57-L81

Common patterns we watch for: link

[bpasero]

Thanks for providing examples, just to clarify what #139881 will allow you to do: vscode.workspace.createFileSystemWatcher always allowed you to pass in a Uri via RelativePattern, i.e.:

createFileSystemWatcher(new vscode.RelativePattern(vscode.Uri.file('<some folder path>'), '*.js'))

Previously we would silently ignore a provided path that is outside the workspace. After my change, paths will be watched, even when they are outside the workspace with 2 different strategies depending on the pattern used:

• recursive when you provide a complex glob pattern (e.g. **/*.js) via parcel-bundler/watcher
• flat when you provide a simple pattern (e.g. *.js) via fs.watch

However, when it comes to how to handle excludes, I am still not sure what to do so that extensions can efficiently use this new behaviour for file watching. To clarify: a user setting files.watcherExclude allows a user to configure glob patterns that will filter file events. The original idea behind this was to give the user a way to reduce CPU load when a workspace contains large folders that are known to change a lot but should be ignored for watching (such as node modules folder or any build output folder). By default we configure this setting to be:

	"files.watcherExclude": {
		"**/.git/objects/**": true,
		"**/.git/subtree-cache/**": true,
		"**/node_modules/*/**": true,
		"**/.hg/store/**": true
	}

Besides performance considerations, this setting is also required on Linux to reduce the number of opened file handles the watcher consumes. Unfortunately on Linux, recursive watching requires 1 file handle per folder in the hierarchy and only with these exclude rules are we able to reduce this overhead.

To be consistent, this setting needs to apply for extension watched paths as well, otherwise a user would not be in control anymore how file watching works. Given we have 2 different watch strategies, I am thinking:

• for recursively watched folders, apply excludes
• for non-recursive folders, do not apply excludes

The downside of this approach is that currently an extension cannot indicate the excludes to use, nor overwrite the default excludes we apply. This probably needs a new overload of the createFileSystemWatcher method to provide them (similar to findFiles method that has a exclude?: GlobPattern | null parameter).

However, even if we were to provide extensions with a way to override excludes, you would still not be able to watch the node_modules folder within a workspace. The reason for that is that we setup recursive watchers for all workspace folders right on startup and we don't allow extensions to add additional watchers for the workspace, because that would result in multiple watchers on the same paths competing against each other.

Open for thoughts and ideas how to proceed. My feeling is that my PR makes git extensions happy because watchers can be installed on git repositories outside the opened workspace. However those extensions that need to watch in a location that is excluded by default (i.e. node_modules) will have a harder time to use this for their needs I believe...

[eamodio]

@bpasero will this be landing in the January release and available for extensions or will it have to be proposed first? I'm hoping and assuming the former as I didn't see any proposed d.ts changes in the PR. I'm hoping to get my hands on this ASAP as the Git Worktrees feature shipping soon in GitLens kind of requires this (which I've recently come to realize) for many setups.

Say if you have a repo in:
/code/project/repo

By default, I will put worktrees in: (though this is still TBD)
/code/project/repo/.git/.worktrees/<branch-name>
And if you then open that folder, the real .git folder will be a parent of the workspace folder.

Alternatively, a user can pick a different folder (often I've seen people create worktree folders as peer's to the main):
/code/project/repo.<branch-name> or something like that

Which again would suffer the same problem since again the real .git folder will be outside the workspace folder.

Thanks!

[bpasero]

This is planned to be available in our upcoming January release without the need of proposed API. As I had indicated before, an extension can watch any path that is not within the opened workspace folder hierarchy. For non-recursive watchers, no excludes will apply, but for recursive watchers, the same excludes apply that apply for workspace folder watching.

[bpasero]

Small update for a change I pushed for #146066: with this change you can now create a file system watcher that is watching inside the workspace and the files.watcherExclude setting will not apply to you if the watcher is non-recursive.

In other words: this enables you to watch a folder or file and be 100% sure that file events get delivered to you even if the user has configured exclude rules (or when our default exclude rules would impact the results).

This does not apply for recursive watching though because the exclude rules are there for a reason, to reduce overhead for recursive watching.

[bpasero]

Another update: with #194776 there is now proposed new API for createFileSystemWatcher that enables an extension to fully control file watching both recursive and flat bypassing any of the VS Code configured exclude rules.

With that change, file events are exclusively routed back to the extension making the request, thus reducing the performance overhead. No component in VS Code core and no other extension will receive these file events.

The only requirement for this to work is that you have to provide a set of glob patterns to exclude yourself. If no exclude rules are provided, we resort back to the old behaviour where in-workspace recursive watching is blocked.

vscode/src/vscode-dts/vscode.proposed.createFileSystemWatcher.d.ts

Lines 10 to 37 in 895a9ff

	export interface FileSystemWatcherOptions {
	/**
	* Ignore when files have been created.
	*/
	readonly ignoreCreateEvents?: boolean;
	/**
	* Ignore when files have been changed.
	*/
	readonly ignoreChangeEvents?: boolean;
	/**
	* Ignore when files have been deleted.
	*/
	readonly ignoreDeleteEvents?: boolean;
	/**
	* An optional set of glob patterns to exclude from watching.
	* Glob patterns are always matched relative to the watched folder.
	*/
	readonly excludes?: string[];
	}
	export namespace workspace {
	export function createFileSystemWatcher(pattern: RelativePattern, options?: FileSystemWatcherOptions): FileSystemWatcher;
	}

Finalization is tracked in #169724