automatic reload when file changes externally

[dead10ck]

It would be great if it were detected when the file you are editing has been changed externally, and if possible, automatically reload the file.

I imagine it could work similarly to vim and kakoune, i.e. if there are no modifications that have yet to be written, just reload the file, and treat it as a single modification that can be undone. If there are modifications that haven't been written yet, prompt the user about what they would like to do:

• Ignore the changes, keep that's in the editor buffer, and let a write happen over the external change, whatever it was
• Reload the file anyway, essentially "adding" to the unsaved changes

[archseer]

See #588

[greg-enbala]

Humbly, I prefer helix always alert me before it reloads an open file even if there are no changes to the opened file.

• If there are unsaved changes i think it's clear that you may not want to lose them
• If the document hasn't been changed since it was opened, perhaps I'm deliberately keeping it open while I look at something else from another branch
• If the document has been changed since it was opened and saved, perhaps I accidentally checked out an old version or perhaps i deliberately checked out an old version and have it open in another editor.

Long story short there are always good reasons for me to have helix hold on to a version of the file that's not the current version in the file system.

[jtrees]

Just some quick feedback:

I'm currently trying Helix as an alternative to Kakoune and the lack of file watching really hurts. I frequently have several windows open with Helix where I work on the same files. Since there is no server-client architecture, the buffers get out-of-sync and make this workflow virtually unusable.

I suppose the most viable workaround is to use splits in a single Helix instance, instead of letting my window manager handle the windows.

[ryanfowler]

👋 Hi @pascalkuthe! I see you self-assigned this issue earlier in the year, but I would love to help move this feature along if possible.

I've taken a look at #2653, but it seems like that branch has stalled (and doesn't include reloading open buffers). I'd like to propose some functionality that could be broken up into multiple steps/PRs:

File watching and reloading open buffers
This first step (and likely also the largest) would involve adding "file watching" functionality to the editor. I've looked into both the notify crate, in addition to using an external tool like watchman. I think that using notify is the better approach that doesn't make Helix reliant on external tooling. I think a reasonable first implementation would watch the files in the current working directory, and take a list of excludes (probably globs?) to cut down on noise. VSCode uses some default excludes ("files.watcherExclude") that seem sensible. Once file watching is in place, open buffers that haven't been modified can be reloaded on change.

Watching individual files that are opened, but not in the CWD
For files that are opened but not watched because they are outside of the CWD, they can be separately watched via individual requests to the Watcher.

Support LSP didChangeWatchedFiles
Add support for dynamic_registration with LSPs, and forward requested file changes as necessary. I think this should be quite straightforward once file watching is already in place. Changed files can be compared against the patterns that LSPs request, and sent if there are any matches.

Optimizations for file watching
Specifically for inotify on linux, the exclude globs should be taken into account when traversing the file system to cut down on the number of paths that need to be watched. However, this is likely only a problem on a small number of very large repositories, or on less powerful hardware.

If there is already work underway on this topic, then feel free to dismiss the above. But if not, I'd be interested in people's thoughts and happy to start with the work. Thanks!

[pascalkuthe]

👋 Hi @pascalkuthe! I see you self-assigned this issue earlier in the year, but I would love to help move this feature along if possible.

I've taken a look at #2653, but it seems like that branch has stalled (and doesn't include reloading open buffers). I'd like to propose some functionality that could be broken up into multiple steps/PRs:

File watching and reloading open buffers This first step (and likely also the largest) would involve adding "file watching" functionality to the editor. I've looked into both the notify crate, in addition to using an external tool like watchman. I think that using notify is the better approach that doesn't make Helix reliant on external tooling. I think a reasonable first implementation would watch the files in the current working directory, and take a list of excludes (probably globs?) to cut down on noise. VSCode uses some default excludes ("files.watcherExclude") that seem sensible. Once file watching is in place, open buffers that haven't been modified can be reloaded on change.

Watching individual files that are opened, but not in the CWD For files that are opened but not watched because they are outside of the CWD, they can be separately watched via individual requests to the Watcher.

Support LSP didChangeWatchedFiles Add support for dynamic_registration with LSPs, and forward requested file changes as necessary. I think this should be quite straightforward once file watching is already in place. Changed files can be compared against the patterns that LSPs request, and sent if there are any matches.

Optimizations for file watching Specifically for inotify on linux, the exclude globs should be taken into account when traversing the file system to cut down on the number of paths that need to be watched. However, this is likely only a problem on a small number of very large repositories, or on less powerful hardware.

If there is already work underway on this topic, then feel free to dismiss the above. But if not, I'd be interested in people's thoughts and happy to start with the work. Thanks!

I agree that depending in an external binary like watchmen is not a good idea. However the notify crate is not just non-optimal its implementation is exactly what the kernel documentation tells you not to do. The API is also pretty limited and doesn't play that well with tokio. So to solve this I wanted to implement my own file watcher (that would work more like watchmen, si maimtainting an in memory tree of the the fs that could be quieried, we also have other uses for that). Haven't gotten around to that yet tough

[ryanfowler]

Ah okay, that's fair!

I can create a separate issue for this, but until a file watcher is in place, would you be opposed to adding support for the LSP didChangeWatchedFiles and send events when relevant documents are written to, from within Helix? This would resolve a common pain-point I have of modifying metadata files (e.g. Cargo.toml) in Helix and having to restart the LSP to pull in the changes. That work could then be replaced when full file watching functionality is added sometime in the future.

[pascalkuthe]

That could be a reasonable interim solution to simply send events into a channel in our save implementation and then handle those events in a watcher tokio task. That task can then handle the LSP related stuff (and will already be nontrivial) and keeps the whole thing decoupled from the actual watcher/out of the main code. That does seem like a reasonable first step to me.

You can ignore registering "interest" for now I think (so passing the glob pattern to the actual watcher in the future) and simply filter the glob patterns as required. That's pretty complex and I will probably initially startout watching the entire workspace anyway (I would ignore gitignored file tough to avoid crawling build artifacts). That's what watchmen does too (when you subscribe to sepific diectoies/files it only sends you what you are interested in, but it always track everything internally).

Btw for a really nice experience, I would also send a filewatcher event when we reload inside helix so that if the file changes externally you can still let the LSP know with :reload. I think this also nicely illustrates that we really have two problems:

• the editor becoming aware automatically when a file changes on disk (this issue), currently only manually with :reload or :write
• the editor making the LSP aware when it knows a file has changed (separate concern)

[deifactor]

For what it's worth, I'm running into this a lot where I save my files, run cargo clippy --fix --allow-staged, and then I have to remember to :rla before I continue to work on them. I definitely agree with @pascalkuthe that "reload open files" and "tell LSP when a file has changed" are separate issues.

Would it be okay to just get an initial draft of this functionality that uses notify and only implements the 'reload currently-open files if they changed on disk and are not dirty' part? I'd be up for implementing that.

[TheLostLambda]

This is one of the last few things I really feel I'm missing in Helix — would I be right in thinking the LSP side of this was sorted in #7665 ?

Does that mean what's missing now is just buffer reloading when files change on the disk?

Let me know if that's the case and if there is anything I can do to help out!

[HealsCodes]

One nice addition to the notification and option to auto-reload would also be to have the notification a bit more "in your face".. currently if you try to save a buffer that was externally modified you get a one-line notice in the footer that is extremely easy to miss and disappears on the next cursor move.

I'm constantly running into this issue if I do a git pull --rebase --autostash wich apparently makes helix think any and all open files were modified externally and it usually takes me a few frustrated debug prints() in my code to notice that it's not a problem with the code but with helix refusing to save the file without :w! and showing only the briefest of mentions that that's what's happening.

[rcorre]

I find that I mostly run into this because I suspend helix, run some CLI operations, then resume helix. Reloading changes upon resuming from suspension would fix 90% of this for me. Would that be simpler than implementing a file watcher?

[crabdancing]

@rcorre

Just one person here, but I should probably mention that wouldn't satisfy my use case. I almost always am working simultaneously in an editor and in a terminal. The terminal side of things is often making changes to files (via formatting tools, fixers, sops editing, etc), after which point, I have to manually restart Helix every time, or call :reload-all. This would keep happening regardless of any 'save on suspend' feature.

[webdev23]

As a temporary workaround, we could separately watch for some file changes in a given folder, and send some keystrokes to a byobu/tmux session.

Of course, helix has to run into a tmux session for this to work.

Using inotifywait: sudo apt install inotify-tools

In this simplest example, that command is running into a byobu split, the Helix editor is in the first split. It does reload the file effectively, when helix is in normal mode.

while inotifywait -e modify ~/Desktop; do byobu send-keys -t 0 ":rla" Enter; done;

Using tmux only, we have to pass in the session name and the split number. It allows to be ran in the background. The -r flag is for recursive.

while inotifywait -r -e modify ~/Desktop; do tmux send-keys -t mytmuxsession:0 ":rla" Enter; done;

It works, but of course, this feels very hacky, because it is.
Looking forward for such watcher capabilities built-in.

It could be nice that helix accepts a given POSIX signal to trigger a reload.

Links to help building such watcher script:
https://superuser.com/a/181543
https://unix.stackexchange.com/a/409863

[porridgewithraisins]

another reason this is required: the colors to the left marking git status don't update if I do git commit in another terminal

[TheLostLambda]

Checking in with @pascalkuthe who's down as an assignee — do you have a minimum set of requirements / functionality that you'd like to see for a first draft of this functionality? Are there any partial solutions that would be acceptable?

From looking through things, it seems the obvious place is something that loops into inotify and plays nice with Tokio (would a Linux-only first pass at things be acceptable, or would you like to support all OS's from the get-go?).

For that in-memory filesystem tree you mentioned, what sort of "queries" are you interested in there? Is this another thing that could be added later down the line?

If you have the time, I'd be lovely to get a few indicators as to what you'd see as an acceptable contribution, then some of us with spare time can gradually work our way towards that vision! Working with several Helix instances open in Zellij means that I often run into this limitation and I'm more than happy to try to make some progress on it :)

Thanks a ton!

[pascalkuthe]

The file watcher needs to be a seperate library that should be working well before any PR to helix is amde. It should essentially be a rust version of watchman (as a library instead of as an external process). I got started with ut but it's pretty complex to implement.

[OldManFroggy]

just run into this issue while building a lsp, and testing using Helix, kind of annoying to :rl (reload) all of the time.

I am late to the party, so sorry if this is also covered earlier or elsewhere.

but, is there a way to just add a :watch command that works on active buffer to watrch to see if that file changes, this way you can quick build (yeah, a little hacky) a buffer based watch routine in Helix. (or even simpler, on selecting a buffer check file chanegs, if active buffer just do on a simple timer that starts from a keypress in the buffer and tracks time since last keypress, kinda thang)

this solves issue of and problems with a watch every file solution, and just doing a :watch :watch-off is a lot less hassle.

the tmux / external tools ideas are not user friendly, and doing watchman like tooling and forcing watching across all buffers will not be gerat for those that do not want or need to see an active buffer update.

anyway just my 2c, after hitting rl too manyt times, and simply now using nvim in seperate session to watch the file for updates live... :(

[webdev23]

just run into this issue while building a lsp, and testing using Helix, kind of annoying to :rl (reload) all of the time.

I am late to the party, so sorry if this is also covered earlier or elsewhere.

but, is there a way to just add a :watch command that works on active buffer to watrch to see if that file changes, this way you can quick build (yeah, a little hacky) a buffer based watch routine in Helix. (or even simpler, on selecting a buffer check file chanegs, if active buffer just do on a simple timer that starts from a keypress in the buffer and tracks time since last keypress, kinda thang)

this solves issue of and problems with a watch every file solution, and just doing a :watch :watch-off is a lot less hassle.

the tmux / external tools ideas are not user friendly, and doing watchman like tooling and forcing watching across all buffers will not be gerat for those that do not want or need to see an active buffer update.

anyway just my 2c, after hitting rl too manyt times, and simply now using nvim in seperate session to watch the file for updates live... :(

I had very much the same issue, could simply not use Helix without auto-reloading, so I combined this, it does include a file watcher:

https://github.com/webdev23/shelix

[OldManFroggy]

@webdev23 will take a look later tonight, thanks for the link -> I went oldschool for time being and am now watching local stdout log using tail -f ... works good enough for LSP debugging, must admit Helix is the easiest editor by far for adding a new LSP dont get me started on vscode lsp dev.

[VKondakoff]

I got started with ut but it's pretty complex to implement.

Please, don't give up trying to implement this! Auto-reloading the file, which was changed externally is a must-have feature for a modern (and even post-modern) editor. ;)

[HealsCodes]

Just one small correction: offering an (optional) prompt to reload would be modern.

Sometimes your don't want it to autoreload at all costs or at least want to be able to press U and undo the external changes ;)

Wouldn't be the first time helix saved a file that became a victim of an accidental delete or overwrite - both scenarios where auto reload without prompt would have been fatal.

[ilyagr]

Just one small correction: offering an (optional) prompt to reload would be modern.

I would prefer a prompt if and only if the file was modified both inside and outside the editor. If I haven't modified the file in helix, and it changed on disk, this likely means a VCS checkout, and I don't want a prompt.

[YaLTeR]

Besides, you don't need a prompt to be able to undo across reloads.

[VKondakoff]

What about implementation like this: let's introduce an option on-external-change (what to do when file changes externally) with following parameters:

• ignore (do nothing, if the file is not "dirty")
• reload (default)
• tail (reload and scroll to the end of the file)
• prompt

If the file is "dirty" (has unsaved changes) the prompt to save, reload or ignore should always be fired.

[crabdancing]

@VKondakoff Sounds great! It could even be configured based on regex matches to the file name or somesuch. :)

[VKondakoff]

Here is how it is done in Kakoune.

There is an option:

autoreload enum(yes|no|ask)
default ask
auto reload the buffers when an external modification is detected

Here is a pop-up which is shown when autoreload is set to ask:

[ilyagr]

The file watcher needs to be a seperate library that should be working well before any PR to helix is amde. It should essentially be a rust version of watchman (as a library instead of as an external process).

I have no experience here but, naively, https://github.com/watchexec/watchexec/tree/main/crates/lib might work.

Update: Watchexec seems to be a wrapper around the notify crate, so the following comments apply to it.

[gagbo]

The file watcher needs to be a seperate library that should be working well before any PR to helix is amde. It should essentially be a rust version of watchman (as a library instead of as an external process). I got started with ut but it's pretty complex to implement.

Just so there’s a trace of the reasoning here, what are the issues that make you ignore the notify crate? When I started looking into wrapping inotify sys crate I saw this, and I’ve been wondering what the problem is ever since. The unreliability of inotify on a big number of watchers?

[kirawi]

See #8073 (comment) regarding that

And more directly, https://matrix.to/#/!zMuVRxoqjyxyjSEBXc:matrix.org/$j0U306IWclmC5iO_kgCvn286QGruwM71iqFXem2-xBc?via=matrix.org&via=mozilla.org&via=tchncs.de

The notify crate is the current goto solution in rust but it's too unreliable in my opinion. There is no automatic way to recover from dropped events (which can occur frequently, articels on the linux kernel mailing list explaining how to use inotify specically recommend not to use the architecture notify is using). Instead a client would have to restart notify in those situations which is very slow and makes downstream logic complicated.
There is also no way to query the current fs state (which a filewatcher needs to keep anyway to work properly so maintaining that downstream is both inefficient and quite cumbersome.
There are a bunch more technical shortcomings but those are details that are hard to understand.

[quantonganh]

Just one person here, but I should probably mention that wouldn't satisfy my use case. I almost always am working simultaneously in an editor and in a terminal. The terminal side of things is often making changes to files (via formatting tools, fixers, sops editing, etc), after which point, I have to manually restart Helix every time, or call :reload-all. This would keep happening regardless of any 'save on suspend' feature.

another reason this is required: the colors to the left marking git status don't update if I do git commit in another terminal

If you are using WezTerm, with Helix at the top and a terminal at the bottom, please check out my workaround.

The idea is to emit an event when switching back from terminal (bottom) to the Helix editor (top):

wezterm.on('reload-helix', function(window, pane)
  local top_process = basename(pane:get_foreground_process_name())
  if top_process == 'hx' then
    local bottom_pane = pane:tab():get_pane_direction('Down')
    if bottom_pane ~= nil then
      local bottom_process = basename(bottom_pane:get_foreground_process_name())
      wezterm.log_info('bottom process: ' .. bottom_process)
      if bottom_process == 'lazygit' or bottom_process == 'fish' then
        local action = wezterm.action.SendString(':reload-all\r')
        window:perform_action(action, pane);
      end
    end
  end
end)

  {
    key = '[',
    mods = 'CMD',
    action = act.Multiple {
      act.ActivatePaneDirection 'Up',
      act.EmitEvent 'reload-helix',
    }
  },

[GetPsyched]

Is this being worked on? Seems like there's implementation discussion going on but no linked PR.

However, I just want to state that I agree with both sides (auto-reload and no auto-reload), hence I would prefer if there was a config option for this and I'm not forced into losing my unsaved changes just because something ran in the background.

[Jackojc]

Would really benefit from this feature in my workflow. Often working in a codebase where I need to rebase and files may become stale (then I might accidentally overwrite something without realising). Kakoune prompts me to reload which I think is the best of both worlds.

[TheZoq2]

Modified @quantonganh's wezterm solution a bit to reload any time window focus is changed

-- https://wezfurlong.org/wezterm/config/lua/pane/get_foreground_process_name.html
-- Equivalent to POSIX basename(3)
-- Given "/foo/bar" returns "bar"
-- Given "c:\\foo\\bar" returns "bar"
function basename(s)
  return string.gsub(s, '(.*[/\\])(.*)', '%2')
end

wezterm.on('window-focus-changed', function(window, pane)
    if window:is_focused() then
        local top_process = basename(pane:get_foreground_process_name())
        print(top_process)
        if top_process == 'hx' then
            local action = wezterm.action.SendString(':rla\r')
            window:perform_action(action, pane);
        end
    end
end)

[7flash]

Actually love current behavior of helix allowing me to keep stale versions of my file, as others mentioned, very useful for comparison in between branches. But also need this feature as an alternative to "tail -f" for watching log files.

[TheZoq2]

Modified @quantonganh's wezterm solution a bit to reload any time window focus is changed

-- https://wezfurlong.org/wezterm/config/lua/pane/get_foreground_process_name.html
-- Equivalent to POSIX basename(3)
-- Given "/foo/bar" returns "bar"
-- Given "c:\\foo\\bar" returns "bar"
function basename(s)
  return string.gsub(s, '(.*[/\\])(.*)', '%2')
end

wezterm.on('window-focus-changed', function(window, pane)
    if window:is_focused() then
        local top_process = basename(pane:get_foreground_process_name())
        print(top_process)
        if top_process == 'hx' then
            local action = wezterm.action.SendString(':rla\r')
            window:perform_action(action, pane);
        end
    end
end)

Oh, I just noticed a potential pitfall with this: reload discard changes instead of requiring reload! for changed files, so this is not a great solution :/

[kirawi]

VSCode seems to use https://github.com/parcel-bundler/watcher

I think we can just implement something like this? I'm not sure what @pascalkuthe had in mind for a fleshed out watcher but I think if it works for VSCode than it'll likely be enough for us. A POC w/ bindings to Parcel's implementation might be worth exploring.

[p00ya]

I think a much simpler and specific solution could be implemented. Namely, call stat for the Document in the focused window when it's revised and show a vim-style popup if the mtime has changed before save-time. This is a relatively low performance overhead (only a single path being polled at a time, in response to user edits) and even that could be further reduced with a debounce interval.

Push notifications via a file watcher could later be integrated as an optimization; either using the notify crate, whatever Pascal is working on (if it's ever finished), or something else.

The stat-based solution might not be fit for wider purposes like having a full FS cache for the LSP, but it seems to me like #1125 deserves a short-term solution given the number of whacky solutions people are implementing already (sending signals to helix or even piping keystrokes to their terminal emulator 🤢).

I'm happy to implement the stat-based approach for myself. I tried to touch base with Pascal on Matrix a couple of weeks ago to discuss a PR but got no response.

VSCode seems to use https://github.com/parcel-bundler/watcher

An external process like that is disagreeable for the same reasons as watchman - perhaps more so since it's Node bloatware vs a C++ binary.

While I'm here, it's worth discussing Pascal's reasons for objecting to the notify crate:

The notify crate is the current goto solution in rust but it's too unreliable in my opinion. There is no automatic way to recover from dropped events (which can occur frequently, articels on the linux kernel mailing list explaining how to use inotify specically recommend not to use the architecture notify is using). Instead a client would have to restart notify in those situations which is very slow and makes downstream logic complicated.

This is an intrinsic problem with using a best-effort system API like inotify - the notify crate does at least expose Event::needs_rescan, and yes it requires good-old readdir/stat and making the inotify syscall again, and you probably want to do that periodically anyway. Whatever performance penalty all that involves, you'd incur it no matter what you're wrapping the syscall up in - the best you can do is mitigate it by putting the syscall in a background thread and using epoll to multiplex the I/O (e.g. with tokio).

There is also no way to query the current fs state (which a filewatcher needs to keep anyway to work properly so maintaining that downstream is both inefficient and quite cumbersome.

Just use readdir/stat (via tokio) to query the actual FS (which modern kernels will cache already). For the purposes of #1125, the use of kernel APIs like inotify/fsevent is just an optimization that can save some of these other calls, not a replacement for them.

From Pascal's message on matrix:

System threads are not optimized that way and just spawn 100 threads is going to cause a ton of overhead. Each thread has roughly the same priority so if we spawn 100 notify watchers they get more priority they take significant processing time away from tokio/the rest of the editor. Therefore notify offers an API to easily watch many different files with the same watcher (thread) however you then need a seperate tokio tasks that recives all files changed events and forwards them. That means you need to build some kind of registry to track which channel is interested in which is not trivial to do efficently.

I agree what the notify crate is doing is inferior to integrating with a tokio reactor, but the thread starvation concerns would be pretty minor for #1125 (i.e. there only needs to be a single inotify fd for all open Documents; you can have multiple watches on the one queue/thread).

Also, FWIW, a relatively small change to the notify crate would let it use tokio to wait on the inotify FD, rather than it spawning new threads.

Another issue that Pascal pointed out from reddit:

notify tires to solve two very hard problems: papering over abysmal syscalls provided by various OSes, and implementing a robust model which prevents losing notifications. On the second point, I’ve found it to be insufficient for building robust functionality. Watching fs really requires defining watchman-style clock ids

Using a watchman-style serial clockid is only really relevant if you want to sync cached metadata, which I think is massively over-engineering the solution to #1125 compared to the stat approach I'm suggesting. Helix is already naive/racy in its Document::save_impl implementation, which just compares the Document's mtime to the FS before writing. Helix doesn't need to know about the order of events, just that the file changed externally.

[kirawi]

An external process like that is disagreeable for the #1125 (comment) - perhaps more so since it's Node bloatware vs a C++ binary.

What I proposed is to rewrite it in Rust if a POC using bindings to the C++ library (the JavaScript is only for the WASM wrapper) pans out.

[p00ya]

I have a proof-of-concept for the polling approach in this branch: master...p00ya:helix:file-changed

I haven't implemented a menu/picker yet; it just indicates an external modification via [!] in the status line. I've mainly been testing for correctness and performance, which seem okay so far (with a 2s debounce). Not quite ready for a PR but early feedback welcome.

What I proposed is to rewrite it in Rust if a POC using bindings to the C++ library (the JavaScript is only for the WASM wrapper) pans out.

Ah, I see. That doesn't sound too bad.