[RFC] Inject distant support into neovim lua functions

[chipsenkbeil]

I'd given this thought before and dismissed the idea, but I want to bring it back into the backlog with a refresh look. The idea is to support injecting custom replacement functions into vim.loop, vim.fs, and vim.fn (and maybe others?) where methods that would perform IO or spawn processes get changed to ones that can work remotely.

How would it work?

For each function we want to wrap, we write some custom function that takes in the same arguments and returns a structure that maps to whatever the function would normally return.

The trick is to detect whether we're working remotely or locally. We can already check if there is an active client, but that still wouldn't solve the situation of mixing local and remote commands together. The next thought is to check the buffer to see if it is remote or not. If we're in a non-remote buffer (e.g. plugin.buf.has_data() == false), fall back to the default implementation. Otherwise, use our custom one for distant.

The caveat would be for plugins that create their own buffers, which obviously would not have remote data. The only thought I have there is to also overwrite methods like vim.api.nvim_buf_create so we can automatically set the client id via plugin.buf.set_client_id(...). Not sure how well this would work.

We'd obviously want to make this opt-in as it would break many people's setups. And another option to provide alongside would be a temporary conversion via something like plugin.inject(function() end) where everything is injected, the function is called, and then everything is reset.

Rolling release of feature

It would take a LOT of work to map every function before releasing this. The thought is to handle some of the ones below, and override the __index of loop and other metatables to check what is being accessed. If we want __index to work, we'd need to remove all of the other fields of the table to make sure it always goes through __index since it is only triggered when a field is absent.

In the case that we can get __index to work as a proxy, we can check what is being accessed AND if we are going to try to use the distant API. If we know we need to invoke the distant API and the accessed method is not supported, we can error out.

---

Processes

vim.fn.system({cmd} [, {input}])

• This is a wrapper around system() of vim. You feed it a command as either a string or list of strings, and it returns a a string.
  • <CR><NL> is replaced with <NL>
  • NUL characters are replaced with SOH (0x01)
  • Optional input gets written to stdin
  • Error code gets set to v:shell_error

Seems straightforward to support by doing a plugin.api.wrap and then invoking this.

Job Control

vim.fn.jobstart({cmd} [, {opts}])

• Spawns a command as a job, which is simple enough.
• Has similar string vs list of strings situation, and returns a job id which we can map to our process id.
• Quite a few options, many of which we already support like cwd, env, pty/width/height
  • detach - we would remember if this is false and, if neovim is exiting, trigger a kill command
  • overlapped - ignore this
  • rpc - error out if this is true

Can support this by doingplugin.api.wrap and then invoking this. We need to capture some of the options to add to our wrap like cwd, env, pty/width/height. For the others, I think we can leave as-is.

The other job control methods related to this should all work as expected because neovim just thinks we're running a normal process.

File System

vim.loop.new_fs_event()

• Creates a uv_fs_event_t handle for watching some path.
• Captures events in a callback with change and rename as the possibilities.
• Can take some flags, all of which are false by default
  • watch_entry - ???
  • stat- ???
  • recursive - I'm assuming to watch directory changes, but don't know

We should be able to map this to our plugin.api.watch API, and map our changes to those they offer. I'm not sure what the flags do, so would need to figure that out. The other watch methods like fs_event_stop() should work fine if we return an equivalent object.

vim.loop.fs_stat(path, [callback])

• Has a really big table of returned data (when synchronous)
  • atime, mtime, ctime we can get parse from the unix time of metadata's accessed, modified, created
  • type we can get from the file_type of metadata
  • size we can get from the len of metadata
  • mode we can get from unix if set and convert back into the bits for the number AND set the bit for the directory
    • e.g. 16877 is 100000 111 101 101 and has a directory bit set
  • birthtime is unsupported (don't know what this is and how it's different)
  • blksize and block are unsupported
  • dev, flags, gen, gid, ino, nlink, rdev, uid are supported

Should correspond to our plugin.api.metadata method with resolve_file_type as true. There is also fs_fstat and fs_lstat. For fs_lstat, we would have resolve_file_type as false.

From https://stackoverflow.com/a/37268695:

lstat() is identical to stat(), except that if pathname is a symbolic link, then it returns information about the link itself, not the file that it refers to.

fstat() is identical to stat(), except that the file about which information is to be retrieved is specified by a file descriptor (instead of a file name).

For the file descriptor, we should be able to keep track of those from calls to fs_open, which returns a file descriptor.

-- print(vim.inspect(vim.loop.fs_stat('.')))
{
  atime = {
    nsec = 459790968,
    sec = 1683950054
  },
  birthtime = {
    nsec = 839501547,
    sec = 1628224581
  },
  blksize = 4096,
  blocks = 0,
  ctime = {
    nsec = 398172950,
    sec = 1683950054
  },
  dev = 16777230,
  flags = 0,
  gen = 0,
  gid = 20,
  ino = 7152401,
  mode = 16877,
  mtime = {
    nsec = 398172950,
    sec = 1683950054
  },
  nlink = 15,
  rdev = 0,
  size = 480,
  type = "directory",
  uid = 501
}

vim.loop.fs_realpath(path, [callback])

• Resolves path of symbolic links, extra slashes, and relative references. Will also error out if part of the path is missing.

Should be able to use plugin.api.metadata with canonicalize set to true. I can't remember if distant will give an error if canonicalization fails. If it falls back to a default, we may need to add a flag to support reporting an error, or change the behavior.

System Information

There's a lot of these, all with the os_ prefix. I'm going to take a stab at listing a couple, but many of the others beyond these won't have easy support unless we try to retrieve the information using some spawned process trickery.

vim.loop.os_uname()

• Has four string fields
  • machine: corresponds to our arch from system_info
    • From uname -m
  • release: has nothing equivalent right now from system_info
    • From uname -r
  • sysname: is our operating system from system_info
    • From uname -s
    • We need to map the values because os_uname() reports "Darwin" whereas we report "macos"
  • version: has nothing equivalent
    • From uname -v

This is used in nvimlsp-config, so we need to support it. We could potentially spawn four uname processes that map to the above, but not sure what this executes on Windows.

print(vim.inspect(vim.loop.os_uname()))

-- On MacOS
{
  machine = "arm64",
  release = "21.6.0",
  sysname = "Darwin",
  version = "Darwin Kernel Version 21.6.0: Mon Dec 19 20:46:01 PST 2022; root:xnu-8020.240.18~2/RELEASE_ARM64_T8101"
}

-- On Windows
{
  machine = "x86_64",
  release = "10.0.22000",
  sysname = "Windows_NT",
  version = "Windows 10 Home"
}

vim.loop.os_gethostname()

• Returns a string with the hostname, presumably from calling hostname on Unix systems and Windows.

Could potentially just farm this out to a spawned process, unless we update system information to include the hostname.

vim.loop.os_getenv(name, [size])

• Returns the environment variable, or some error constants

We don't actually have a way to retrieve environment variables with distant unless you farm it out to a process. Maybe system information could do this?

Miscellaneous

vim.loop.cwd()

• Returns the current working directory.

Should be able to use plugin.api.system_info to get the current working directory of the server. If we have a custom cwd specified from our settings, we would return that instead.

[chipsenkbeil]

nvimlsp-config support

For this plugin, I'm taking a quick look at all references to io, vim.cmd, vim.fn, vim.loop, and vim.lsp to see what it uses.

io

• open - ignore (ONLY USED IN scripts/docgen.lua)
• lines - inject
  • used in util.lua and vala_ls.lua where the util uses it in insert_package_json used by stylelint_lsp.lua and eslint.lua

vim.fn

• executable - inject
• expand - ignore (I think it's safe to ignore this one? Unless my concern about buffer/file names is valid)
• exepath - inject
  • used by config.lua (not a server) on a server config's command to resolve the full path of the executable
• filereadable - inject
• getcwd - inject
• getenv - inject
• getpid - inject
  • Should this be the pid of the distant server? It is used used in omnisharp.lua with --hostPID
• glob - inject
• has - inject
  • Caveat here is that we only need it for the situation where vim.fn.has('win32') is used since that implies Windows on the remote machine
• isdirectory - ignore (ONLY USED IN test/minimal_init.lua)
• jobstart - inject
• readdir - inject
• stdpath - inject (detect remote OS and use appropriate vim stdpath?)
  • Seems to be used for specifying logging path for an option with the vdmj.lua config
  • Used for part of the command for powershell_es.lua
• system - ignore (ONLY USED IN test/minimal_init.lua)

vim.loop

• cwd - inject
• fs_mkdtemp - ignore (ONLY USED IN scripts/docgen.lua)
• fs_realpath - inject
• fs_stat - inject
• new_timer - ignore
• os_homedir - inject
• os_uname - inject

vim.lsp

• start_client - inject

There's a bunch of others for vim.lsp, but we only care about start_client since that's what we need to wrap. Everything else does not touch the filesystem, spawn processes, etc to my knowledge. There are also a lot of calls to vim.api to get buffer names. My only concern there is if the plugin won't trigger start_client because of the distant://{ID}@ prefix on buffers. If that's the case, we'd need to overwrite buffer name retrieval as well.

For vim.cmd, I'm trying to see if any vimscript commands that do IO/processes are being invoked. If so, we'd have to replace cmd, parse the string, and detect that. If that's the case, this would be a deal-breaker. From skimming the source code, I don't see anything like that, it's just used to set autocommands or invoke :edit for zk.lua when running :ZkNew.

[chipsenkbeil]

nvim-dap support

For this plugin, I'm taking a quick look at all references to io, vim.cmd, vim.fn, and vim.loop to see what it uses. It does make one call to vim.lsp.util, which isn't important.

io

• lines - ignore (used to read a local .vscode/launch.json to extend with entries)
• open - ignore (used for logging of DAP, which we want locally)
• stderr - ignore (used to set neovim stderr for tests)
• stdout - ignore (used to set neovim stdout for tests)

vim.fn

• getcwd - ignore (used to look up .vscode/launch.json relative to local machine cwd)
• delete - ignore (ONLY USED IN tests/debugpy_spec.lua)
• getpid - ignore (returns neovim pid used to filter it out from a list of processes)
• has - inject
  • Caveat here is that we only need it for the situation where vim.fn.has('linux') and vim.fn.has('linux') are used since that implies Linux on the remote machine
• input - ignore
• jobstart - inject
• mkdir - ignore (used for logging of DAP, which we want locally)
• stdpath - ignore (used for logging of DAP, which we want locally)
• system - inject (used to retrieve process information to know which process to debug)
• trim - ignore

vim.loop

• fs_stat - ignore (used to examine local .vscode/launch.json)
• getaddrinfo - inject
  • used to resolve a host; if it fails, they return the unresolved host
• new_tcp - inject
  • used to connect to spawned processes (servers?) that I'm assuming expose a TCP port for debug purposes
• new_timer - ignore
  -os_uname - ignore (used to figure out path separator for logging)
• spawn - inject
  • used to spawn an external terminal
  • used to spawn a server executable
  • used to spawn a session from an adapter
• version - ignore
• walk - ignore (ONLY USED IN tests/run_server.lua)

For vim.cmd, I'm trying to see if any vimscript commands that do IO/processes are being invoked. If so, we'd have to replace cmd, parse the string, and detect that. If that's the case, this would be a deal-breaker. From skimming the source code, I don't see anything like that, it's just used to set autocommands, invoke commands like :DapProgressUpdate, edit a preview window (:pedit), or perform window manipulation.

In terms of gotchas, the new_tcp is the toughest one if we ignore the mix of needing and not needing IO based on reading a local VSCode configuration or writing log files. This is where distant would need to support TCP forwarding similar to SSH where we could run it and it forwards tcp traffic between the local machine and remote machine.

[chipsenkbeil]

From the two examples above, this is clearly quite complex and I don't think the automatic injection will be feasible unless we do something like one of these options:

1. Exclude based on the origin of the call to the function (e.g. you can say to ignore originating from log.lua), and let the origin be configurable. You would detect the origin using debug.getinfo() and see if where we come from:

   function traceback ()
     local level = 1
     while true do
       local info = debug.getinfo(level, "Sl")
       if not info then break end
       if info.what == "C" then   -- is a C function?
         print(level, "C function")
       else   -- a Lua function
         print(string.format("[%s]:%d",
                             info.short_src, info.currentline))
       end
       level = level + 1
     end
   end
   

2. Support turning on & off injection explicitly, still struggling when one plugin needs it remote and another needs it local
3. (most likely) Inject with a function where the injection happens, the function is invoked, and then the injection is undone. This would let you wrap calls to other plugin methods in a way that still supports distant. It just means that you have to do the injection more explicitly versus the seamless method.