Support forward slash in Windows paths

[EmilGedda]

Windows accepts forward slashes as path component separators in most of their API's (and in explorer.exe as well as cmd.exe). It would be nice if neo-tree.nvim did the same.

This does not work on Windows:

require("neo-tree.command").execute({ toggle = true, dir = "foo/bar" })

Looks like it tries to open a non-existant path.

This does work:

require("neo-tree.command").execute({ toggle = true, dir = "foo\\bar" })

And the contents of bar is shown.

[pysan3]

Hey @miversen33

Is there any change this issue can be solved with (or something similar to) #1023 ?

[miversen33]

So the gist of this request is basically, whatever function actually resolves the path, it should accept this kind of pathing as well (though do we make it only accept this path if the os is windows? Idk).

As an alternative, we could cheese it and have the filesystem.navigate function just convert all foo/bar references to foo\\bar references (basically convert / -> \\ behind the scenes). I don't imagine it would be tough to do either. Just adding the following

state.path = state.path:gsub('/', '\\')
path_to_reveal = path_to_reveal:gsub('/', '\\')
path = path:gsub('/', '\\')

to lua/neo-tree/sources/filesystem.init#L120

Thinking about it, I don't actually hate that idea. @cseickel thoughts on this?

[cseickel]

So the gist of this request is basically, whatever function actually resolves the path, it should accept this kind of pathing as well (though do we make it only accept this path if the os is windows? Idk).

As an alternative, we could cheese it and have the filesystem.navigate function just convert all foo/bar references to foo\\bar references (basically convert / -> \\ behind the scenes). I don't imagine it would be tough to do either. Just adding the following

state.path = state.path:gsub('\', '//')
path_to_reveal = path_to_reveal:gsub('\', '//')
path = path:gsub('\', '//')

to lua/neo-tree/sources/filesystem.init#L120

Thinking about it, I don't actually hate that idea. @cseickel thoughts on this?

I think you have your slashes reversed. The request is to support forward slashes, unix style. Your suggestion is about escaping backslashes.

I'm guessing there is a branch of code somewhere that interprets a path based on windows conventions if the os is windows which is not recognizing the use of a unix style path separator. I'd need more details on where the problem is before I can make any suggestions.

I honestly never knew that windows would accept both unix style and windows style separators. It seems completely insane to me. What happens if you use both? Does backslash become an escape character or an alternate path separator?

[miversen33]

I think you have your slashes reversed. The request is to support forward slashes, unix style. Your suggestion is about escaping backslashes.

Not quite. I am replacing forward slashes with escaped backslashes. This means a user can request directory foo/bar and the filesystem source will see it as foo\\bar (which is what it expects currently I believe).

EDIT: Reading hard. You are correct, though my general suggestion still stands. Edited the above as well

It seems a bit hacky, but unless I am mistaken, it shouldn't cause any issues with vim.loop (btw this is now vim.uv) or plenary. It is essentially forcing consistent paths before we start messing about with the path string

I honestly never knew that windows would accept both unix style and windows style separators. It seems completely insane to me. What happens if you use both? Does backslash become an escape character or an alternate path separator?

So fun fact, apparently windows treats / and \ the same. I can (for example) navigate to C:/Users\miver and it works just the same as C:\Users\miver and C:/Users/miver. Tested quick in powershell, command prompt and the file explorer. It seems NTFS doesn't care.

Because who needs consistency right? 🙃

[pysan3]

If I understand correctly, in cmd the escape char is ^ and in powershell, it's <backtick> (damn how do I put a backtick in markdown lol)

Both cases, \ does not mean to escape something and is used for path separator.

Neovim's core team seems to prefer using / in Windows: neovim/neovim#15220 (link is broken in that issue, see this commit for details).

But old vim functions prefer backslashes. :=vim.fn.getcwd() => C:\... so now it's a complete mess.

[cseickel]

Because who needs consistency right? 🙃

so now it's a complete mess.

100%

@miversen33 I think you are on the right track, but the place to enforce that consistency is probably where the user input is first taken in, in the command module. Maybe in this function?

neo-tree.nvim/lua/neo-tree/command/parser.lua

Line 81 in 7f6fa04

M.resolve_path = function(path, validate_type)

[pysan3]

I noticed this inconsistency when digging into #1026 and I'm still not sure why there hasn't been much problem in neo-tree's code yet.

1. Opening file.

Let's try to open C:/Users/User/AppData/Local/nvim/lua/target{1..5}.lua while CWD = C:/Users/User/AppData/Local/nvim.
There's a lot of ways to do it.

1. :e lua/target1.lua or :e lua\target1.lua
2. :e C:/Users/User/AppData/Local/nvim/lua/target2.lua (Neotree uses this I think?)
3. :e C:\Users\User\AppData\Local\nvim\lua\target3.lua
4. :e ~/AppData/Local/nvim/lua/target4.lua
5. :e ~\AppData\Local\nvim\lua\target5.lua

2. Look at :ls

The results of files opened with each command correspondes to below.

1. lua\target1.lua
2. ~/AppData/Local/nvim/lua/target2.lua
3. ~\AppData\Local\nvim\lua\target3.lua
4. ~/AppData/Local/nvim/lua/target4.lua
5. ~\AppData\Local\nvim\lua\target5.lua

So, as you can see, C:/Users/User is always substituted by ~ but the type of slashes does matter.

3. Look at vim.fn.expand("%:p")

1. C:\Users\User\AppData\Local\nvim\lua\target1.lua
2. C:\Users\User/AppData/Local/nvim/lua/target2.lua !!!
3. C:\Users\User\AppData\Local\nvim\lua\target3.lua
4. C:\Users\User/AppData/Local/nvim/lua/target4.lua !!!
5. C:\Users\User\AppData\Local\nvim\lua\target5.lua

Now, here's where it starts to get weird.
The result of 2 and 4 have / and \\ mixed up because of the :p.

4. mksession!

Let's save the current session with :mks! which results in a file named Session.vim right inside $CWD.

When you peek inside this file, you'll see lines that prints as below, which has all backslashes replaced with /,
and uses ~.

badd +1 `lua\target1.lua`
badd +1 `~/AppData/Local/nvim/lua/target2.lua`
badd +1 `~/AppData/Local/nvim/lua/target3.lua`
badd +1 `~/AppData/Local/nvim/lua/target4.lua`
badd +1 `~/AppData/Local/nvim/lua/target5.lua`

This behavior is however different from vim which does not substitute backslashes by default.
Introduced in this commit,
related issue.
(Use :h sessionoptions to change the behavior. nvim deletes this option and make / the default.)

vim.fn.getcwd()

This however returns path with all backslashes.

The biggest issue that should be caused by this inconsistency is is_subpath where string.sub(path, 1, string.len(base)) == base might not be true due to difference in slashes.

However neo-tree is surprisingly working as expected in most cases probably thanks to windowize_path but there are certain places where this is not used but should be critical. Like here. I'm quite confused now lol.

@cseickel Maybe we should wrap all calls to vim.fn.expand, getcwd, etc and vim.api.nvim_buf_get_name and substitute backslashes to / and force to use forwardslash throughout the project? Or shall we simply keep it as-is?

[miversen33]

I think you are on the right track, but the place to enforce that consistency is probably where the user input is first taken in, in the command module. Maybe in this function?

Maybe we should wrap all calls to vim.fn.expand, getcwd, etc and vim.api.nvim_buf_get_name and substitute backslashes to / and force to use forwardslash throughout the project? Or shall we simply keep it as-is?

By doing this, we (as Neo-tree) are making the active decision that we will sanitize paths before passing them to the underlying source to handle. Not to say that's a bad thing, just a decision that I didn't feel comfortable making (which is why I moved the suggestion into the filesystem source instead of above in the command resolution, which is where I was initially looking).

IMO, while taking this approach may be the best decision, it might be wise to take it as a 2 step approach. Resolving this direct issue (allowing forward slashes in the path), and then targeting path sanitation/fixing for 3.x

My biggest concern is that one of these approaches is much smaller and easier to validate/verify than the other :)

[cseickel]

By doing this, we (as Neo-tree) are making the active decision that we will sanitize paths before passing them to the underlying source to handle.

That's a good point. I didn't consider that someone could be on windows but still be passing an argument to a source that interacts with a *nix host such as a docker container or remote host.

Given that, your suggestion of handling within the filesystem source is a good fix for this specific issue.

Longer term we should handle paths in a consistent manner throughout the plugin and external sources. As @pysan3 pointed out, the issue is complicated and there are likely many edge cases waiting to be found within our codebase. I think we should drop all representations of paths as strings and replace them with plenary Path objects or come up with our own abstraction that can handle the windows/nix differences properly. Hopefully Plenary's implementation works or we can get fixes in where needed. This might be a v4.x change.

[miversen33]

That's a good point. I didn't consider that someone could be on windows but still be passing an argument to a source that interacts with a *nix host such as a docker container or remote host.

I appreciate you :)

I'll slap together a PR for this this evening, the change is really small so it shouldn't take me too long to do and test.

I think we should drop all representations of paths as strings and replace them with plenary Path objects or come up with our own abstraction that can handle the windows/nix differences properly. Hopefully Plenary's implementation works or we can get fixes in where needed

IIRC Plenary Path's are a touch buggy but I haven't touched them since I was playing around with Telescope back in the day. I do think letting pathing be someone else's problem is a great solution. Properly handling paths is hard.

[WillEhrendreich]

I was trying to look into this a little bit, because it seemed insane and messy. surely enough, that's right, it is insane and messy. I'm totally for the sanitizing everything with the vim.fs.normalize approach, though IIRC there's some interesting problems and discussion around that subject in neovim recently..

ok here's the pr..

it looks like using this to sanitize might be ok now? neovim/neovim@8a7e335

[miversen33]

Quick test with vim.fs.normalize nets the below results

:lua print(vim.fs.normalize("C:\\Users/miver"))
C:/Users/miver

I think we might be able to get away with using normalize in yes. It will take more testing to be sure it works as expected generally though :)

[pysan3]

Could you also stress test non ascii chars as well as emojis included in the path?

We might also have to check how the git related codes work since Windows Git uses a weird encoding for non-ascii characters.

This just came to my mind but would it be fixed if we execute chcp 65001 every time before the git command? I don't have a Windows PC right now so I'm sorry to bother you.

Ref: #731 (comment)

[miversen33]

Could you also stress test non ascii chars as well as emojis included in the path?

We might also have to check how the git related codes work since Windows Git uses a weird encoding for non-ascii characters.

I think we are probably safe. I made a truly horrendous discovery in that NTFS allows emojis in your path

Edit: so does xfs, so apparently linux doesn't care if your path is valid ascii or not either 🙃

Using vim.fs.normalize seems to be the best way to handle this whole path escaping thing IMO

This just came to my mind but would it be fixed if we execute chcp 65001 every time before the git command? I don't have a Windows PC right now so I'm sorry to bother you.

I have no idea what this is so I can't really speak to it unfortunately

Edit 2:
vim.fs.normalize doesn't actually check to see if the location exists, it just seems to standardize the path. For example, I ran

On my linux machine (where the path C:\ is nonsensical) and the function still cleaned it up to something that is "standard". So it is most likely (speculating as I don't feel like digging into the vim source to verify) just running some string regex replaces to make the path "normal" (hence the function name).

[WillEhrendreich]

On my linux machine (where the path C:\ is nonsensical) and the function still cleaned it up to something that is "standard". So it is most likely (speculating as I don't feel like digging into the vim source to verify) just running some string regex replaces to make the path "normal" (hence the function name).

I will fetch dis.

copy-la-pasta, here's the code fasta!

from https://github.com/neovim/neovim/blob/a3f4598226c4d01e4fbc41181a1ad21793862fe3/runtime/lua/vim/fs.lua#L331

--- Normalize a path to a standard format. A tilde (~) character at the
--- beginning of the path is expanded to the user's home directory and any
--- backslash (\\) characters are converted to forward slashes (/). Environment
--- variables are also expanded.
---
--- Examples:
--- <pre>lua
---   vim.fs.normalize('C:\\\\Users\\\\jdoe')
---   --> 'C:/Users/jdoe'
---
---   vim.fs.normalize('~/src/neovim')
---   --> '/home/jdoe/src/neovim'
---
---   vim.fs.normalize('$XDG_CONFIG_HOME/nvim/init.vim')
---   --> '/Users/jdoe/.config/nvim/init.vim'
--- </pre>
---
---@param path (string) Path to normalize
---@param opts table|nil Options:
---             - expand_env: boolean Expand environment variables (default: true)
---@return (string) Normalized path
function M.normalize(path, opts)
  opts = opts or {}

  vim.validate({
    path = { path, { 'string' } },
    expand_env = { opts.expand_env, { 'boolean' }, true },
  })

  if path:sub(1, 1) == '~' then
    local home = vim.uv.os_homedir() or '~'
    if home:sub(-1) == '\\' or home:sub(-1) == '/' then
      home = home:sub(1, -2)
    end
    path = home .. path:sub(2)
  end

  if opts.expand_env == nil or opts.expand_env then
    path = path:gsub('%$([%w_]+)', vim.uv.os_getenv)
  end

  return (path:gsub('\\', '/'):gsub('/+', '/'):gsub('(.)/$', '%1'))
end

yeah, it's simply transforming it, not making sure it exists. It does do some expansion of env vars though, so that's cool. I don't know if that helps, but it's cool.

shocking about the emoji thing.. 🤷‍♂️

I'm envisioning a dystopian future full of filesystems that look like
C:/🎁/👻/😆/🐱‍👤/🏎/🥳/(╯°□°）╯︵ ┻━┻/Q2 fiscal year 1323/straightLaced_biznassREPORTZ.🦅

we will soon have to support the .🦅 file extension.. you have been warned.

[miversen33]

Thank you @WillEhrendreich I appreciate you.

So this is taking longer than expected as I am unsure the best thing to do here. @cseickel, since we can verify that vim.fs.normalize doesn't do any validation and is just "cleaning" a path, should we infact move that up the scope of the request as we talked about here

I feel rather comfortable with that as we can see the source for the function isn't doing anything potentially nefarious with the paths being passed in (and thus we aren't validating them, just normalizing them for subsequent sources. Which IMO is good).

[cseickel]

So this is taking longer than expected as I am unsure the best thing to do here. @cseickel, since we can verify that vim.fs.normalize doesn't do any validation and is just "cleaning" a path, should we infact move that up the scope of the request as we talked about here

I feel rather comfortable with that as we can see the source for the function isn't doing anything potentially nefarious with the paths being passed in (and thus we aren't validating them, just normalizing them for subsequent sources. Which IMO is good).

I would leave this up to you @miversen33, because netman is the most likely extension to have a problem with us changing the paths so you would be the best person to make decisions on that. Also, I don't have a Windows computer that I can use for testing so I can't really verify anything.

[miversen33]

I've submitted PR #1051 that should resolve this. I tested on my windows and linux machines and it appears to be operating "as expected".

@cseickel fun side note, Netman abuses the shit out of Nui's node.extra attribute to do basically everything so it doesn't care about the path on a node :)

[miversen33]

PR #1051 has been merged into main. I am going to close this issue out, though if my resolution did not fix this, please either tag me on this so it can be reopened, or open a new issue :)