Invalid path error during sparse checkout in versions newer than 2.24.0

[puigru]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.28.0.windows.1
cpu: x86_64
built from commit: 77982caf269b7ee713a76da2bcf260c34d3bf7a7
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Versión 10.0.17763.1339]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad++
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Nothing special about the enviroment. Can reproduce on a fresh install.
However, the repository I'm working with contains a folder with filenames considered invalid on Windows (e.g :)

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git config core.sparseCheckout true
(echo /* & echo !other.7z) >.git\info\sparse-checkout
git checkout

• What did you expect to occur after running these commands?

The repository would be checked out except for the folder other.7z, which contains filenames considered invalid on Windows.
Works as expected if I downgrade to a version older than 2.25.0, such as 2.24.0 or 2.23.0.

• What actually happened instead?

git reports invalid path errors and doesn't check out the repository, despite the path being excluded by the sparse checkout

error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.CGX'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.CGX.BAK'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.COL'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.COL.BAK'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/var/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/var/srd/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/exp/:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/fzero.lzh/fzero/RP5A22:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/fzero.lzh/fzero/RP5C77:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/kart/mak.lzh/mak/:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/msdos/prn.c'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/util/...'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/util/prn.c'

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

https://github.com/XXLuigiMario/gigaleak

EDIT: Found the origin of this issue, this is caused by the core.protectNTFS setting. Disabling it lets the checkout go through without issues. However, this is a bug because no problematic paths are involved in this checkout!

[rimrul]

This is not a bug. CON and PRN are reserved names, ... ends in a period, the others contain a pipe or a colon.

[PhilipOakley]

Maybe we need a note somewhere to clarify that sparse checkout will still check the validity of all filenames, and the excluding the reserved names from the checkout won't stop the error report of an invalid path.

Side thought: While CON and PRN are reserved, so are any files with any extensions to those base names (CON.*, PRN.*) - I haven't looked at the sparse definitions given by @XXLuigiMario for that aspect (and don't for get the case insensitivity, so con.* and prn.*).

[puigru]

@rimrul I'm aware, but that's precisely why I chose to do a sparse checkout. To prevent git from trying to create these files. It shouldn't stop me in my tracks for something I'm not trying to do.

In addition, the error message doesn't make it clear what's going on. It sounds as if it is still trying to create those files despite being excluded, and doesn't provide any guidance in case you know what you're doing.

[rimrul]

Honestly, I didn't read the issue carefully enough. It does indeed seem counter-intuitive to get these errors for files that shouldn't be part of the sparse checkout.

[PhilipOakley]

@XXLuigiMario did you confirm that your .git\info\sparse-checkout was actually effective in stopping the checkout of the directory at that level.

Test with a non zip directory and file of appropriate names. (the .7z suggests that you have a zipped file containing those inappropriate paths - I could be wrong)

Also, it maybe that Windows is "confusing" the zip file as being a directory for the checkout purposes. (I have not attempted to copy, clone, or test the repo..)

[puigru]

@PhilipOakley Yes, other.7z originated from an archive but is actually the name of a folder in this instance. The sparse-checkout rule was effective because once setting core.protectNTFS to false it checks out the repository without issue with the exception of the problematic folder.

I have created a test repo on a Linux machine as follows:

.
├── problem
│   ├── bad:file
│   └── yi|kes
└── README

1 directory, 3 files

Cloned it on Windows and repeated the commands from the issue, replacing other.7z with problem.
With core.protectNTFS enabled I get:

error: invalid path 'problem/bad:file'
error: invalid path 'problem/yi|kes

Once disabled:

Your branch is up to date with 'origin/master'.

And the README file is created, as expected.

[PhilipOakley]

With core.protectNTFS enabled I get:

error: invalid path 'problem/bad:file'
error: invalid path 'problem/yi|kes

Once disabled:

Your branch is up to date with 'origin/master'.

So, the core.protectNTFS is being over zealous, or over early, in its 'protection' of the NTFS system from potential bad file names. (an opportunity to review the code @XXLuigiMario ;-)

Maybe the sparse checkout needs to be more cautious about where the protection happens. Maybe @derrickstolee has some insight on the sparse approach.

[derrickstolee]

I don't think this is a sparse-checkout issue, but instead it is just a core.protectNTFS issue. That config option doesn't allow bad paths into the index at all, even if those paths are not being populated by the sparse-checkout.

Changing this behavior has the potential to re-introduce security problems, so it is not something I will recommend. Perhaps someone who really knows what is going on could look into it, but I doubt this is a good idea.

[PhilipOakley]

a core.protectNTFS issue. That config option doesn't allow bad paths into the index at all, even if those paths are not being populated by the sparse-checkout.

Thanks for that clarification.

Changing this behavior has the potential to re-introduce security problems,

Worth a clarification to make sure everyone is on the same page about how the issues are handled (separate discussion)

Looks like the docs need a tweak to clarify the index vs file system aspect:

core.protectNTFS

    If set to true, do not allow checkout of paths that
    would cause problems with the NTFS filesystem, 
    e.g. conflict with 8.3 "short" names.
    Defaults to true on Windows, and false elsewhere.

Any change would also affect core.protectHFS

[puigru]

@PhilipOakley I would also suggest tweaking how this situation is presented to the user within git, currently it makes it seem like it is actually trying to create these paths (and failing) instead of a security feature.

[rimrul]

Maybe we could add a hint to sparse checkout if one of those 2 options is enabled, that there might be error messages about those files, even if they're excluded from checkout?

[iamlgh]

I think this something should be done about it, even if it's just documentation. If you don't set "git config core.protectNTFS false", then you can't clone, change to a branch, or pull files from a branch that has these errors, with sparse checkout. It's fatal, it's not that there are just error messages, as @rimrul stated.

[wdtj]

This was indeed the case with my issue (above). I've spent about 2 days trying to resolve the issue. I whole hardheartedly suggest that something be changed to prevent this in the future. Either a warning to set core.protectNTFS when sparse-checkout is specified or similar an enhancement to remap bad filepaths.

With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to re-open this issue or accept #2803.

[derrickstolee]

As @dscho went to great lengths to detail what would need to happen here without re-introducing a security issue, I would say that this is currently not meeting the bar for our effort at the moment.

The best workaround is to not have these paths in your repo if you can avoid it. (I know we would have a difficult time making the case that torvalds/linux should rename their aux.c file, but usually it is best to use paths that work on all platforms whenever possible.)

I'm investigating an area that may make this possible to work around in some cases but not 100% of the time. It just happens to be a side-effect of the much-more-important case that I'm handling. When that is ready I'll update this issue if that works to help this situation.

[mikekaganski]

I don't know if it's the same issue, but the same error happens when the paths contain a backslash (which is normal on Windows), like this one for https://bibisect.libreoffice.org/win32-5.1

error: invalid path 'instdir/program/python-core-3.3.3/lib\__pycache__/quopri.cpython-33.pyc'
error: invalid path 'instdir/program/python-core-3.3.3/lib\__pycache__/random.cpython-33.pyc'
error: invalid path 'instdir/program/python-core-3.3.3/lib\__pycache__/re.cpython-33.pyc'

$ git --version
git version 2.38.0.windows.1

[PhilipOakley]

error happens when the paths contain a backslash (which is normal on Windows),

Unfortunately, that's the one 'special' character on the Linux/Posix side of Git. While some care is applied to try and correctly detect the 'isDir' character, it's best to avoid mixed usage if possible.

[dscho]

error: invalid path 'instdir/program/python-core-3.3.3/lib\__pycache__/quopri.cpython-33.pyc'

I think there is a misunderstanding going on here. This path contains a directory whose name is lib\__pycache__. It's not a __pycache directory inside a lib directory. The directory's name is literally lib\__pycache__.

This is important because on Linux, you can have such a directory, and you can have a __pycache__ directory inside a lib directory next to it.

But on Windows, directory names cannot contain backslash characters. Git is doing the right thing here to error out.

[mikekaganski]

I am very much sure that the repository was created on Windows. The files (and directories) for it were created on Windows, this is absolutely guaranteed (LibreOffice for Windows can only be built on Windows still). As such, I doubt very much that this repo contains some files/paths that are impossible on Windows.

But since I'm not a git expert, I can't check what is in the metadata of this. Just sharing my thoughts :)

[PhilipOakley]

doubt very much that this repo contains some files/paths that are impossible on Windows.

But.... Git assumes Linux/Posix paths by default (it is the SCM for Linux, by design and intent), so Git for Windows is between a rock and a hard place on these issues. Sometimes you have to bite the bullet and focus on the common path 🥲 .

[mikekaganski]

@PhilipOakley :) Please note that issues are to identify and resolve problems, and I posted here to inform about a possible problem, to provide a way to check it (with the respective URL), and possibly to fix. I had workarounded it for myself using git config core.protectNTFS false; and I am not the author of the repo. So I find any comments telling how to avoid it at creation to be really off-topic, unless they mean official "NOTABUG" (which I'd be OK as a resolution, and then simply considered that some bug happened at creation time long ago, using some unknown old git version - then it would be useful to know what specifically happened, and if there was a user error that could be avoided).

[ghost]

error: invalid path 'database/templates/forms/006-grap-money/6006a2401b055/image/bg.svg:Zone.Identifier'
how to solve this
i am try to clone the repo but i am not getting the source code
kindly suggest a answer.

Receiving objects: 100% (3902/3902), 166.39 MiB | 1.17 MiB/s, done.
Resolving deltas: 100% (975/975), done.
error: invalid path 'database/templates/forms/006-grap-money/6006a2401b055/image/bg.svg:Zone.Identifier'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

[dscho]

@mikekaganski is it possible that the invalid path (I am talking about lib\__pycache__, which should be lib/__pycache__ instead) is committed at the tip of the branch you are checking out? If so, that needs to be fixed. You can work around the problem by setting core.protectNTFS=false but you should immediately rename the path using git mv afterwards and commit, and somehow get that integrated into the upstream repository.

[dscho]

error: invalid path 'database/templates/forms/006-grap-money/6006a2401b055/image/bg.svg:Zone.Identifier'

@Dheeraj-kumar1 this is indeed an invalid path. The :Zone.Identifier suffix is a so-called "Alternate Data Stream" (see e.g. this blog post for more information) that is attached to the file in question, and it is not supposed to be committed, and it is certainly not supposed to be checked out. You need to remove this before checking out (e.g. by cloning with --no-checkout and then playing games with the Git index all while operating under core.protectNTFS=false).