Path.is_dir returns false for folders in OneDrive using "Files On Demand"

[roblourens]

Files On Demand is a new OneDrive feature that only downloads files to the local disk when needed.

• Install the latest Windows and OneDrive updates
• Open OneDrive settings, enable Files On Demand
• Run this code with the path to your OneDrive folder:

    let metadata = Path::new("C:\\Users\\roblourens\\OneDrive").metadata().expect("failed");

    println!("{:?}", metadata.file_type());
    println!("{:?}", metadata.is_dir());
    println!("{:?}", metadata.is_file());
    println!("{:?}", metadata.len());

Output:

FileType(ReparsePoint)
false
false
4096

Here's a PR in libuv that fixed an issue with Files On Demand, in case there's a similar issue in Rust: https://github.com/libuv/libuv/pull/1522/files

Ref: BurntSushi/ripgrep#705 and microsoft/vscode#35433

[jonas-schievink]

Are you sure this is a bug? Seems like those folders are indeed reparse points and only look like folders to the user.

[roblourens]

I guess I'm not sure what the expected result is. Node says it's a directory:

require('fs').lstatSync('C:\\Users\\roblourens\\OneDrive').isDirectory(); // => true

[retep998]

When given a reparse point metadata is supposed to return information on the target of the reparse point, not the reparse point itself. To quote the documentation "This function will traverse symbolic links to query information about the destination file." Therefore the fact that this doesn't follow the reparse point and simply says that it is a reparse point is a bug.

[roblourens]

Is there anything I can help with on this bug? It's the root cause of a nasty bug that prevents vscode from searching inside some OneDrive folders. I could attempt a PR, if someone can give me some pointers.

[retep998]

@roblourens You could poke around with calling GetFileInformationByHandle directly on that directory and figure out how to determine that it is indeed a directory.

[roblourens]

Don't know if this helps:

let metadata = Path::new(onedrive_file).metadata().expect("failed");
println!("{:?}", metadata.file_attributes()); // 1056

let metadata2 = Path::new(onedrive_dir).metadata().expect("failed");
println!("{:?}", metadata2.file_attributes()); // 1040

The file attributes for a file and a directory in onedrive - both have FILE_ATTRIBUTE_REPARSE_POINT. The file has FILE_ATTRIBUTE_ARCHIVE (?). The directory has FILE_ATTRIBUTE_DIRECTORY which seems to make sense.

https://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx

[parkovski]

The implementation of FileType actually has a lot of problems. Compare the Unix implementation to the Windows implementation. Let's ignore the fact that there are a lot more factors on the Windows side for now. The particular issue in this thread is caused by line 523, where to be considered a directory, we expect the reparse point flag to be false and the dir flag to be true. That's, um, bad. But pretty much every function in that impl is wrong - it should work the same way the Unix version does, where it stores the native mode and each is_* checks for the existence of the appropriate flag. Maybe someone more familiar with Unix can say whether a symlink to a file should be treated as SYMLINK | FILE or just SYMLINK?

Now, is_symlink_dir is a little harder to solve. Actually, if you say that mount points are symlinks which this does, you're kind of in messy territory here because reparse points are totally up to the FS driver. So one way or another, symlink detection is broken here because you are basically saying a symlink is a reparse point that resolves to another FS location, but you can't make that judgment just from the IO_REPARSE_TAG_* flags. All those tell you is that it's a particular thing that Microsoft implemented, but there's no reason you can't have other implementations that act like symlinks. To actually figure this stuff out correctly you need to do what libuv is doing and try to get the FS driver to treat the reparse point as what you're looking for.

For the purpose of getting ripgrep to work, I would just refactor the methods to check for native flags like I described in the first paragraph. There should not be a SYMLINK_DIR mode - you should check for the existence of a symlink flag and a dir flag, but I'd make a doc comment above that function noting that it's broken and actually only tells you if the thing is a Win32 symlink or a mount point. How it should work may need some discussion, I'm not sure.

[roblourens]

Thanks for the notes. I may try to work around this downstream just for fixing search in VS Code, but I think that coming up with the right fix for Rust is outside my area of expertise. But I'm hoping that someone is able to look at this because I think it's quite a bad bug.

[BurntSushi]

@pitdicker Do you happen to have any context on what led to this change with respect to this bug?

[pitdicker]

I sort-of remember writing that code, but forgot the details. It was an effort to get symlinks on Windows in a mostly working state. Would have to dive into the details again to be of much use to be honest.

As @retep998 commented file types on Windows behave pretty different from Unix. I would not call the logic 'wrong' as @parkovski does, but more a best effort to a cross-platform interface. I think the easiest solution is to change (_, true, _) => FileType::ReparsePoint, to something like (not tested):

(false, true, _) => FileType::File,
(true, true, _) => FileType::Dir,

So just act as if it really is a file or directory if we don't recognize the reparse point as a symlink or directory junction.

[ssokolow]

@roblourens For the record, FILE_ATTRIBUTE_ARCHIVE is an old DOS-era piece of metadata meant for backup tools.

The FAT driver sets the Archive flag whenever you modify a file and a DOS-era backup tool operating in incremental backup mode can then go through, save copies of any files marked "Archive", and then clear the bit to indicate to its future self that they haven't changed since the previous incremental backup.

(In other words, the "archive" part is short for "archival needed" or "please archive")