Add flag to only disable pathFsOpts.ClientInodes (but keep caches)

[adrian-bl]

gocryptfs currently doesn't play well with btrfs subvolumes due to pathfs's inode mapping:

Steps to reproduce

Assume the following setup:

/foo <-- btrfs filesystem and cipherdir
/foo/M6LB0sjNfK2HT2c0UbAaMw <--- subvolume (btrfs subvol create .....)
/foo/cULkSfhPk6LTWKsPPYO2Sw <--- another subvolume

M6LB0sjNfK2HT2c0UbAaMw and cULkSfhPk6LTWKsPPYO2Sw are the encrypted names of two directories in the root of the encrypted filesystem. Let's call them dir1 and dir2 in cleartext.

Mounting such a setup with gocryptfs /foo /bar will serve a filesystem at /bar with 2 subfolders, just as expected:

$ ls /bar
dir1 dir2

However: The issue with such a setup is that M6LB0sjNfK2HT2c0UbAaMw and cULkSfhPk6LTWKsPPYO2Sw are considered to be separated devices by BTRFS / VFS and therefore may use the same inodes (this is not exclusive to btrfs subvolumes: ZFS or a mount within the cipherdir would have the same eissue).

When this happens (inode collision in the 2 folders), gocryptfs logs
2019/11/04 21:24:22 Found linked inode, but Nlink == 1, ino=8191, fullPath="... and the client sees various fun errors (usually '..not a directory..').

Looking at the go-fuse code shows that the error comes from here:
https://github.com/hanwen/go-fuse/blob/master/fuse/pathfs/pathfs.go#L570
..after it detected that something is fishy with its clientInodeMap - which is not surprising as the code only tracks the inode but omits the device id - therefore making it easy to have 'bogus' data in the cache.

My current workaround is to use the --sharedstorage flag which disables pathFSOpts.ClientInodes - but also disables all caching, which isn't really needed in this situation.

It would therefore be great to have a flag/option to only disable pathFSOpts.ClientInodes while still keeping the caching functionality.

(Also: it may make sense to disable ClientInodes on BTRFS and ZFS by default? .. not sure :-) )

Oh: and thanks a lot for writing gocryptfs!

[rfjakob]

This is actually a bug in gocryptfs. The device number in those subvolumes is different, right? gocryptfs should not get confused by inodes on different devices.

[adrian-bl]

The device number in those subvolumes is different, right?

Yes: each btrfs or zfs subvolume has its own unique device id. (...something which gocryptfs should probably also expose in the fuse mount)

[rfjakob]

Should be fixed now. Inodes from different devices are now mapped to the number range above 10000000000000000000 . If you have to opportunity to test again this would be very much appreciated!

[adrian-bl]

Thanks a lot for looking into this!

I can confirm that the issue i was seeing is fixed in the current HEAD.

Simple verification test:

TEST contains 2 btrfs subvolumes, each subvolume has a few files:

with gocryptfs at HEAD:

# find TEST -exec stat {} \; | awk '/Device/ { print $4 } ' | sort | uniq -c |sort -n
      1 10000000000000000000
      1 10000000000000000001
      1 10000000000000000002
      1 10000000000000000012
      1 10000000000000000013
      1 10000000000000000014
      1 10000000000000000015
      1 10000000000000000016
      1 10000000000000000017
      1 10000000000000000018
      1 10000000000000000019
      1 10000000000000000020
      1 10000000000000000021
      1 10000000000000000022
      1 10000000000000000023
      1 10000000000000000024
      1 10000000000000000025
      1 10000000000000000026
      1 10000000000000000027
      1 10000000000000000028
      1 10000000000000000029
      1 10000000000000000030
      1 10000000000000000031
      1 10000000000000000032
      1 10000000000000000033
      1 10000000000000000034
      1 10000000000000000035
      1 10000000000000000036
      1 10000000000000000037
      1 10000000000000000038
      1 10000000000000000039
      1 10000000000000000040
      1 10000000000000000041
      1 10000000000000000042
      1 10000000000000000043
      1 10000000000000000044
      1 10000000000000000045
      1 262
      1 269

Version without bb6155a:

# find TEST -exec stat {} \; | awk '/Device/ { print $4 } ' | sort | uniq -c |sort -n
find: ‘TEST/b/xyz’: Not a directory   # <- ouch
      1 260
      1 261
      1 263
      1 264
      1 265
      1 266
      1 267
      1 268
      1 271
      1 272
      1 273
      2 256 # <- dup
      2 258 # <- dup
      2 262 # <- dup
      2 269 # <- dup
      2 270 # <- dup