Initial version of cfs-fuse

[alexlarsson]

This is a highly efficient implementation of the parts of erofs that we use in composefs.
With this performance of ls -lR of a fuse mount like this is broadly similar to that of a kernel filesystem, except it works rootless. It is stateless in that all it does is mmap the filesystem image and each fuse request is served directly from this mapping.

It does remap the trusted.overlay.* xattrs to user.overlay.*, with the goal of using this with overlayfs, but unfortunately that requires the user of the overlayfs mount option userxattr, which is incompatilble with redirects and metacopy, so this doesn't work atm.

We could perhaps fix that. Alternative approaches is to make it do the reads itself, but then we don't get the page-sharing aspects. One way to fix that is if we ever get the "fuse support to pass fd:s to the caller" kernel patches that has been floating around lately.

[alexlarsson]

I updated this to add some code to support reading files (by parsing the overlay redirect xattr). So, with this we have basic support for full access. However, performance may not be great, and also so far we have no verity validation in place.

[cgwalters]

I only gave this a quick skim. I can imagine this being useful at least for cross-checking, so no objections to me to merging.

[cgwalters]

@giuseppe asked:

how does it compare to the fuse file system that already exists for erofs? Could this be handled in erofs-tools?

I think a key part of the answer here is that we'd have to add support for the "basedir" bits; it looks to me like the erofs-utils fuse implementation is really really basic. It doesn't have overlay.redirect support I think.

But yes...hmm, maybe it is actually simpler to see if they'd take a patch for that; or worst case if not, just fork the binary for now and add the support. That'd make it easier to unfork later hopefully.

Oh and, it's still using the old FUSE API. It also looks like it's not multithreaded?

[alexlarsson]

@giuseppe asked:

how does it compare to the fuse file system that already exists for erofs? Could this be handled in erofs-tools?

I think a key part of the answer here is that we'd have to add support for the "basedir" bits; it looks to me like the erofs-utils fuse implementation is really really basic. It doesn't have overlay.redirect support I think.

But yes...hmm, maybe it is actually simpler to see if they'd take a patch for that; or worst case if not, just fork the binary for now and add the support. That'd make it easier to unfork later hopefully.

Oh and, it's still using the old FUSE API. It also looks like it's not multithreaded?

There are two replies here.

The first one is that the cfsfuse implements the combination of erofs+overlayfs (although strictly limited to the features composefs uses), whereas erofs fuse supports only the erofs part (but with much more erofs features). It is not possible to use a non-root overlayfs with erofs-fuse to implement the composefs features, because we fundamentally rely on the overlayfs redirect feature, which is incompatible with the overlay userxattrs option which would let us use the redirects.

The second reply is that the two are implemented very differently. The erofs one uses the generic implementation from the erofs-utils, and that code has all sorts of generic code needed for all the tools in that codebase. However, the cfsfuse implementation has inline support for the just the required features. This allows the cfsfuse implementation to be extremely tight. For example, there are zero allocations at runtime, every fs operation is just a few dereferences into a mmaped copy of the filesystem data.

The end result is that cfsfuse performs much better (for the subset we need):

$ erofsfuse cs9.cfs m
$ time ls -lR m > /dev/null
real	0m5,149s
user	0m0,275s
sys	0m0,723s
$ time ls -lR m > /dev/null
real	0m4,939s
user	0m0,247s
sys	0m0,654s

vs

$ tools/cfsfuse m/ -o source=cs9.cfs,basedir=objects
$ time ls -lR m > /dev/null
real	0m1,100s
user	0m0,192s
sys	0m0,538s
$ time ls -lR m > /dev/null
real	0m0,530s
user	0m0,155s
sys	0m0,318s

[alexlarsson]

I pushed some further tweaks that should improve performance more (use splice, use clone-fd).

[giuseppe]

ok then I am fine to merge, just some comments

[hsiangkao]

@giuseppe asked:

how does it compare to the fuse file system that already exists for erofs? Could this be handled in erofs-tools?

I think a key part of the answer here is that we'd have to add support for the "basedir" bits; it looks to me like the erofs-utils fuse implementation is really really basic. It doesn't have overlay.redirect support I think.
But yes...hmm, maybe it is actually simpler to see if they'd take a patch for that; or worst case if not, just fork the binary for now and add the support. That'd make it easier to unfork later hopefully.
Oh and, it's still using the old FUSE API. It also looks like it's not multithreaded?

There are two replies here.

The first one is that the cfsfuse implements the combination of erofs+overlayfs (although strictly limited to the features composefs uses), whereas erofs fuse supports only the erofs part (but with much more erofs features). It is not possible to use a non-root overlayfs with erofs-fuse to implement the composefs features, because we fundamentally rely on the overlayfs redirect feature, which is incompatible with the overlay userxattrs option which would let us use the redirects.

The second reply is that the two are implemented very differently. The erofs one uses the generic implementation from the erofs-utils, and that code has all sorts of generic code needed for all the tools in that codebase. However, the cfsfuse implementation has inline support for the just the required features. This allows the cfsfuse implementation to be extremely tight. For example, there are zero allocations at runtime, every fs operation is just a few dereferences into a mmaped copy of the filesystem data.

The end result is that cfsfuse performs much better (for the subset we need):

$ erofsfuse cs9.cfs m
$ time ls -lR m > /dev/null
real	0m5,149s
user	0m0,275s
sys	0m0,723s
$ time ls -lR m > /dev/null
real	0m4,939s
user	0m0,247s
sys	0m0,654s

vs

$ tools/cfsfuse m/ -o source=cs9.cfs,basedir=objects
$ time ls -lR m > /dev/null
real	0m1,100s
user	0m0,192s
sys	0m0,538s
$ time ls -lR m > /dev/null
real	0m0,530s
user	0m0,155s
sys	0m0,318s

Yes， I suggest don't use erofsfuse for now (or if you later have extended features) since erofsfuse is a too native implementation only for compatibility. I asked some school students to work on improve some performance of erofsfuse this summer but erofs-utils won't address composefs use cases since I think it should be in the composefs project.