Support fallocate(2)

[behlendorf]

Observed by xfstests 075, fallocate(2) is not yet supported

"fallocate is used to preallocate blocks to a file. For filesystems which support the fallocate system call, this is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros."

QA output created by 075
brevity is wit...

-----------------------------------------------
fsx.0 : -d -N numops -S 0
-----------------------------------------------
fsx: main: filesystem does not support fallocate, disabling
: Operation not supported

-----------------------------------------------
fsx.1 : -d -N numops -S 0 -x
-----------------------------------------------

-----------------------------------------------
fsx.2 : -d -N numops -l filelen -S 0
-----------------------------------------------
fsx: main: filesystem does not support fallocate, disabling
: Operation not supported

-----------------------------------------------
fsx.3 : -d -N numops -l filelen -S 0 -x
-----------------------------------------------                                             

[adilger]

It is potentially difficult to meaningfully implement fallocate() for ZFS, or any true COW filesystem. The intent of fallocate is to pre-allocate/reserve space for later use, but with a COW filesystem the pre-allocated blocks cannot be overwritten without allocating new blocks, writing into the new blocks, and releasing the old blocks (if not pinned by snapshots). In all cases, having fallocated blocks (with some new flag that marks them as zeroed) cannot be any better than simply reserving some blocks out of those available for the pool, and somehow crediting a dnode with the ability to allocate from those reserved blocks.

[behlendorf]

Exactly. Implementing this correctly would be tricky and perhaps not that valuable since fallocate(2) is Linux-specific. I would expect most developers to use the more portable posix_fallocate() which presumably falls back to an alternate approach when fallocate(2) isn't available. I'm not aware of any codes which will be to inconvenienced by not having fallocate(2) available... other than xfstests apparently.

[RedBeard0531]

Well you could in theory do something tricky like just creating a sparse file of the correct size. This would avoid the wasted space of storing the zeroed-out data that wouldn't be reusable anyway due to COW. It would unfortunately break the contract that you won't get ENOSPC, but you can't give that guarantee with COW and you would be less likely to hit that after using an enhanced posix_fallocate() since it wouldn't be wasting space on the zeroed pages. Out of curiousity, would there be any difference in final on-disk layout of a sparse file that is filled in vs a file that is first allocated by zero-filling?

I work on mongodb and we use posix_fallocate to quickly allocate large files that we can then mmap. It seems to be the quickest way to preallocate files and have a high probability of contiguous allocations (which again isn't possible due to COW). While I doubt anyone will try to run mongodb on zfs-linux anytime soon (my interest in the project is for a home server), I just wanted to give feedback from a user-space developer's point of view.

[ryao]

Commit cb2d190 should have closed this.

[behlendorf]

I was leaving this issue open because the referenced commit only added support for FALLOC_FL_PUNCH_HOLE. There are still other fallocate flags which are not yet handled.

[cwedgwood]

@dechamps this doesn't seem to be working for 3.6.x. Looking at your patch for this it looks like this is expected. Is there an update for recent kernels?

11570 open("holes", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
11570 write(3, "\252\252"..., 4194304) = 4194304
11570 fallocate(3, 03, 65536, 196608)   = -1 EOPNOTSUPP (Operation not supported)

3 = fd
03 = mode, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE

[dechamps]

My patch only implements FALLOC_FL_PUNCH_HOLE alone, which is not a valid call to fallocate(). It never worked on any kernel, and will never work until someone implements FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE. Right now it's just a placeholder, basically.

[cwedgwood]

@dechamps thanks for that clarification

even with FALLOC_FL_PUNCH_HOLE (only):

12260 open("holes", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 4
12260 write(4, "\252\252"..., 4194304) = 4194304
12260 fallocate(4, 02, 65536, 196608)   = -1 EOPNOTSUPP (Operation not supported)

02 = mode, FALLOC_FL_PUNCH_HOLE

[RJVB]

Apparently fallocate is still not supported on zfs?
But could that be the reason that I cannot seem to use fallocate at all on my systems that have a zfs root, not even on the /boot partition which is on a good ole ext3 slice?

[behlendorf]

@RJVB fallocate() for the zfs filesystems has not yet been implemented, however the won't have any impact on an ext3 filesystem.

[morsik]

You can always use:
dd if=/dev/zero of=bigfile bs=1 count=0 seek=100G
Works immediately.

[behlendorf]

As of 0.6.4 the FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE behavior of fallocate(2) is supported. But as noted above, for a variety of reasons implementing a meaningful fallocate(2) to reserve space is problematic for a COW filesystem.

[CMCDragonkai]

I'm using 0.7.2-1, and I noticed that if you run posix_fallocate on a file with the same size as the length specified, it returns with EBADF. This doesn't happen when I do it on tmpfs.

//usr/bin/env make -s "${0%.*}" && ./"${0%.*}" "$@"; s=$?; rm ./"${0%.*}"; exit $s

#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

int main () {
  int fd = open("./randomfile", O_WRONLY | O_CREAT, S_IRUSR | S_IWUSR);
  if (fd == -1) {
    perror("open()");
  }
  int status = posix_fallocate(fd, 0, 100);
  if (status != 0) {
    printf("%s\n", strerror(status));
  }
  return 0;
}

Running the above on an empty or non-existent file works fine, as soon as you run it again, it fails with EBADF. This is bit strange behaviour.

[behlendorf]

@CMCDragonkai that does seem odd. Can you please open a new issue with the above comment so we can track and fix this issue.

[pandada8]

Is allocating disk space (set mode = 0) supported now ?
I notice fallocate still return EOPNOTSUPP

BTW, will fallocate generate less fragmentation than just truncate in random write scenes ?

[DeHackEd]

No, because ZFS's copy-on-write semantics just plain don't allow that.

[CMCDragonkai]

@behlendorf #6860

[shodanshok]

@behlendorf While it is not possible (due to CoW) to have a fully working fallocate, it would be preferable to have at least a partially-working implementation: some applications[1] use fallocate to create very big files and on non-fallocation filesystem this is a very slow operation. Granted that ZFS and its CoW defeat one of the main fallocate feature (ie: to really reserve space in advance), paying the slow (and SSD-wearing) "fill entire file with 0" behavior is also quite bad.

How would you consider to implement a "fake" fallocation, where fallocate returns success but no real allocation is done? After all, even after a "real" fallocate, reserved space is not guaranteed, as any snapshot can eat into the really available disk space.

[1] One of such application is virt-manager: RAW disk images are, by default, fully fallocated. This, depending on disk size, mean GB or TB of null data (zeroes) written to HDDs/SSDs.

[RJVB]

I'd say that makes sense. According to my fallocate man page

The default operation (i.e., mode is zero) of fallocate() allocates and initializes to zero the disk space within the range specified by offset and len. The file size (as reported by stat(2)) will be changed if offset+len is greater than the file size.

I don't see how that is different from writing `len` 0s to disk starting at `offset`, am I missing something. Either way, maybe doing that write inside the ZFS driver means it can be done more efficiently than when using userland calls?

[shodanshok]

@RJVB On filesystem supporting fallocate, the filesystem reserves len/blocksize blocks and marks them as uninitialized. This has the following consequences:

• as blocks are marked as reserved/allocated, the user space application which called fallocate is sure that sufficient space is available to write to all such blocks;

• as no user data are written (and only some very terse metadata are flushed to disk), fallocate returns almost immediately, enabling very fast file allocations.

Point n.1 (space reservation) is at odds with ZFS because as a CoW filesystems, by its very nature, it continuously allocate new data blocks while keeping track of past ones via snapshots. This means that you can't really count on fallocate to guarantee sufficient disk space to write all blocks, unless you tap into the reservation and/or quota properties. However, if I remember correctly, these properties only apply to an entire dataset, rather than to a single file.

And here come point n.2 - fast file allocation. On platform where fallocate is not natively supported, both the user space application and the libc function can force a full file allocation by writing zeroes for all its length. This is very slow, cause unnecessary wear on SSD and is basically useless on ZFS. Hence my suggestion to always return "true" for fallocate, even when doing nothing (ie: faking a successfull fallocate).

Opinions are welcomed!

[RJVB]

- as no user data are written (and only some very terse metadata are flushed to disk), `fallocate` returns almost immediately, enabling very fast file allocations.

You'll notice that the manual does mention writing 0s to disk. And as to COW systems being at odds with space reservation: btrfs supports it and AFAIK that's a COW filesystem too.

tap into the reservation and/or quota proprierties. However, if I remember correctly, these proprierties only apply to an entire dataset, rather than to a single file.

That doesn't matter for the space reservation aspect, right? Available space is not a per-file property but one of the filesystem/dataset. OK, the file size may not show up as one might expect, but aren't we all used to open files not always showing their exact size while available disk space does show the actual value.

the libc function can force a full file allocation by writing zeroes for all its length. This is very slow, cause unnecessary wear on SSD *and* is basically useless on ZFS.

Why is it basically useless? You still get the space reservation, no? My point is that doing the write in the driver might be more efficient. I don't disagree with your suggestion but software that nowadays falls back to the brute-force method because fallocate() fails might start behaving unexpectedly. Maybe a driver parameter that can be controlled at runtime could activate a low-level actual write-zeroes-to-disk implementation? Another solution would be to implement a per-file reservation attribute, something like a simulated file size (which can only be larger than the actual file size) which is taken into account for the determination of available disk space (but not for used disk space?). I really fail to see how this would not provide a usable implementation. There are probably combinations with existing file size/content and the fallocate offset/size parameter that I can't get my head around, but you should always be able to let the fallocate() call fail if you detect one of those. That wouldn't be the 1st ZFS feature that is developed on ZoL first and only then presented for upstreaming to OpenZFS. And this one might be easy to get upstreamed; I presume *BSD have fallocate too (or could use some form of it). And FWIW, the fake fallocate feature would probably have to be presented for upstreaming too.

[shodanshok]

@RJVB

And as to COW systems being at odds with space reservation: btrfs supports it and AFAIK that's a COW filesystem too

fallocate on BTRFS behave differently than on non-CoW filesystems: while it really allocates blocks for the selected file, any rewrite (after the first write) triggers a new block allocation. This means that file fragmentation is only slightly reduced, and can potentially expose some rough corner with a near-full filesystem.

Why is it basically useless? You still get the space reservation, no?

If you tap onto existing quota/reservation system (which, anyway, operate on dataset rather than single file), yes, I'll end with working space reservation. But if you only count on the fallocated reserved block, any snapshot pinning old data can effectively cause a out of space condition even when writing on fallocated files. Something similar to that:

• fallocate a file writing all zeroes to it
• create a snapshot
• rewrite your fallocated file
• new space is allocated, which can result in ENOSP condition.

My point is that doing the write in the driver might be more efficient. I don't disagree with your suggestion but software that nowadays falls back to the brute-force method because fallocate() fails might start behaving unexpectedly. Maybe a driver parameter that can be controlled at runtime could activate a low-level actual write-zeroes-to-disk implementation?

I really fail to see why an user space application should fail when presented with a sparse file rather than a preallocated one. However, as you suggest, simple let the option be user selectable. In short, while a BTRFS-like fallocate would be the ideal solution, even a fake (user-selectable) implementation would be desiderable.

[rlaager]

@von-copec We are only talking about fallocate() in the ZFS POSIX layer (ZPL) filesystems. If someone puts ext4 on top of a zvol or a file, then ext4 still behaves exactly as it always has.

[ghost]

@von-copec We are only talking about fallocate() in the ZFS POSIX layer (ZPL) filesystems. If someone puts ext4 on top of a zvol or a file, then ext4 still behaves exactly as it always has.

I understand, I was attempting to emphasize that the behavior of another filesystem layer versus the ZPL would be considered correct when it is on top of a (sparse) ZVOL, and so the ZPL doing the same thing would be the "same amount of correctness".

[adilger]

An update on this topic. In the course of implementing fallocate(mode=0) for Lustre-on-ZFS (https://review.whamcloud.com/36506) the dmu_prealloc() function was being used to implement the preallocation. While this patch isn't working yet, it is informative on this topic. The dmu_prealloc() function has been in ZFs for a long time, but is only used on Illumos to preallocate space in a ZVOL for a core dump, so that the core can later write directly into the ZVOL blocks without invoking ZFS, in case ZFS is itself broken by the crash:

zvol_dump_init->zvol_prealloc->dmu_prealloc->dmu_buf_will_not_fill()

This sets DB_NOFILL on every dbuf. The interesting thing is that dmu_prealloc() actually preallocates the blocks on disk, and the DB_NOFILL->WP_NOFILL results in the leaf (data) blocks being marked in dmu_write_policy() with ZIO_COMPRESS_OFF and ZIO_CHECKSUM_OFF. This is essentially what fallocate(mode=0) wants, namely to have reserved space that is not compressed and can be overwritten (at least once, anyway) without running out of space.

Several open questions exist, since there is absolutely no documentation anywhere about this code:

• what does dmu_prealloc() do to blocks that were previously allocated? fallocate(mode=0) must not modify existing blocks, only allocate new blocks.
• the DB_NOFILL appears to make reads of these buffers return zero, which seems correct, so long as it is cleared when the blocks are overwritten. Otherwise, it would not be good if normal writes cannot be read back.
• can the DB_NOFILL buffers be overwritten by normal DMU writes, clearing the DB_NOFILL state?
• does the ZIO_CHECKSUM_OFF flag persist if the block is overwritten via normal DMU IO? That would be unfortunate, as it means dmu_prealloc() blocks would not be safe for user data, but could be fixed with a new WP_* flag.

This seems to be a path toward implementing fully-featured ZFS fallocate(mode=0), possibly with some digging in the guts of the code if the semantics are not quite as needed.

If this doesn't work out, it still seems practical to go the easy route, for which I've made a simple patch that implements what was previously described here and could hopefully be landed with a minimum of fuss. I don't have any idea how long it would take the dmu_prealloc() approach to finish, but it would need the changes in my patch anyway.

[RJVB]

Several open questions exist, since there is absolutely no documentation anywhere about this code:

Probably an open door, but have you tried to answer your questions by poking around under an Illumos implementation?

[adilger]

Several open questions exist, since there is absolutely no documentation anywhere about this code:
Probably an open door, but have you tried to answer your questions by poking around under an Illumos implementation?

Yes, the Illumos implementation references this function exactly once, in the code path referenced above, but no actual comments exist in the code that describe these functions.

[RJVB]

Yes, the Illumos implementation references this function exactly once, in the code path referenced above, but no actual comments exist in the code that describe these functions.

I meant empirically, triggering the situations/behaviours you have questions about.

[shodanshok]

@adilger I am right that this preallocation would use the preallocated blocks for the first write only? If so, this seems somewhat similar to BTRFS approach. If so, I am missing why an application (Lustre, in this case) should expect fallocate being really honored considering that:

• a snapshot can easily eat into the pool free space, leading to ENOSPC even if prellocation was successful

• rewriting the file will cause ongoing fragmentation, negating any performance benefit from the previous allocation (with some time)

Disabling compression and checksum seems a way too high price to pay for the very limited benefit (if any) which can be obtained by "true" preallocation on ZFS.

Considering how posix_fallocate simply write zeroes on a filesystem not supporting fallocate, and that these zeroes would be converted to a sparse file if compression is enabled, I would simply suggest to create a module option/pool property/flag "faking" true fallocate (returning 0 but ignoring the operation entirely). I know this sounds bad because it broke the "contract" of the fallocate API itself; however, no such guarantees exists for compressing, CoW filesystem (which seems similar to what you are proposing here, right? #10408)

[adilger]

@shodanshok, I understand and agree that all of those issues exist.

Lustre is a distributed parallel filesystem that layers on top of ZFS, so it isn't the thing that is generating the fallocate() request. It is merely passing on the fallocate() request from a higher-level application down to ZFS, after possibly remapping the arguments appropriately.

"I would simply suggest to create a module option/pool property/flag
"faking" true fallocate (returning 0 but ignoring the operation entirely)"

I've essentially done exactly that with my PR#10408. However, while this probably works fine for a large majority of use cases, it would fail if eg. an application is trying to fallocate multiple files in advance of writing, or in parallel, but there is not actually enough free space in the filesystem. In that case, each individual fallocate() call would verify enough space is available, but the aggregate of those calls is not available. Fixing this would need "write once" semantics for reserved blocks (similar to what dmu_prealloc() provides), or at least an in-memory grant that reserves space from statfs() for the dnode that is released as dbufs are written to the dnode. That would at least avoid obvious multi-file issues, but not prevent other writers from consuming this space. It would also get tricky with fallocate() over a non-sparse file, and whether writes are overlapping, etc. so would not be the preferred solution IMHO.

[behlendorf]

Closing. As discussed above basic fallocate() support was added by #10408.

[svaroqui]

This bug is fixed in MariaDB 10.1.48, 10.2.35, 10.3.26, 10.4.16, 10.5.7 by MariaDB Pull Request #1658 a.k.a. adding fall-back logic for the code EOPNOTSUPP