[libc++] use copy_file_range for fs::copy #109211
Conversation
|
Thank you for submitting a Pull Request (PR) to the LLVM Project! This PR will be automatically labeled and the relevant teams will be notified. If you wish to, you can add reviewers by using the "Reviewers" section on this page. If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers. If you have further questions, they may be answered by the LLVM GitHub User Guide. You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums. |
llvmbot
added
the
libc++
|
@llvm/pr-subscribers-libcxx Author: Jannik Glückert (Jannik2099) ChangesHi, this optimizes This behaviour is in line with Boost.Filesystem and libstdc++ (where I implemented this some time ago). It's also the default behaviour of the standard file copy operations in .NET Core, Java, and (hopefully) soon python, and in the I don't have the opportunity to test this on FreeBSD, but I think the FreeBSD folks will want a look at this. Sadly I don't know any by name, perhaps someone could CC the relevant person(s). As for the Lastly I am not sure if the largefile handling (see usage of Full diff: https://github.com/llvm/llvm-project/pull/109211.diff 2 Files Affected:
diff --git a/libcxx/src/CMakeLists.txt b/libcxx/src/CMakeLists.txt
index 48c5111a0acbf6..3f97d3e730a42c 100644
--- a/libcxx/src/CMakeLists.txt
+++ b/libcxx/src/CMakeLists.txt
@@ -173,6 +173,12 @@ if (APPLE AND LLVM_USE_SANITIZER)
endif()
endif()
+include(CheckCXXSymbolExists)
+check_cxx_symbol_exists("copy_file_range" "unistd.h" LIBCXX_USE_COPY_FILE_RANGE)
+if(LIBCXX_USE_COPY_FILE_RANGE)
+ list(APPEND LIBCXX_COMPILE_FLAGS "-D_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE")
+endif()
+
split_list(LIBCXX_COMPILE_FLAGS)
split_list(LIBCXX_LINK_FLAGS)
diff --git a/libcxx/src/filesystem/operations.cpp b/libcxx/src/filesystem/operations.cpp
index d771f200973528..42d16c0546ae10 100644
--- a/libcxx/src/filesystem/operations.cpp
+++ b/libcxx/src/filesystem/operations.cpp
@@ -32,6 +32,7 @@
# include <dirent.h>
# include <sys/stat.h>
# include <sys/statvfs.h>
+# include <sys/types.h>
# include <unistd.h>
#endif
#include <fcntl.h> /* values for fchmodat */
@@ -178,8 +179,35 @@ void __copy(const path& from, const path& to, copy_options options, error_code*
namespace detail {
namespace {
+#if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE)
+bool copy_file_impl_copy_file_range(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+ size_t count = read_fd.get_stat().st_size;
+ // a zero-length file is either empty, or not copyable by this syscall
+ // return early to avoid the syscall cost
+ if (count == 0) {
+ ec = {EINVAL, generic_category()};
+ return false;
+ }
+ do {
+ ssize_t res;
+ // do not modify the fd positions as copy_file_impl_sendfile may be called after a partial copy
+ off_t off_in = 0;
+ off_t off_out = 0;
+
+ if ((res = ::copy_file_range(read_fd.fd, &off_in, write_fd.fd, &off_out, count, 0)) == -1) {
+ ec = capture_errno();
+ return false;
+ }
+ count -= res;
+ } while (count > 0);
+
+ ec.clear();
+
+ return true;
+}
+#endif
#if defined(_LIBCPP_FILESYSTEM_USE_SENDFILE)
-bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+bool copy_file_impl_sendfile(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
size_t count = read_fd.get_stat().st_size;
do {
ssize_t res;
@@ -194,6 +222,36 @@ bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_cod
return true;
}
+#endif
+
+#if defined(_LIBCPP_FILESYSTEM_USE_SENDFILE)
+bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+ bool has_copied = false;
+# if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE)
+ has_copied = copy_file_impl_copy_file_range(read_fd, write_fd, ec);
+ if (has_copied) {
+ return true;
+ }
+ // EINVAL: src and dst are the same file (this is not cheaply
+ // detectable from userspace)
+ // EINVAL: copy_file_range is unsupported for this file type by the
+ // underlying filesystem
+ // ENOTSUP: undocumented, can arise with old kernels and NFS
+ // EOPNOTSUPP: filesystem does not implement copy_file_range
+ // ETXTBSY: src or dst is an active swapfile (nonsensical, but allowed
+ // with normal copying)
+ // EXDEV: src and dst are on different filesystems that do not support
+ // cross-fs copy_file_range
+ // ENOENT: undocumented, can arise with CIFS
+ // ENOSYS: unsupported by kernel or blocked by seccomp
+ if (ec.value() != EINVAL && ec.value() != ENOTSUP && ec.value() != EOPNOTSUPP && ec.value() != ETXTBSY &&
+ ec.value() != EXDEV && ec.value() != ENOENT && ec.value() != ENOSYS) {
+ return false;
+ }
+ ec.clear();
+# endif
+ return copy_file_impl_sendfile(read_fd, write_fd, ec);
+}
#elif defined(_LIBCPP_FILESYSTEM_USE_COPYFILE)
bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
struct CopyFileState {
|
CC @emaste @Jannik2099 Note that the CI environment does contain a FreeBSD setup, which shows a failure (a somewhat boring one): (There are also a couple other similar failing cases in the same CI run on buildkite.) |
|
Thanks! Yeah, I forgot to slap on Another note, how TOCTOU-safe should this be? Right now I fstat both in the copy_file_range and in the sendfile impl. If the file size changes inbetween and copy_file_range fails halfway through, this may lead to inconsistent output. |
Jannik2099
force-pushed
the
copy_file_range
branch
from
4e7d8b9
to
a5dc5bf
Compare
|
So I looked at the FreeBSD failure again... FreeBSD does have I'll think of a more robust check. |
so turns out that was on purpose, because FreeBSD I'll have to restructure this a bit then. Unrelated, I also took the opportunity to support copying special zero-length files on linux (such as those in |
opportunistically use copy_file_range (Linux, FreeBSD) where possible. This allows for fast copies via reflinks, and server side copies for network filesystems. Fall back to sendfile if not supported. Signed-off-by: Jannik Glückert <jannik.glueckert@gmail.com>
Virtual linux filesystems such as /proc and /sys may contain files that appear as zero length, but do contain data. These require a traditional userspace read + write loop. Signed-off-by: Jannik Glückert <jannik.glueckert@gmail.com>
Jannik2099
force-pushed
the
copy_file_range
branch
from
a624e29
to
3bc0f7c
Compare
|
Ok, I think this is ready for review now. To summarize: Add This covers the following scenarios:
I have also added a test for the I am not super happy with how |
|
Please rebase onto main to re-trigger CI. The CI instability should be resolved now. |
|
(I'll get to this review in my queue, for now just trying to get CI back on track) |
| #include "test_macros.h" | ||
| #include "filesystem_test_helper.h" | ||
|
|
||
| using namespace std::filesystem; |
There was a problem hiding this comment.
Please use namespace fs = std::filesystem instead. We generally discourage using namespace foo; (I know some places in the test suite don't follow that).
| // This is because the to-be-read data is usually generated ad-hoc by the reading syscall | ||
| // These files can not be copied with kernel-side copies like copy_file_range or sendfile, | ||
| // and must instead be copied via a traditional userspace read + write loop. | ||
| int main(int, char**) { |
There was a problem hiding this comment.
Nitpick: main must always explicitly return 0; in our tests, because that otherwise breaks tests under -ffreestanding (since under that flag main is not treated as a special function).
| // <filesystem> | ||
|
|
||
| // bool copy_file(const path& from, const path& to); | ||
| // bool copy_file(const path& from, const path& to, error_code& ec) noexcept; |
There was a problem hiding this comment.
Right now, you're only testing this overload of the function. Could you update the test to test the other variants as well?
| // error_code& ec) noexcept; | ||
|
|
||
| #include <filesystem> | ||
| #include <cassert> |
There was a problem hiding this comment.
Missing #include <system_error>.
| // a tad fragile if lit ever changes the output name, | ||
| // but the easiest way to assert that the correct data was copied. | ||
| assert(file_size(dest) == sizeof("t.tmp.exe")); |
There was a problem hiding this comment.
Could you instead derive something from argv[0]?
| #if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE) || defined(_LIBCPP_FILESYSTEM_USE_SENDFILE) | ||
| bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) { | ||
| # if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE) | ||
| if (copy_file_impl_copy_file_range(read_fd, write_fd, ec)) { | ||
| return true; | ||
| } | ||
| // EINVAL: src and dst are the same file (this is not cheaply | ||
| // detectable from userspace) | ||
| // EINVAL: copy_file_range is unsupported for this file type by the | ||
| // underlying filesystem | ||
| // ENOTSUP: undocumented, can arise with old kernels and NFS | ||
| // EOPNOTSUPP: filesystem does not implement copy_file_range | ||
| // ETXTBSY: src or dst is an active swapfile (nonsensical, but allowed | ||
| // with normal copying) | ||
| // EXDEV: src and dst are on different filesystems that do not support | ||
| // cross-fs copy_file_range | ||
| // ENOENT: undocumented, can arise with CIFS | ||
| // ENOSYS: unsupported by kernel or blocked by seccomp | ||
| if (ec.value() != EINVAL && ec.value() != ENOTSUP && ec.value() != EOPNOTSUPP && ec.value() != ETXTBSY && | ||
| ec.value() != EXDEV && ec.value() != ENOENT && ec.value() != ENOSYS) { | ||
| return false; | ||
| } | ||
| ec.clear(); | ||
| # endif | ||
|
|
||
| # if defined(_LIBCPP_FILESYSTEM_USE_SENDFILE) | ||
| if (copy_file_impl_sendfile(read_fd, write_fd, ec)) { | ||
| return true; | ||
| } | ||
| // EINVAL: unsupported file type | ||
| if (ec.value() != EINVAL) { | ||
| return false; | ||
| } | ||
| ec.clear(); | ||
| # endif | ||
|
|
||
| return copy_file_impl_fstream(read_fd, write_fd, ec); | ||
| } |
There was a problem hiding this comment.
I would rewrite this as follows, I think that makes it easier to understand and we duplicate very little code:
#if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE)
bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
if (copy_file_impl_copy_file_range(read_fd, write_fd, ec)) {
return true;
}
// EINVAL: src and dst are the same file (this is not cheaply
// detectable from userspace)
// EINVAL: copy_file_range is unsupported for this file type by the
// underlying filesystem
// ENOTSUP: undocumented, can arise with old kernels and NFS
// EOPNOTSUPP: filesystem does not implement copy_file_range
// ETXTBSY: src or dst is an active swapfile (nonsensical, but allowed
// with normal copying)
// EXDEV: src and dst are on different filesystems that do not support
// cross-fs copy_file_range
// ENOENT: undocumented, can arise with CIFS
// ENOSYS: unsupported by kernel or blocked by seccomp
if (ec.value() != EINVAL && ec.value() != ENOTSUP && ec.value() != EOPNOTSUPP && ec.value() != ETXTBSY &&
ec.value() != EXDEV && ec.value() != ENOENT && ec.value() != ENOSYS) {
return false;
}
ec.clear();
return copy_file_impl_fstream(read_fd, write_fd, ec);
}
#elif defined(_LIBCPP_FILESYSTEM_USE_SENDFILE)
bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
if (copy_file_impl_sendfile(read_fd, write_fd, ec)) {
return true;
}
// EINVAL: unsupported file type
if (ec.value() != EINVAL) {
return false;
}
ec.clear();
return copy_file_impl_fstream(read_fd, write_fd, ec);
}
#elif defined(_LIBCPP_FILESYSTEM_USE_COPYFILE)
...And once you rewrite like that, you could also consider inlining copy_file_impl_copy_file_range and copy_file_impl_sendfile into their respective copy_file_impl. IDK if that would simplify the code but it might.
There was a problem hiding this comment.
That doesn't work, copy_file_range will (usually) fail on cross-fs copy, and we'd want to fall back to sendfile in that case, and not abandon ship and do a userspace copy.
Since only Linux and FreeBSD support copy_file_range, and only linux supports sendfile on files, perhaps something like:
#ifdef __linux__ || __FreeBSD__
copy_file_impl() {
#ifdef HAS_COPY_FILE_RANGE
if(copy_file_range()) { return; }
#endif
#ifdef HAS_SENDFILE
if(sendfile()) { return; }
#endif
fstream();
}
#endif
Of course, this will get procedually more messy when another Unix platform implements any combination of copy_file_range and sendfile.
There was a problem hiding this comment.
There is something I don't understand. _LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE, _LIBCPP_FILESYSTEM_USE_SENDFILE and _LIBCPP_FILESYSTEM_USE_COPYFILE are mutually exclusive (right?). My rewriting didn't change any logic at all, it only reorganized the code a bit to define the whole function inside each #if branch. Unless I missed something and my "refactoring" actually changed the logic?
There was a problem hiding this comment.
No, _LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE and _LIBCPP_FILESYSTEM_USE_SENDFILE are not mutually exclusive. The intent is to attempt copy_file_range, and fall back to sendfile otherwise.
There was a problem hiding this comment.
Ah, I see what I missed now.
I would suggest something like this (which is basically what you wrote above IIUC):
#if HAS_SENDFILE
copy_file_impl_sendfile() { ... }
#endif
#if HAS_COPY_FILE_RANGE
copy_file_impl_copy_file_range() { ... }
#endif
#if HAS_COPY_FILE
copy_file_impl_copyfile() { ... }
#endif
copy_file_impl_fstream() { ... }
copy_file_impl() {
#if something
full implementation
#elif something
full implementation
#else
full implementation
#endif
}Does that make sense? Decouple the implementation of the different functions and the place where you compose them together, so that we can see at a glance what copy_file_impl does when looking at it.
There was a problem hiding this comment.
if you want the full implementation inlined into copy_file_impl, then what's the point of defining e.g. copy_file_impl_sendfile above?
There was a problem hiding this comment.
By "full implementation", I simply meant that you wouldn't interleave code within the #ifdef and code outside the #ifdefs, I didn't mean that you would inline everything recursively into copy_file_impl.
| include(CheckCXXSymbolExists) | ||
| check_cxx_symbol_exists("copy_file_range" "unistd.h" LIBCXX_USE_COPY_FILE_RANGE) | ||
| if(LIBCXX_USE_COPY_FILE_RANGE) | ||
| list(APPEND LIBCXX_COMPILE_FLAGS "-D_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE") | ||
| endif() |
There was a problem hiding this comment.
What Linux versions don't provide copy_file_range? I'd rather assume support from the OS or test for something general like #ifdef __linux__ in operations.cpp than add a CMake check for a single OS feature like this.
There was a problem hiding this comment.
since linux 4.5, according to the man page. RHEL 8 is at linux 4.18, for comparison. I don't think any still supported distro uses < 4.5
On FreeBSD, it has been added in FreeBSD 13, and FreeBSD 12.x went EoL on 31. December 2023
So yeah, if you want I'll just drop this and do an #ifdef __linux__ || __FreeBSD__
Hi,
this optimizes
std::filesystem::copy_fileto use thecopy_file_rangesyscall (Linux and FreeBSD) when available. It allows for reflinks on filesystems such as btrfs, zfs and xfs, and server-side copy for network filesystems such as NFS.This behaviour is in line with Boost.Filesystem and libstdc++ (where I implemented this some time ago). It's also the default behaviour of the standard file copy operations in .NET Core, Java, and (hopefully) soon python, and in the
cpprogram from GNU coreutils.I don't have the opportunity to test this on FreeBSD, but I think the FreeBSD folks will want a look at this. Sadly I don't know any by name, perhaps someone could CC the relevant person(s).
As for the
LIBCXX_USE_COPY_FILE_RANGEcmake check, I was unsure where to put it, and this is probably not the desired location.Lastly I am not sure if the largefile handling (see usage of
off_t) is correct.