From 3a81a5650b0ac421f5961b25e2fa1eb204b11db2 Mon Sep 17 00:00:00 2001 From: Matthew Ahrens Date: Tue, 10 Mar 2020 10:59:10 -0700 Subject: [PATCH] deprecate deduplicated send streams Dedup send can only deduplicate over the set of blocks in the send command being invoked, and it does not take advantage of the dedup table to do so. This is a very common misconception among not only users, but developers, and makes the feature seem more useful than it is. As a result, many users are using the feature but not getting any benefit from it. Dedup send requires a nontrivial expenditure of memory and CPU to operate, especially if the dataset(s) being sent is (are) not already using a dedup-strength checksum. Dedup send adds developer burden. It expands the test matrix when developing new features, causing bugs in released code, and delaying development efforts by forcing more testing to be done. As a result, we are deprecating the use of `zfs send -D` and receiving of such streams. This change adds a warning to the man page, and also prints the warning whenever dedup send or receive are used. In a future release, we plan to: 1. remove the kernel code for generating deduplicated streams 2. make `zfs send -D` generate regular, non-deduplicated streams 3. remove the kernel code for receiving deduplicated streams 4. make `zfs receive` of deduplicated streams process them in userland to "re-duplicate" them, so that they can still be received. Closes #7887 Signed-off-by: Matthew Ahrens --- cmd/zfs/zfs_main.c | 10 ++++++++++ include/libzfs_impl.h | 1 + lib/libzfs/libzfs_sendrecv.c | 20 ++++++++++++++++++++ man/man8/zfs-receive.8 | 12 ++++++++++++ man/man8/zfs-send.8 | 3 +++ 5 files changed, 46 insertions(+) diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c index d2ec39893e94..86410510eb0d 100644 --- a/cmd/zfs/zfs_main.c +++ b/cmd/zfs/zfs_main.c @@ -4328,6 +4328,16 @@ zfs_do_send(int argc, char **argv) } } + if (flags.dedup) { + (void) fprintf(stderr, + gettext("WARNING: deduplicated send is " + "deprecated, and will be removed in a\n" + "future release. (In the future, the flag will be " + "accepted, but a\n" + "regular, non-deduplicated stream will be " + "generated.)\n\n")); + } + if (flags.parsable && flags.verbosity == 0) flags.verbosity = 1; diff --git a/include/libzfs_impl.h b/include/libzfs_impl.h index 276241e936a7..45c344c83666 100644 --- a/include/libzfs_impl.h +++ b/include/libzfs_impl.h @@ -71,6 +71,7 @@ struct libzfs_handle { int libzfs_pool_iter; char libzfs_chassis_id[256]; boolean_t libzfs_prop_debug; + boolean_t libzfs_dedup_warning_printed; }; #define ZFSSHARE_MISS 0x01 /* Didn't find entry in cache */ diff --git a/lib/libzfs/libzfs_sendrecv.c b/lib/libzfs/libzfs_sendrecv.c index 3dc7b0705c4f..a5b18bdad0f2 100644 --- a/lib/libzfs/libzfs_sendrecv.c +++ b/lib/libzfs/libzfs_sendrecv.c @@ -4699,6 +4699,26 @@ zfs_receive_one(libzfs_handle_t *hdl, int infd, const char *tosnap, (void) printf("found clone origin %s\n", origin); } + if (!hdl->libzfs_dedup_warning_printed && + (DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) & + DMU_BACKUP_FEATURE_DEDUP)) { + (void) fprintf(stderr, + gettext("WARNING: This is a deduplicated send stream. " + "The ability to send and\n" + "receive deduplicated send streams is deprecated. " + "In the future, the\n" + "ability to receive a deduplicated send stream with " + "\"zfs receive\" will be\n" + "removed. However, in the future, a utility will be " + "provided to convert a\n" + "deduplicated send stream to a regular " + "(non-deduplicated) stream. This\n" + "future utility will require that the send stream be " + "located in a\n" + "seek-able file, rather than provided by a pipe.\n\n")); + hdl->libzfs_dedup_warning_printed = B_TRUE; + } + boolean_t resuming = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) & DMU_BACKUP_FEATURE_RESUMING; boolean_t raw = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) & diff --git a/man/man8/zfs-receive.8 b/man/man8/zfs-receive.8 index ac5711ca6712..24e4e16cc97b 100644 --- a/man/man8/zfs-receive.8 +++ b/man/man8/zfs-receive.8 @@ -105,6 +105,18 @@ destroyed by using the .Nm zfs Cm destroy Fl d command. .Pp +Deduplicated send streams can be generated by using the +.Nm zfs Cm send Fl D +command. +\fBThe ability to send and receive deduplicated send streams is deprecated.\fR +In the future, the ability to receive a deduplicated send stream with +.Nm zfs Cm receive +will be removed. +However, in the future, a utility will be provided to convert a +deduplicated send stream to a regular (non-deduplicated) stream. +This future utility will require that the send stream be located in a +seek-able file, rather than provided by a pipe. +.Pp If .Fl o Em property Ns = Ns Ar value or diff --git a/man/man8/zfs-send.8 b/man/man8/zfs-send.8 index 2561cb61cf54..1deedc214bb8 100644 --- a/man/man8/zfs-send.8 +++ b/man/man8/zfs-send.8 @@ -87,6 +87,9 @@ By default, a full stream is generated. .Bl -tag -width "-D" .It Fl D, -dedup Generate a deduplicated stream. +\fBDeduplicated send is deprecated and will be removed in a future release.\fR +(In the future, the flag will be accepted but a regular, non-deduplicated +stream will be generated.) Blocks which would have been sent multiple times in the send stream will only be sent once. The receiving system must also support this feature to receive a deduplicated