Introduce a flag to skip comparing the local mac when raw sending #12981

gamanakis · 2022-01-16T00:22:58Z

Motivation and Context

This is an updated version of PR #11300.
Closes #10523.
Closes #11221.
Closes #11294.
Closes #12594 .

Description

Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferrable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

How Has This Been Tested?

Introduced a new test in ZTS.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

module/zfs/dnode_sync.c

gamanakis · 2022-01-16T19:32:38Z

063ab2e: fix typos.

gamanakis · 2022-01-17T09:43:09Z

@behlendorf This is an updated version of #11300. After much fiddling around with the code I realized that once we zero out the dnode structures of user accounting in os_phys, we will only be able to recalculate user accounting if we dirty all dnodes in the dataset. So I resorted back to the flag option which seems to be working fine.

gamanakis · 2022-01-18T19:41:40Z

I will simplify this by relying on the existing _COMPLETE flags, without introducing a new one.

gamanakis · 2022-01-18T20:54:20Z

2e96c8c: Simplified to rely on existing flag OBJSET_FLAG_USERACCOUNTING_COMPLETE.

tests/zfs-tests/tests/functional/userquota/userspace_send_encrypted.ksh

behlendorf

Thanks for updating this, re-dirtying everything to update the accounting does seem like the best way to handle this since we can't rely on the zk_version.

module/os/freebsd/zfs/zio_crypt.c

module/os/linux/zfs/zio_crypt.c

gamanakis · 2022-01-19T08:26:57Z

2e96c8c: small fixes from @nabijaczleweli and @behlendorf.

gamanakis · 2022-01-19T13:00:16Z

85eb8fd and ef9d2d5: Fixed the awk arithmetic in the test to return user accounting in integer MB.

Raw receiving a snapshot back to the originating dataset is currently impossible because of user accounting being present in the originating dataset. One solution would be resetting user accounting when raw receiving on the receiving dataset. However, to recalculate it we would have to dirty all dnodes, which may not be preferrable on big datasets. Instead, we rely on the os_phys flag OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is incomplete when raw receiving. Thus, on the next mount of the receiving dataset the local mac protecting user accounting is zeroed out. The flag is then cleared when user accounting of the raw received snapshot is calculated. Signed-off-by: George Amanakis <gamanakis@gmail.com>

Raw receiving a snapshot back to the originating dataset is currently impossible because of user accounting being present in the originating dataset. One solution would be resetting user accounting when raw receiving on the receiving dataset. However, to recalculate it we would have to dirty all dnodes, which may not be preferable on big datasets. Instead, we rely on the os_phys flag OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is incomplete when raw receiving. Thus, on the next mount of the receiving dataset the local mac protecting user accounting is zeroed out. The flag is then cleared when user accounting of the raw received snapshot is calculated. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: George Amanakis <gamanakis@gmail.com> Closes openzfs#12981 Closes openzfs#10523 Closes openzfs#11221 Closes openzfs#11294 Closes openzfs#12594 Issue openzfs#11300

This is not associated with a specific upstream commit but apparently a local diff applied as part of: commit e92ffd9b626833ebdbf2742c8ffddc6cd94b963e Merge: 3c3df3660072 17b2ae0 Author: Martin Matuska <mm@FreeBSD.org> Date: Sat Jan 22 23:05:15 2022 +0100 zfs: merge openzfs/zfs@17b2ae0b2 (master) into main Notable upstream pull request merges: openzfs#12766 Fix error propagation from lzc_send_redacted openzfs#12805 Updated the lz4 decompressor openzfs#12851 FreeBSD: Provide correct file generation number openzfs#12857 Verify dRAID empty sectors openzfs#12874 FreeBSD: Update argument types for VOP_READDIR openzfs#12896 Reduce number of arc_prune threads openzfs#12934 FreeBSD: Fix zvol_*_open() locking openzfs#12947 lz4: Cherrypick fix for CVE-2021-3520 openzfs#12961 FreeBSD: Fix leaked strings in libspl mnttab openzfs#12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD openzfs#12981 Introduce a flag to skip comparing the local mac when raw sending openzfs#12985 Avoid memory allocations in the ARC eviction thread Obtained from: OpenZFS OpenZFS commit: 17b2ae0

gamanakis marked this pull request as ready for review January 16, 2022 10:00

nabijaczleweli reviewed Jan 16, 2022

View reviewed changes

module/zfs/dnode_sync.c Outdated Show resolved Hide resolved

rincebrain added Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature labels Jan 16, 2022

gamanakis force-pushed the raw_sends23f_squash branch from e98cd64 to 063ab2e Compare January 16, 2022 19:32

gamanakis force-pushed the raw_sends23f_squash branch from 063ab2e to 2e96c8c Compare January 18, 2022 20:53

nabijaczleweli reviewed Jan 18, 2022

View reviewed changes

tests/zfs-tests/tests/functional/userquota/userspace_send_encrypted.ksh Outdated Show resolved Hide resolved

tests/zfs-tests/tests/functional/userquota/userspace_send_encrypted.ksh Outdated Show resolved Hide resolved

behlendorf approved these changes Jan 19, 2022

View reviewed changes

module/os/freebsd/zfs/zio_crypt.c Outdated Show resolved Hide resolved

module/os/linux/zfs/zio_crypt.c Outdated Show resolved Hide resolved

gamanakis force-pushed the raw_sends23f_squash branch from 2e96c8c to 707b949 Compare January 19, 2022 08:26

gamanakis force-pushed the raw_sends23f_squash branch from 707b949 to 85eb8fd Compare January 19, 2022 12:59

gamanakis force-pushed the raw_sends23f_squash branch from 85eb8fd to ef9d2d5 Compare January 19, 2022 13:02

behlendorf added the Status: Accepted Ready to integrate (reviewed, tested) label Jan 19, 2022

gamanakis mentioned this pull request Jan 20, 2022

Sometimes raw send on encrypted datasets does not work when copying snapshots back #12594

Closed

behlendorf merged commit 63a2645 into openzfs:master Jan 21, 2022

gamanakis mentioned this pull request Feb 4, 2022

Input/Output Error when sending an encrypted incremental dataset back to it's source #11983

Closed

gamanakis mentioned this pull request Apr 10, 2022

Encrypted dataset created under current SmartOS/Illumos cannot be mounted on Linux, gives Input/output error #13301

Open

AttilaFueloep mentioned this pull request May 30, 2022

Filesystem can not be mounted: Input/output error #13521

Open

rincebrain mentioned this pull request Jul 31, 2022

I/O error on mounting encrypt fs after upgrading #13709

Closed

Andreas-Marx mentioned this pull request Nov 17, 2022

permanent errors (ereport.fs.zfs.authentication) reported after syncoid snapshot/send workload #11688

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a flag to skip comparing the local mac when raw sending #12981

Introduce a flag to skip comparing the local mac when raw sending #12981

gamanakis commented Jan 16, 2022 •

edited

Loading

gamanakis commented Jan 16, 2022

gamanakis commented Jan 17, 2022

gamanakis commented Jan 18, 2022

gamanakis commented Jan 18, 2022

behlendorf left a comment

gamanakis commented Jan 19, 2022

gamanakis commented Jan 19, 2022 •

edited

Loading

Introduce a flag to skip comparing the local mac when raw sending #12981

Introduce a flag to skip comparing the local mac when raw sending #12981

Conversation

gamanakis commented Jan 16, 2022 • edited Loading

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

gamanakis commented Jan 16, 2022

gamanakis commented Jan 17, 2022

gamanakis commented Jan 18, 2022

gamanakis commented Jan 18, 2022

behlendorf left a comment

Choose a reason for hiding this comment

gamanakis commented Jan 19, 2022

gamanakis commented Jan 19, 2022 • edited Loading

gamanakis commented Jan 16, 2022 •

edited

Loading

gamanakis commented Jan 19, 2022 •

edited

Loading