`from_ivset_guid`check missing on resumed recv #9818

problame · 2020-01-07T18:11:35Z

System information

ZoL 0.8.2

Describe the problem you're observing

Given fs an FS with snapshots taken before the bookmark_v2 feature was enabled.
And given recvfs which does have bookmark_v2 enabled.
Both on 0.8.2.
Both have zfs_disable_ivset_guid_check=0.

zfs send -w -i @... fs@pre_bookmark_v2 | zfs recv -s recvfs

fails with the ZFS_ERR_FROM_IVSET_GUID_MISSING error
but leaves a resume token on recvfs.

A subsequent

zfs send -t TOKEN_FROM_RECV | zfs recv -s recvfs

succeeds, thereby effectively circumventing the existence check for matching from_ivset_guid.

Describe how to reproduce the problem

See above.

The text was updated successfully, but these errors were encountered:

tcaputi · 2020-01-08T21:24:34Z

OK. So there seems to be two issues here. First, the first send probably shouldnt leave a resume token. Second, I'm not sure how the resumable send is getting past the same checks that stopped the first send. If you have a pool without bookmark_v2 handy, can you provide the output of the following command?:

zfs send -t TOKEN_FROM_RECV | zstreamdump

If its not easy, I can work around it. In the meantime I'll work on the first issue.

problame · 2020-01-09T10:05:13Z

Forwarded instructions to the owner of the affected pool (don't have shell access to it).

problame · 2020-01-09T12:48:09Z

$ sudo zfs send -w zroot/root/home@zfs-auto-snap_daily-2018-10-25-22h00U | dd bs=1M count=5 | sudo zfs recv -s zroot/testrecv
dd: warning: partial read (1340 bytes); suggest iflag=fullblock
0+5 records in
0+5 records out
22476 bytes (22 kB, 22 KiB) copied, 0,0367145 s, 612 kB/s
cannot receive new filesystem stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t TOKEN
$ zfs get receive_resume_token zroot/testrecv
NAME            PROPERTY              VALUE                                                                                                                                                                                                                                                                                                                                                        SOURCE
zroot/testrecv  receive_resume_token  TOKEN  -
$ sudo zfs send -t TOKEN | zstreamdump
BEGIN record
	hdrtype = 1
	features = 1520004
	magic = 2f5bacbac
	creation_time = 5bd23ce6
	type = 2
	flags = 0xc
	toguid = 1b9d9f2e31d9c5a1
	fromguid = 0
	toname = zroot/root/home@zfs-auto-snap_daily-2018-10-25-22h00U
nvlist version: 0
	resume_object = 0x2
	resume_offset = 0x0
	crypt_keydata = (embedded nvlist)
	nvlist version: 0
		DSL_CRYPTO_SUITE = 0x8
		DSL_CRYPTO_GUID = 0x64c7a3d28b35aaad
		DSL_CRYPTO_VERSION = 0x1
		DSL_CRYPTO_MASTER_KEY_1 = 0x2 0xcc 0x86 0x53 0x3f 0x70 0x8b 0xe2 0x8c 0xcb 0xc 0xe9 0xb1 0x8 0x5a 0x3d 0xaa 0x7 0x62 0x30 0x8b 0x31 0x6c 0xf7 0x63 0x11 0x15 0x48 0x2c 0x5a 0x55 0x4b
		DSL_CRYPTO_HMAC_KEY_1 = 0x14 0xe3 0x3b 0x31 0x83 0xca 0xa5 0xc1 0xc5 0xdc 0x7d 0xd8 0x16 0xd5 0x14 0x7f 0xdf 0x6a 0xee 0x21 0x6c 0xa8 0xc6 0x88 0x76 0xeb 0x8c 0x51 0xdc 0xa4 0x3b 0x88 0x34 0x2a 0x76 0x44 0x11 0xfd 0x43 0x30 0x79 0x22 0xf4 0x3b 0x29 0x98 0x77 0x32 0x6 0x65 0x70 0x96 0x24 0xe8 0x41 0x9 0x95 0x7f 0x97 0x6f 0x3a 0x76 0x77 0x1
		DSL_CRYPTO_IV = 0xa6 0x21 0x81 0x9a 0xa6 0xba 0x6 0x45 0x98 0x2b 0xa2 0x50
		DSL_CRYPTO_MAC = 0xf6 0x9e 0xd1 0x93 0x4d 0x2a 0x51 0x3b 0xb7 0x62 0x84 0x3f 0x6c 0x6b 0x83 0x4f
		portable_mac = 0xd9 0x52 0xf1 0xee 0x7f 0x52 0x66 0x86 0x5e 0x98 0x84 0xbf 0xf6 0xfd 0x38 0x51 0x73 0xe 0xf8 0xe6 0xdc 0x1f 0xfc 0x2d 0x13 0xaf 0x84 0xc3 0xfa 0x2e 0xf0 0x53
		keyformat = 0x3
		pbkdf2iters = 0x55730
		pbkdf2salt = 0x22790cf34d4e3724
		mdn_checksum = 0x0
		mdn_compress = 0x0
		mdn_nlevels = 0x6
		mdn_blksz = 0x4000
		mdn_indblkshift = 0x11
		mdn_nblkptr = 0x3
		mdn_maxblkid = 0x6fff
		to_ivset_guid = 0x0
		from_ivset_guid = 0x0
	(end crypt_keydata)

problame · 2020-01-09T13:09:33Z

To clarify: the pool that produces above send has already been upgraded to bookmark_v2, but the snapshot that is being sent has been taken before the upgrade.

tcaputi · 2020-01-09T19:05:45Z

To clarify: the pool that produces above send has already been upgraded to bookmark_v2, but the snapshot that is being sent has been taken before the upgrade.

Understood. Thank you.

I noticed in the issue description that you experienced the problem when doing a zfs send -i. However, in the zstreamdump output you provided, you redid the send from scratch (without -i). I'm not seeing anything that could have caused the second issue I mentioned yet and you will not get the ZFS_ERR_FROM_IVSET_GUID_MISSING error without a "from" dataset. If possible, can you redo it with the -i?

Currently, the handling for errata openzfs#4 has two issues which allow the checks for this issue to be bypassed using resumable sends. The first issue is that drc->drc_fromsnapobj is not set in the resuming code as it is in the non-resuming code. This causes dsl_crypto_recv_key_check() to skip its checks for the from_ivset_guid. The second issue is that resumable sends do not clean up their on-disk state if they fail the checks in dmu_recv_stream() that happen before any data is received. As a result of these two bugs, a user can attempt a resumable send of a dataset without a from_ivset_guid. This will fail the initial dmu_recv_stream() checks, leaving a valid resume state. The send can then be resumed, which skips those checks, allowing the receive to be completed. This commit fixes these issues by setting drc->drc_fromsnapobj in the resuming receive path and by ensuring that resumablereceives are properly cleaned up if they fail the initial dmu_recv_stream() checks. Fixes: openzfs#9818 Signed-off-by: Tom Caputi <tcaputi@datto.com>

problame · 2020-01-10T11:06:48Z

Here the same with an incremental, albeit with different snapshots (both still before the bookmark_v2 upgrade, and we put a zfs hold on them now).

Note that we had to recv with -o origin= because there's not enough space in the pool to hold a complete copy of the incremental source. Shouldn't matter for our purposes here, though.

$ sudo zfs send -w -i @zfs-auto-snap_weekly-2019-12-16-10h02U zroot/root/home@zfs-auto-snap_weekly-2019-12-23-10h12U | dd bs=1M count=5 | sudo zfs recv -s -o origin=zroot/root/home@zfs-auto-snap_weekly-2019-12-16-10h02U zroot/testrecv2
dd: warning: partial read (1340 bytes); suggest iflag=fullblock
0+5 records in
0+5 records out
165436 bytes (165 kB, 162 KiB) copied, 0,00750338 s, 22,0 MB/s
cannot receive new filesystem stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t TOKEN

$ sudo zfs send -t TOKEN | zstreamdump
BEGIN record
	hdrtype = 1
	features = 1520004
	magic = 2f5bacbac
	creation_time = 5e009314
	type = 2
	flags = 0xc
	toguid = 100fe06bf9817334
	fromguid = c9f1c091e6b2f984
	toname = zroot/root/home@zfs-auto-snap_weekly-2019-12-23-10h12U
nvlist version: 0
	resume_object = 0x2
	resume_offset = 0x8000
	crypt_keydata = (embedded nvlist)
	nvlist version: 0
		DSL_CRYPTO_SUITE = 0x8
		DSL_CRYPTO_GUID = 0x64c7a3d28b35aaad
		DSL_CRYPTO_VERSION = 0x1
		DSL_CRYPTO_MASTER_KEY_1 = 0x2 0xcc 0x86 0x53 0x3f 0x70 0x8b 0xe2 0x8c 0xcb 0xc 0xe9 0xb1 0x8 0x5a 0x3d 0xaa 0x7 0x62 0x30 0x8b 0x31 0x6c 0xf7 0x63 0x11 0x15 0x48 0x2c 0x5a 0x55 0x4b
		DSL_CRYPTO_HMAC_KEY_1 = 0x14 0xe3 0x3b 0x31 0x83 0xca 0xa5 0xc1 0xc5 0xdc 0x7d 0xd8 0x16 0xd5 0x14 0x7f 0xdf 0x6a 0xee 0x21 0x6c 0xa8 0xc6 0x88 0x76 0xeb 0x8c 0x51 0xdc 0xa4 0x3b 0x88 0x34 0x2a 0x76 0x44 0x11 0xfd 0x43 0x30 0x79 0x22 0xf4 0x3b 0x29 0x98 0x77 0x32 0x6 0x65 0x70 0x96 0x24 0xe8 0x41 0x9 0x95 0x7f 0x97 0x6f 0x3a 0x76 0x77 0x1
		DSL_CRYPTO_IV = 0xa6 0x21 0x81 0x9a 0xa6 0xba 0x6 0x45 0x98 0x2b 0xa2 0x50
		DSL_CRYPTO_MAC = 0xf6 0x9e 0xd1 0x93 0x4d 0x2a 0x51 0x3b 0xb7 0x62 0x84 0x3f 0x6c 0x6b 0x83 0x4f
		portable_mac = 0x24 0x11 0xaf 0xc3 0xbb 0x63 0x6 0xfc 0x11 0x5b 0x33 0x95 0x81 0xd4 0x95 0xbf 0x4d 0x74 0x19 0x3a 0x8a 0xa8 0xaf 0xaa 0xc5 0xf9 0xbd 0x3c 0x7d 0x97 0xc7 0xb2
		keyformat = 0x3
		pbkdf2iters = 0x55730
		pbkdf2salt = 0x22790cf34d4e3724
		mdn_checksum = 0x0
		mdn_compress = 0x0
		mdn_nlevels = 0x6
		mdn_blksz = 0x4000
		mdn_indblkshift = 0x11
		mdn_nblkptr = 0x3
		mdn_maxblkid = 0xd7ff
		to_ivset_guid = 0x0
		from_ivset_guid = 0x0
	(end crypt_keydata)

tcaputi · 2020-01-10T16:27:16Z

Thanks. I think this should be fixed by #9829

Currently, the handling for errata #4 has two issues which allow the checks for this issue to be bypassed using resumable sends. The first issue is that drc->drc_fromsnapobj is not set in the resuming code as it is in the non-resuming code. This causes dsl_crypto_recv_key_check() to skip its checks for the from_ivset_guid. The second issue is that resumable sends do not clean up their on-disk state if they fail the checks in dmu_recv_stream() that happen before any data is received. As a result of these two bugs, a user can attempt a resumable send of a dataset without a from_ivset_guid. This will fail the initial dmu_recv_stream() checks, leaving a valid resume state. The send can then be resumed, which skips those checks, allowing the receive to be completed. This commit fixes these issues by setting drc->drc_fromsnapobj in the resuming receive path and by ensuring that resumablereceives are properly cleaned up if they fail the initial dmu_recv_stream() checks. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tom Caputi <tcaputi@datto.com> Closes #9818 Closes #9829

Currently, the handling for errata #4 has two issues which allow the checks for this issue to be bypassed using resumable sends. The first issue is that drc->drc_fromsnapobj is not set in the resuming code as it is in the non-resuming code. This causes dsl_crypto_recv_key_check() to skip its checks for the from_ivset_guid. The second issue is that resumable sends do not clean up their on-disk state if they fail the checks in dmu_recv_stream() that happen before any data is received. As a result of these two bugs, a user can attempt a resumable send of a dataset without a from_ivset_guid. This will fail the initial dmu_recv_stream() checks, leaving a valid resume state. The send can then be resumed, which skips those checks, allowing the receive to be completed. This commit fixes these issues by setting drc->drc_fromsnapobj in the resuming receive path and by ensuring that resumablereceives are properly cleaned up if they fail the initial dmu_recv_stream() checks. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tom Caputi <tcaputi@datto.com> Closes openzfs#9818 Closes openzfs#9829

Currently, the handling for errata openzfs#4 has two issues which allow the checks for this issue to be bypassed using resumable sends. The first issue is that drc->drc_fromsnapobj is not set in the resuming code as it is in the non-resuming code. This causes dsl_crypto_recv_key_check() to skip its checks for the from_ivset_guid. The second issue is that resumable sends do not clean up their on-disk state if they fail the checks in dmu_recv_stream() that happen before any data is received. As a result of these two bugs, a user can attempt a resumable send of a dataset without a from_ivset_guid. This will fail the initial dmu_recv_stream() checks, leaving a valid resume state. The send can then be resumed, which skips those checks, allowing the receive to be completed. This commit fixes these issues by setting drc->drc_fromsnapobj in the resuming receive path and by ensuring that resumablereceives are properly cleaned up if they fail the initial dmu_recv_stream() checks. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tom Caputi <tcaputi@datto.com> Closes openzfs#9818 Closes openzfs#9829

behlendorf added Type: Defect Incorrect behavior (e.g. crash, hang) Component: Send/Recv "zfs send/recv" feature labels Jan 7, 2020

tcaputi mentioned this issue Jan 9, 2020

Fix errata #4 handling for resuming streams #9829

Merged

12 tasks

behlendorf closed this as completed in #9829 Jan 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`from_ivset_guid`check missing on resumed recv #9818

`from_ivset_guid`check missing on resumed recv #9818

problame commented Jan 7, 2020 •

edited

Loading

tcaputi commented Jan 8, 2020

problame commented Jan 9, 2020

problame commented Jan 9, 2020

problame commented Jan 9, 2020

tcaputi commented Jan 9, 2020

problame commented Jan 10, 2020

tcaputi commented Jan 10, 2020

from_ivset_guidcheck missing on resumed recv #9818

from_ivset_guidcheck missing on resumed recv #9818

Comments

problame commented Jan 7, 2020 • edited Loading

System information

Describe the problem you're observing

Describe how to reproduce the problem

tcaputi commented Jan 8, 2020

problame commented Jan 9, 2020

problame commented Jan 9, 2020

problame commented Jan 9, 2020

tcaputi commented Jan 9, 2020

problame commented Jan 10, 2020

tcaputi commented Jan 10, 2020

`from_ivset_guid`check missing on resumed recv #9818

`from_ivset_guid`check missing on resumed recv #9818

problame commented Jan 7, 2020 •

edited

Loading