zfs send/recv compressed streams from ubuntu 22.04 to ubuntu 20.04 may corrupt your files

[phiser678]

System information

Type	Version/Name	Second OS
Distribution Name	ubuntu	ubuntu
Distribution Version	20.04	22.04
Kernel Version	5.4.0-132-generic	5.15.0-53-generic
Architecture	x86_64	x86_64
OpenZFS Version	0.8.3-1ubuntu12.14	2.1.4-0ubuntu0.1

Describe the problem you're observing

I was doing backups from the newer Ubuntu 22.04 OS to our backup server, but this one is still at Ubuntu 20.04.
When I looked at some files, some of them got corrupted. I checked everything, scrub, network, ram memory to make sure it is not hardware related.

Describe how to reproduce the problem

At the moment I can only reproduce it on my ubuntu 20.04 machine. This is what I do. The zpool is called tank. I have made the backup as such:

ubuntu20.04# ssh ubuntu22.04 zfs send -c tank/test@2|zfs recv tank/test

The file contains a file demo/index.json, if I open it, I can see the json in text.
Now I copy it on the local disk (or a remote server gives the same effect):

zfs send -c  tank/test@2|zfs recv tank/test2

I open the file in demo/index.json, I see a garbage file. Now without compression:

zfs send test@test@2|zfs recv tank/test3

open the file in /tank/test3, I find the normal file.

The problem is not all files in the backup stream get corrupted. I isolated a file and put it in a
test.zip

The zip file contains 2 zfs streams which I got on the ubuntu 20.04 backup:

• bad: is created without -c : zfs send tank/test@2 >bad
• bad.lz4 is made with -c with compression lz4: zfs send -c tank/test@2 >bad.lz4

Suggestion

The suggestion is to use zfs send/recv without -c, or update the backup server to Ubuntu 22.04, this also fixed the bug, but I'm not 100% sure if older zfs clients will mess up to the newer zfs version? Any thoughts?

Include any warning/errors/backtraces from the system logs

No warnings

[ryao]

Wow. This is a very nice report. I cannot look at it immediately, but it is on my radar now. Thanks. :)

[phiser678]

I have a follow up.
So, the bad zfs streams originate from ubuntu 22.04 machines, but in our department we now have about 20 ubuntu 22.04 workstations to back up, and I checked only 6 of them have this weird behavior!
Someone suggested to test compression with gzip instead of lz4, and I noticed if I repack the stream without -c option, the error is gone. However if I use -c compression option in the send command the problem persists.
Another solution would be to just repack the zfs datasets without -c on the affected ubuntu 22.04 machines, and the problem would also be gone...

[rincebrain]

This reminds me of #12762.

It seems like we are sometimes emitting blocks with compression type 0 and actually compressed contents - at least, that's my quick interpretation of this diff segment between zstream dump -vd on the bad and bad.lz4 streams:

Now, the two interesting questions are:

1. is it set to uncompressed on the sender side, or are we losing that information in making the send stream? @phiser678 could you possibly run zdb -dbdbdbdbdbdb path/to/dataset 3283 on whichever dataset you made the "bad.lz4" send stream from (not receiving it first, the original, because I want to know whether it was wrong originally or not)
2. Does this work if you send it to a 2.1.x machine, because I don't see why that should work, here - it's certainly producing mangled data when I receive it on a recent git checkout...

[phiser678]

# zdb -dbdbdbdbdbdb tank/test 3283
Dataset tank/test [ZPL], ID 432759, cr_txg 14182522, 266K, 8 objects, rootbp DVA[0]=<0:16aa1a48e000:3000> DVA[1]=<0:15fc8daaa000:3000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=14186458L/14186458P fill=8 cksum=134b4af99b:34f14af5d97f:4d3210897c9955:4f7709c6f937bf6c

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      3283    1   128K     1K    10K     512     1K  100.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               176   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/demo/index.json
	uid     1000
	gid     1000
	atime	Sun Oct 23 17:29:30 2022
	mtime	Wed Oct 12 09:31:55 2022
	ctime	Wed Nov 16 14:45:53 2022
	crtime	Sun Oct 23 17:29:30 2022
	gen	167888
	mode	100644
	size	514
	parent	2
	links	1
	pflags	840800000004
Indirect blocks:
               0 L0 DVA[0]=<0:168872de3000:3000> [L0 ZFS plain file] fletcher4 lz4 unencrypted LE contiguous unique single size=400L/400P birth=14182523L/14182523P fill=1 cksum=1d13ea9254:17779947edc2:9b5d04dddeb10:2be16f44ce6062c

		segment [0000000000000000, 0000000000000400) size    1K

I checked ashift, the 20.04 server is explicitly set to 12, the 22.04 ubuntu is 0, but I guess this defaults to 12.

[rincebrain]

Interesting.

So, 0 just means "whatever we autodetect the disks to", not necessarily 12, but I think the send stream is wrong anyway before it ever cares about the receiver if it's reporting uncompressed blocks.

Could you quickly double check the output of "zfs version" for me, both lines? Because this really seems like what should have been resolved by #12762's fix, but is also, according to your report, a version after when that should be in...

[phiser678]

Thanks, so the version of Ubuntu 20.04 is 0.8.3-1ubuntu12.14 like I have written on top. It's the default zfs from Ubuntu 20.04 together with the default Ubuntu Linux kernel.

[phiser678]

The workstation originating the faulty zfs stream is indeed formatted as 512 bit block size and the receiving 20.04 server has ashift 2^12 thus 4096

[rincebrain]

I meant for the sender and receiver, since the fix should be on the sender side, but that version should be new enough to have it, in theory.

...in theory because Ubuntu has a bad habit of shipping mixed kernel and userland versions.

edit: ...wait, what the crap?

According to that zdb output, it stored the file compressed...with the same logical/physical size, which would explain the check failing, but raises further questions...especially when the DVA says it's 0x3000 and PSIZE is 0x400.

[rincebrain]

Could you share the output of zdb -l [pool] | grep ashift on the sender side pool, as well as the exact vdev layout? (I know you reported what zpool get ashift said, but I'm trying to figure out how the zdb line you shared is remotely reasonable.)

e: I made a reproducer.

For anyone following along at home, go write something to a single disk vdev pool that compresses with lz4, then do send -c | recv into a draid pool.

You wind up with things like:

               0 L0 DVA[0]=<0:e0145000:9000> [L0 ZFS plain file] fletcher4 lz4 unencrypted LE contiguous unique single size=400L/400P birth=14L/14P fill=1 cksum=1d13ea9254:17779947edc2:9b5d04dddeb10:2be16f44ce6062c

And then boom goes the dynamite when you do send again.

[phiser678]

The zdb -l out gives ashift: 12 on the sender. There is indeed only 1 disk in the sender. The receiving is a raidz2 of 13 disks. So, now you are able to reproduce it?

[rincebrain]

I can reproduce it, yes, but I'm confused how you ended up with that zdb output on a pool with a single disk, as I ended up using a draid vdev for it...

[phiser678]

Sorry for the confusion, but the zdb -dbdbdbdbdbdb tank/test 3283 was on our Ubuntu 20.04 server!

[rincebrain]

Could you do it on the 22.04 server? I think I know what I'll find now, but just to be sure.

I'm also not sure if this is an accurate reproduction right now, so we'll see, but I think that's what's happening, is that it's getting confused by the size not being different and marking them uncompressed in the send stream.

[phiser678]

This is on the Ubuntu 22.04 workstation:

# zdb -dbdbdbdbdbdb tank/test 3283
Dataset tank/test [ZPL], ID 70653, cr_txg 1368510, 108K, 8 objects, rootbp DVA[0]=<0:1d0f931000:1000> DVA[1]=<0:19ab827000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=1368511L/1368511P fill=8 cksum=1134bc29e2:2e659fb2d599:42530b971fca03:42e2645bbd9259b1

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      3283    1   128K     1K     4K     512     1K  100.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               176   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/demo/index.json
	uid     1000
	gid     1000
	atime	Sun Oct 23 17:29:30 2022
	mtime	Wed Oct 12 09:31:55 2022
	ctime	Wed Nov 16 14:45:53 2022
	crtime	Sun Oct 23 17:29:30 2022
	gen	167888
	mode	100644
	size	514
	parent	2
	links	1
	pflags	840800000004
Indirect blocks:
               0 L0 DVA[0]=<0:17bb34a000:1000> [L0 ZFS plain file] fletcher4 lz4 unencrypted LE contiguous unique single size=400L/400P birth=1368511L/1368511P fill=1 cksum=1d13ea9254:17779947edc2:9b5d04dddeb10:2be16f44ce6062c

		segment [0000000000000000, 0000000000000400) size    1K

[phiser678]

ok, I messed up I guess, so I redid the send/recv on the Ubuntu 20.04 machine:

ubuntu 20.04 # zfs send -c tank/test@2|zfs recv tank/error

# zdb -dbdbdbdbdbdb tank/test 3283
Dataset tank/test [ZPL], ID 179028, cr_txg 16463085, 108K, 8 objects, rootbp DVA[0]=<0:123f3044000:1000> DVA[1]=<0:1126002c000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=16463100L/16463100P fill=8 cksum=13caa072cb:375a42691e45:51ca8909f5cfcd:54e15e4067cac9cd

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      3283    1   128K     1K     4K     512     1K  100.00  ZFS plain file (K=inherit) (Z=inherit)
                                               176   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/demo/index.json
	uid     1000
	gid     1000
	atime	Sun Oct 23 17:29:30 2022
	mtime	Wed Oct 12 09:31:55 2022
	ctime	Wed Nov 16 14:45:53 2022
	crtime	Sun Oct 23 17:29:30 2022
	gen	167888
	mode	100644
	size	514
	parent	2
	links	1
	pflags	840800000004
Indirect blocks:
               0 L0 DVA[0]=<0:18843e9000:1000> [L0 ZFS plain file] fletcher4 lz4 unencrypted LE contiguous unique single size=400L/400P birth=16463086L/16463086P fill=1 cksum=1d13ea9254:17779947edc2:9b5d04dddeb10:2be16f44ce6062c

		segment [0000000000000000, 0000000000000400) size    1K

and then the tank/error:

# zdb -dbdbdbdbdbdb tank/error 3283
Dataset tank/error [ZPL], ID 179034, cr_txg 16463162, 108K, 8 objects, rootbp DVA[0]=<0:11260152000:1000> DVA[1]=<0:1884564000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=16463192L/16463192P fill=8 cksum=13bf557a89:3763c9fb6aa1:5251e05de350b1:5611bbe780bf47be

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      3283    1   128K     1K     4K     512     1K  100.00  ZFS plain file (K=inherit) (Z=inherit)
                                               176   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/demo/index.json
	uid     1000
	gid     1000
	atime	Sun Oct 23 17:29:30 2022
	mtime	Wed Oct 12 09:31:55 2022
	ctime	Wed Nov 16 14:45:53 2022
	crtime	Sun Oct 23 17:29:30 2022
	gen	167888
	mode	100644
	size	514
	parent	2
	links	1
	pflags	840800000004
Indirect blocks:
               0 L0 DVA[0]=<0:1cce4166000:1000> [L0 ZFS plain file] fletcher4 uncompressed unencrypted LE contiguous unique single size=400L/400P birth=16463163L/16463163P fill=1 cksum=1d13ea9254:17779947edc2:9b5d04dddeb10:2be16f44ce6062c

		segment [0000000000000000, 0000000000000400) size    1K

the lz4 is changed to uncompressed.

[phiser678]

If I redo the zfs send -c tank/test@2|zfs recv tank/error on a Ubuntu 22.04 server, lz4 remains!

[rincebrain]

Oh, I see, I misunderstood, I thought those send streams came from the 22.04 box.

This really is just #12762. Go nag Ubuntu to cherrypick the patch or run a version that's not from 2020.

[phiser678]

Well, yes. The tank/test stream comes from an Ubuntu 22.04, it is once received on the Ubuntu 20.04 that it gets corrupted when it has to resend to a secondary backup server (or it's own).
I already upgraded our backup servers to Ubuntu 22.04. As this is fixed in 22.04 I guess we can close the issue?

[rincebrain]

AFAIK 0.8.x isn't getting another release in...ever, so I don't think it's likely that the fix will be backported into something useful here. If Ubuntu wants to support it, they can backport it, I don't think the change is that disruptive, but the reason I didn't just open with that is I misunderstood that those streams were the ones sent from 22.04 to 20.04, not what 20.04 produced afterward of it. My apologies for the confusion and running around.

I would seriously suggest opening a Launchpad bug against 20.04 linking to this thread and #12762 and asking them to cherrypick it since it's easy to reproduce mangling your data with it.

[gdevenyi]

#10333