ZFS 2.0.4 build issue on Linux/sparc64: __bswapdi2 and __bswapsi2 missing

[koachan]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	N/A
Linux Kernel	5.9.15
Architecture	sparc64
ZFS Version	2.0.4
SPL Version	2.0.4

Describe the problem you're observing

ZFS codebase includes some byteswapping routines (mainly, in zstd code) that are implemented as __builtin_bswap32/__builtin_bswap64, which then gets compiled into __bswapsi2 and __bswapdi2 libgcc calls. Those are missing when compiling as a kernel module, so I'm getting errors like these:

ERROR: modpost: "__bswapdi2" [/zfs/module/zstd/zzstd.ko] undefined!
ERROR: modpost: "__bswapsi2" [/zfs/module/zstd/zzstd.ko] undefined!

I decided to file it here since it only happens when zstd is built as part of ZFS, but please let me know if I should file this to zstd people instead :)

Describe how to reproduce the problem

Build ZFS as usual on a sparc64 machine:

$ git clone https://github.com/openzfs/zfs && cd zfs
$ git checkout zfs-2.0.4
$ ./autogen.sh && ./configure && make

Include any warning/errors/backtraces from the system logs

  MODPOST /zfs/module/Module.symvers
ERROR: modpost: "__bswapdi2" [/zfs/module/zstd/zzstd.ko] undefined!
ERROR: modpost: "__bswapsi2" [/zfs/module/zstd/zzstd.ko] undefined!
make[4]: *** [scripts/Makefile.modpost:111: /zfs/module/Module.symvers] Error 1
make[4]: *** Deleting file '/zfs/module/Module.symvers'
make[3]: *** [Makefile:1714: modules] Error 2
make[3]: Leaving directory '/usr/src/linux-5.10.27-gentoo'
make[2]: *** [Makefile:48: modules-Linux] Error 2
make[2]: Leaving directory '/zfs/module'
make[1]: *** [Makefile:883: all-recursive] Error 1
make[1]: Leaving directory '/zfs'
make: *** [Makefile:744: all] Error 2

[koachan]

Oh, just adding that latest master commit (610cb4f) still suffers from the same issue.

[rincebrain]

So, I fixed the immediate issue of those missing functions using an implementation from elsewhere in the codebase, but while the resulting module loads and passes almost every test I put it through, it fails on the minor issue that attempting to import it on one of my x86_64 systems results in errors reading many of the files. (These files read fine if the pool is then imported again on the originating sparc64 system.)

So I'm suspicious that something in the ZFS glue code might be doing something endian-unsafe. (I'm reasonably confident the byteswap implementation is not the problem, having tried both the version from elsewhere in the codebase and the fallback version in zstd's own codebase, and gotten the same results.)

[ikozhukhov]

i have fixed build on sparc64 for DilOS by commit: https://bitbucket.org/dilos/dilos-illumos/commits/0978372eb920c486b41b0a8d7d80f022a6774da8
i have tested on sun4v T5220 and sun4u sunfire V490 - all works as well with zstd.
but DilOS is illumos based platform.

[rincebrain]

@ikozhukhov have you tried importing any pools with zstd-compressed files written from the sun4[uv] systems on an x86 system, and reading those files?

I tried overriding the definitions of clz[ds]i2 like you did just in case, and the resulting module has the same pathological behavior.

[ikozhukhov]

zstd not working between SPARC <> Intel, but LZ4 and GZIP are both fine:

root@sparc9:/var/tmp# uname -a
SunOS sparc9 5.11 2.0.2.85 sun4u sparc SUNW,Sun-Fire-V490

create a pool on a sparse file:
root@sparc9:/var/tmp# mkdir -p sparc
root@sparc9:/var/tmp# cd sparc
root@sparc9:/var/tmp/sparc# truncate -s 1g disk01.data
root@sparc9:/var/tmp/sparc# zpool create -o ashift=12 -o cachefile=none tpool $PWD/disk01.data
root@sparc9:/var/tmp/sparc# zpool list tpool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tpool   960M   372K   960M        -         -     0%     0%  1.00x    ONLINE  -

Create some datasets with different compression algorithms:
root@sparc9:/var/tmp/sparc# zfs create -o compression=zstd tpool/zstd-fs
root@sparc9:/var/tmp/sparc# zfs create -o compression=gzip tpool/gzip-fs
root@sparc9:/var/tmp/sparc# zfs create -o compression=lz4 tpool/lz4-fs

Copy a text file to every dataset:
root@sparc9:/var/tmp/sparc# cp ../os-upgrade.log /tpool/zstd-fs
root@sparc9:/var/tmp/sparc# cp ../os-upgrade.log /tpool/gzip-fs
root@sparc9:/var/tmp/sparc# cp ../os-upgrade.log /tpool/lz4-fs

Compute MD5 of the files:
root@sparc9:/var/tmp/sparc# md5sum /tpool/*-fs/os-upgrade.log
2070688ec9252c92a5aa5a92b109183b  /tpool/gzip-fs/os-upgrade.log
2070688ec9252c92a5aa5a92b109183b  /tpool/lz4-fs/os-upgrade.log
2070688ec9252c92a5aa5a92b109183b  /tpool/zstd-fs/os-upgrade.log

export the pool, pack the sparse file and send it to an intel host system:
root@sparc9:/var/tmp/sparc# zpool export tpool
root@sparc9:/var/tmp/sparc tar cvSf tpool.tar disk01.data
disk01.data
root@sparc9:/var/tmp/sparc# bzip2 -9 tpool.tar

/* sending tpool.tar.bz2 to intel */

root@lenovo:/var/tmp/sparc# uname -a
SunOS lenovo 5.11 2.0.2.85 i86pc i386 i86pc
root@lenovo:/var/tmp/sparc# ls -l
total 681
-rw-r--r-- 1 root root 1073741824 May 10 17:36 disk01.data

import a pool from the file:
root@lenovo:/var/tmp/sparc# zpool import -o cachefile=none -d $PWD tpool
root@lenovo:/var/tmp/sparc# zpool list tpool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tpool   960M  1.04M   959M        -         -     0%     0%  1.00x    ONLINE  -
root@lenovo:/var/tmp/sparc# zfs list -r tpool
NAME            USED  AVAIL     REFER  MOUNTPOINT
tpool           948K   831M      112K  /tpool
tpool/gzip-fs   124K   831M      124K  /tpool/gzip-fs
tpool/lz4-fs    132K   831M      132K  /tpool/lz4-fs
tpool/zstd-fs   124K   831M      124K  /tpool/zstd-fs

try to compute MD5:
root@lenovo:/var/tmp/sparc# md5sum /tpool/*-fs/os-upgrade.log
2070688ec9252c92a5aa5a92b109183b  /tpool/gzip-fs/os-upgrade.log
2070688ec9252c92a5aa5a92b109183b  /tpool/lz4-fs/os-upgrade.log
md5sum: /tpool/zstd-fs/os-upgrade.log: I/O error
root@lenovo:/var/tmp/sparc# zpool export tpool
root@lenovo:/var/tmp/sparc# cd ..
root@lenovo:/var/tmp# mkdir -p intel
root@lenovo:/var/tmp# cd intel

create a pool on the intel host:
root@lenovo:/var/tmp/intel# truncate -s 1g disk02.data
root@lenovo:/var/tmp/intel# zpool create -o cachefile=none -o ashift=12 ipool $POOL/disk02.data

create exactly the same datasets:
root@lenovo:/var/tmp/intel# zfs create -o compression=gzip ipool/gzip-fs
root@lenovo:/var/tmp/intel# zfs create -o compression=lz4 ipool/lz4-fs
root@lenovo:/var/tmp/intel# zfs create -o compression=zstd ipool/zstd-fs

and copy a text file to the datasets:
root@lenovo:/var/tmp/intel# cp ../os-upgrade.log /ipool/zstd-fs
root@lenovo:/var/tmp/intel# cp ../os-upgrade.log /ipool/gzip-fs
root@lenovo:/var/tmp/intel# cp ../os-upgrade.log /ipool/lz4-fs

compute MD5 of the files:
root@lenovo:/var/tmp/intel# md5sum /ipool/*-fs/os-upgrade.log
41301166a8f6f7e6a1293a0c15f453e9  /ipool/gzip-fs/os-upgrade.log
41301166a8f6f7e6a1293a0c15f453e9  /ipool/lz4-fs/os-upgrade.log
41301166a8f6f7e6a1293a0c15f453e9  /ipool/zstd-fs/os-upgrade.log

export the pool, pack the sparse file and send it to the sparc host system:
root@lenovo:/var/tmp/intel# zpool export ipool
root@lenovo:/var/tmp/intel# tar cvSf ipool.tar disk02.data 
disk02.data
root@lenovo:/var/tmp/intel# bzip2 -9 ipool.tar 
root@lenovo:/var/tmp/intel# ls -l
total 8757
-rw-r--r-- 1 root root 1073741824 May 10 17:39 disk02.data
-rw-r--r-- 1 root root    3964180 May 10 17:39 ipool.tar.bz2

/* sending ipool.tar.bz2 to sparc */

root@sparc9:/var/tmp# mkdir -p intel
root@sparc9:/var/tmp# cd intel
root@sparc9:/var/tmp/intel# ls -l
total 4700
-rw-r--r-- 1 root root 1073741824 May 10 17:39 disk02.data
root@sparc9:/var/tmp/intel# zpool import -o cachefile=none -d $PWD ipool   
root@sparc9:/var/tmp/intel# zpool list ipool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ipool   960M  6.03M   954M        -         -     0%     0%  1.00x    ONLINE  -

try to compute MD5 of the files:
root@sparc9:/var/tmp/intel# md5sum /ipool/*-fs/os-upgrade.log
41301166a8f6f7e6a1293a0c15f453e9  /ipool/gzip-fs/os-upgrade.log
41301166a8f6f7e6a1293a0c15f453e9  /ipool/lz4-fs/os-upgrade.log
md5sum: /ipool/zstd-fs/os-upgrade.log: I/O error
root@sparc9:/var/tmp/intel# zpool export ipool

[rincebrain]

Unfortunately, that's consistent with what I'm seeing.

So something is rotten here...

[rincebrain]

Courtesy of some help from Allan Jude...

It seems like the header information for the blocks are getting (endian?)mangled. Normally, zdb -Zddddddbbbbbb [dataset] [objid] would say
[blahblah] ZSTD:size=67387:version=10405:level=5:NORMAL

When reading a file generated on x86_64 on sparc64, you see:
[blahblah] ZSTD:size=14916:version=2663683:level=0:NORMAL

(It seems important to note here that 10405 = 0x28a5, 2663683 = 0x28a503, and the compression level+version are stored in a 32bit struct together as 24b+8b .)

(don't read anything into the size being different, these weren't the same object).

Still poking around to see what's wrong - the endianness macros don't appear to obviously be broken, at least on the sparc64 side...

[rincebrain]

Okay, I tentatively have a patch that fixes this. I'm going to do a bunch more testing and refinement (I haven't tried compiling it on anything except sparc64, for example...), but it does allow my sparc64 system to interoperably use zstd on a pool with an x86_64 machine.

(Bonus fact: this also affected ppc64 systems, and I am presuming all big endian systems - the non-interoperable zstd, that is, ppc64 doesn't have the missing symbols this issue was opened about.)

[yhaenggi]

Whats the current status for this? Im affected as well on sparc64.

[rincebrain]

I made #12022, if you have a burning need you could either run that or, if you absolutely don't plan to ever use zstd and don't want to run a large PR before it merges, just pick out the relevant changes for this bug from it.

I would suggest asking in the PR, because AFAIK, there are no technical complaints outstanding.