Oops on Linux/mips64el with git master and ZTS

[rincebrain]

(This could be related to #11679 , but I'm not sure, because the stack isn't entirely the same.)

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	bullseye
Linux Kernel	5.10.0-6-5kc-malta
Architecture	mips64el
ZFS Version	e723838

Describe the problem you're observing

While running ZTS through the "sanity" runfile, suddenly Oops in my logs.

Describe how to reproduce the problem

See above.

Include any warning/errors/backtraces from the system logs

 [55658.190472] CPU 0 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffffc04c620c, ra == ffffffffc04c7b94
[55658.193312] Oops[#1]:
[55658.193859] CPU: 0 PID: 143246 Comm: zfs Tainted: P           OE     5.10.0-6-5kc-malta #1 Debian 5.10.28-1
[55658.194696] $ 0   : 0000000000000000 0000000000000001 0a6c836569adb9b9 0000000000013ad0
[55658.195989] $ 4   : 0000000000000000 0000000000004000 0000000000004000 ffffffffffff00fe
[55658.196519] $ 8   : 0000000000000001 000000000000000c 0000000000000001 000000000000a88a
[55658.197131] $12   : 90000000d1fbb488 000000000000a400 00000000000001d5 000000000000a88a
[55658.197632] $16   : 0000000000000000 ffffffffc04c61e0 ffffffff81c60000 0000000000004000
[55658.198171] $20   : 000000000000000a 0000000000000020 0000000000000000 0000000000000000
[55658.198675] $24   : 0000000000000000 000000000224d32f
[55658.199298] $28   : 90000000d1fb8000 90000000d1fbb480 0000000000000000 ffffffffc04c7b94
[55658.199825] Hi    : 0000000000000000
[55658.200094] Lo    : 0040c3e2b0f6af24
[55658.201336] epc   : ffffffffc04c620c abd_verify+0x2c/0xd0 [zfs]
[55658.201962] ra    : ffffffffc04c7b94 abd_borrow_buf_copy+0x4c/0x100 [zfs]
[55658.202443] Status: 1000a4e3 KX SX UX KERNEL EXL IE
[55658.202952] Cause : 10800008 (ExcCode 02)
[55658.203215] BadVA : 0000000000000000
[55658.203493] PrId  : 00018900 (MIPS 5KE)
[55658.203823] Modules linked in: loop(E) md_mod(E) nfsd(E) nfsv3(E) nfs_acl(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) sunrpc(E) nfs_ssc(E) fscache(E) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) cirrus(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) sg(E) drm(E) drm_panel_orientation_quirks(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) crc32c_generic(E) sr_mod(E) cdrom(E) virtio_net(E) net_failover(E) failover(E) ata_generic(E) uhci_hcd(E) ehci_hcd(E) virtio_pci(E) virtio_ring(E) virtio(E) i2c_piix4(E) usbcore(E) ata_piix(E) i2c_core(E) usb_common(E)
[55658.208715] Process zfs (pid: 143246, threadinfo=00000000e9111f0f, task=00000000f50175f9, tls=000000fff2ac0720)
[55658.209387] Stack : ffffffff81c604f0 0000000000000000 ffffffffc04c61e0 ffffffff81c60000
[55658.209994]         0000000000004000 ffffffffc04c7b94 ffffffff00000000 0a6c836569adb9b9
[55658.210563]         ffffffffc06ec4e8 90000000d21a8000 0000000000000000 0000000000000010
[55658.211081]         0000000000004000 ffffffffc06630b8 9000000008d645a0 90000000a3c18930
[55658.211596]         0000000000000010 ffffffff81c60000 000000000000000a ffffffffc06ec4e8
[55658.212102]         0000000000000000 0000000000000000 0000000000000000 ffffffffc04d3830
[55658.212637]         90000000a3c18aa4 0000000000004000 90000000b9aa8c98 90000000b84e8068
[55658.213197]         90000000d1fbb594 0000000000042c00 90000000a6b7c000 0000000000016b80
[55658.213764]         ffffffffc06d5958 ffffffff8133e20c 00000000c67e84e0 0a6c836569adb9b9
[55658.214311]         90000000a6b7c000 90000000d1fbb628 ffffffff81c60000 9000000008d645a0
[55658.214879]         ...
[55658.215391] Call Trace:
[55658.215869] [<ffffffffc04c620c>] abd_verify+0x2c/0xd0 [zfs]
[55658.217153] [<ffffffffc04c7b94>] abd_borrow_buf_copy+0x4c/0x100 [zfs]
[55658.217914] [<ffffffffc06630b8>] zio_crypt_copy_dnode_bonus+0x60/0x1b0 [zfs]
[55658.218589] [<ffffffffc04d3830>] arc_buf_fill+0x448/0xdf8 [zfs]
[55658.219280] [<ffffffffc04d4228>] arc_untransform+0x48/0xe0 [zfs]
[55658.219976] [<ffffffffc04e77a4>] dbuf_read_verify_dnode_crypt+0x13c/0x1d8 [zfs]
[55658.220724] [<ffffffffc04ed5d4>] dbuf_read_impl.constprop.0+0x304/0x760 [zfs]
[55658.221458] [<ffffffffc04edc34>] dbuf_read+0x204/0x6a8 [zfs]
[55658.222097] [<ffffffffc04f98b8>] dmu_buf_hold+0x98/0xd8 [zfs]
[55658.222772] [<ffffffffc05ee830>] zap_lockdir+0x78/0x110 [zfs]
[55658.223446] [<ffffffffc05f087c>] zap_lookup+0x7c/0x160 [zfs]
[55658.224119] [<ffffffffc057da34>] sa_setup+0x26c/0x6e8 [zfs]
[55658.224777] [<ffffffffc064fda4>] zfsvfs_init.part.0+0x364/0x508 [zfs]
[55658.225463] [<ffffffffc0650250>] zfsvfs_create_impl+0x308/0x3a0 [zfs]
[55658.226163] [<ffffffffc065039c>] zfsvfs_create+0xb4/0x150 [zfs]
[55658.226790] [<ffffffffc0650514>] zfs_domount+0xdc/0x698 [zfs]
[55658.227469] [<ffffffffc0669c7c>] zpl_mount+0x1f4/0x270 [zfs]
[55658.228356] [<ffffffff813b9ae0>] legacy_get_tree+0x38/0x80
[55658.228707] [<ffffffff8136fc98>] vfs_get_tree+0x38/0x130
[55658.229062] [<ffffffff8139d740>] path_mount+0x7b0/0xb38
[55658.229447] [<ffffffff8139e1ac>] sys_mount+0x254/0x2e0
[55658.229813] [<ffffffff810bb3f0>] syscall_common+0x34/0x58
[55658.230497]
[55658.230730] Code: 03e00825  0060f809  67ac0028 <8c820000> 30430001  14600017  00808025  7c420180  1040001b
[55658.231505]
[55658.231974] ---[ end trace 5a87f2805d77e470 ]---

$ ps aux | grep zfs
rich       13408  0.0  0.0   2584  1812 pts/0    S+   09:39   0:01 /bin/sh scripts/zfs-tests.sh -r sanity
rich       13557  4.0  0.6  92112 12460 pts/0    S+   09:40   6:11 python3 /home/rich/zfs_vanilla/tests/test-runner/bin/test-runner.py -c /home/rich/zfs_vanilla/tests/runfiles/sanity.run -T functional -i /home/rich/zfs_vanilla/tests/zfs-tests -I 1
root      143066  0.0  0.2  10116  5024 pts/0    S+   12:07   0:00 sudo -E -u root /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_from_encrypted.ksh
root      143068  0.2  0.2   6456  4424 pts/0    S+   12:07   0:01 /bin/ksh -p /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_from_encrypted.ksh
root      143384  0.1  0.2   8884  4432 pts/0    S+   12:08   0:00 /home/rich/zfs_vanilla/cmd/zfs/.libs/zfs destroy -r testpool/testfs2
$

[...]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/setup (run as root) [00:24] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_001_pos (run as root) [01:53] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_002_pos (run as root) [00:37] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_003_pos (run as root) [00:25] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_004_neg (run as root) [01:35] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_005_neg (run as root) [00:28] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_006_pos (run as root) [00:36] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_007_neg (run as root) [00:25] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_008_pos (run as root) [02:00] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_009_neg (run as root) [01:39] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_010_pos (run as root) [01:30] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_011_pos (run as root) [00:26] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_012_pos (run as root) [02:11] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_013_pos (run as root) [00:46] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_014_pos (run as root) [01:12] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_015_pos (run as root) [00:36] [PASS]
Test: /home/rich/zfs_vanilla/tests/zfs-tests/tests/functional/cli_root/zfs_receive/zfs_receive_016_pos (run as root) [00:27] [PASS]

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[dhedberg]

We just hit something similar (openSUSE 15.4 on x86_64, zfs 2.1.12):

kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
kernel: #PF: supervisor read access in kernel mode
kernel: #PF: error_code(0x0000) - not-present page
kernel: PGD 2c0c9e067 P4D 0
kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
kernel: CPU: 21 PID: 155257 Comm: lxd Tainted: P        W  OEL    N 5.14.21-150400.24.74-default #1 SLE15-SP4 07091c7be5d59eef57a6ebdfed686f664f470b8f
kernel: Hardware name: ASUSTeK COMPUTER INC. RS500A-E12-RS12U/K14PA-U24 Series, BIOS 0901 05/04/2023
kernel: RIP: 0010:abd_borrow_buf_copy+0x1e/0x90 [zfs]
kernel: Code: 89 f7 e9 25 ad 10 00 0f 1f 44 00 00 0f 1f 44 00 00 55 53 48 89 fb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <f6> 07 01 74 22 48 8b 6f 48 48 8b 54 24 10 65 48 33 14 25 28 00 00
kernel: RSP: 0018:ff58234fcd6036a0 EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ff44b8b0e9d8c000
kernel: RDX: 0000000000004000 RSI: 0000000000004000 RDI: 0000000000000000
kernel: RBP: 0000000000000000 R08: 00000000000002d9 R09: 000000000055fcce
kernel: R10: ff58234fcd603718 R11: 000000000053a758 R12: 0000000000000010
kernel: R13: 000000000000000a R14: ffffffffc29e7590 R15: ffffffffc29e75b0
kernel: FS:  00007fd1d77fe700(0000) GS:ff44b96389d40000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 0000000186e2a005 CR4: 0000000000771ee0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  <TASK>
kernel:  zio_crypt_copy_dnode_bonus+0x2e/0x130 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  arc_buf_fill+0x3fb/0xb40 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? spl_kmem_cache_alloc+0x73/0x750 [spl df0d1403fe30b557d05a20f0126feb831ea7125d]
kernel:  arc_untransform+0x1d/0x80 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  dbuf_read_verify_dnode_crypt+0xf7/0x160 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  dbuf_read_impl.constprop.33+0x2f5/0x6a0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zio_create+0x41c/0x4b0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zio_null+0x2b/0x40 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  dbuf_read+0x1b6/0x5a0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  dmu_buf_hold+0x5a/0x90 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? __mutex_lock.isra.13+0x468/0x4a0
kernel:  zap_lockdir+0x4e/0xc0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zap_lookup_norm+0x5d/0xd0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zap_lookup+0x12/0x20 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  sa_setup+0x216/0x5d0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? dbuf_cache_multilist_index_func+0x29/0x40 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? multilist_insert+0x83/0xc0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? dbuf_rele_and_unlock+0x599/0x670 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? mze_find+0x8f/0xc0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zap_lookup_impl+0x89/0x280 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zap_lockdir+0x90/0xc0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zap_lookup_norm+0x9a/0xd0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zfs_rangelock_cb+0x90/0x90 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? zfs_rangelock_cb+0x90/0x90 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zfsvfs_init+0x39c/0x560 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zfsvfs_create_impl+0x214/0x2e0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zfsvfs_create+0x93/0xd0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  zfs_domount+0x8d/0x510 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  ? register_shrinker_prepared+0x15/0x50
kernel:  zpl_mount+0x172/0x1c0 [zfs a647e9021e8813a1e3977354ef6278e499acd8a9]
kernel:  legacy_get_tree+0x27/0x50
kernel:  vfs_get_tree+0x25/0xd0
kernel:  path_mount+0x6e4/0x9b0
kernel:  do_mount+0x79/0x90
kernel:  __x64_sys_mount+0x86/0xe0
kernel:  do_syscall_64+0x5b/0x80
kernel:  ? syscall_exit_to_user_mode+0x18/0x40
kernel:  ? do_syscall_64+0x67/0x80
kernel:  ? do_syscall_64+0x67/0x80
kernel:  ? syscall_exit_to_user_mode+0x18/0x40
kernel:  ? do_syscall_64+0x67/0x80
kernel:  ? do_syscall_64+0x67/0x80
kernel:  ? do_syscall_64+0x67/0x80
kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
kernel: RIP: 0033:0x55befd480cca
kernel: Code: e8 9b a8 fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 4c 8b 54 24 28 4c 8b 44 24 30 4c 8b 4c 24 38 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 40 ff ff ff ff 48 c7 44 24 48
kernel: RSP: 002b:000000c000a40bc0 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
kernel: RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 000055befd480cca
kernel: RDX: 000000c0008b3608 RSI: 000000c001ee23c0 RDI: 000000c0014aa2c0
kernel: RBP: 000000c000a40c60 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 000000c0008b3600
kernel: R13: 0000000000000000 R14: 000000c0010b0340 R15: ffffffffffffffff
kernel:  </TASK>
kernel: Modules linked in: ext4 crc16 mbcache jbd2 mptcp_diag(N) xsk_diag vsock_diag tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag nf_conntrack_netlink veth vhost_net tun macvtap tap xt_MASQUERADE bridge xfrm_user xfrm_algo >
kernel:  bpfilter ipmi_ssif irdma ice ib_uverbs ib_core intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm irqbypass wmi_bmof pcspkr ast drm_vram_helper drm_ttm_helper ttm drm_kms_helper cec rc_core igb cdc_ether acpi_ipmi nls_iso8859_>
kernel: Supported: No, Proprietary and Unsupported modules are loaded
kernel: CR2: 0000000000000000
kernel: ---[ end trace 8fee561c11b22c55 ]---
kernel: RIP: 0010:abd_borrow_buf_copy+0x1e/0x90 [zfs]
kernel: Code: 89 f7 e9 25 ad 10 00 0f 1f 44 00 00 0f 1f 44 00 00 55 53 48 89 fb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <f6> 07 01 74 22 48 8b 6f 48 48 8b 54 24 10 65 48 33 14 25 28 00 00
kernel: RSP: 0018:ff58234fcd6036a0 EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ff44b8b0e9d8c000
kernel: RDX: 0000000000004000 RSI: 0000000000004000 RDI: 0000000000000000
kernel: RBP: 0000000000000000 R08: 00000000000002d9 R09: 000000000055fcce
kernel: R10: ff58234fcd603718 R11: 000000000053a758 R12: 0000000000000010
kernel: R13: 000000000000000a R14: ffffffffc29e7590 R15: ffffffffc29e75b0
kernel: FS:  00007fd1d77fe700(0000) GS:ff44b96389d40000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 0000000186e2a005 CR4: 0000000000771ee0
kernel: PKRU: 55555554