panic on FreeBSD 13-STABLE - VERIFY3(0 == vdev_obsolete_sm_object(vd, &obsolete_sm_object)) failed (0 == 97)

[rincebrain]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	13-STABLE
FreeBSD Kernel	freebsd/freebsd-src@92f49c769b4e
Architecture	x86_64
ZFS Version	#11992

(I wouldn't file this if there were any kernel changes in this PR, but since it's all lzc_send...)

Describe the problem you're observing

While running ZTS, "fun" ensued with no warning I saw. No errors or panics found after a scrub on the zroot pool after rebooting.

Describe how to reproduce the problem

Unknown? Maybe run ZTS? A ZTS run immediately prior tried and succeeded in zfs unmounting /tmp while things were actively doing IO to it, no idea if related.

Include any warning/errors/backtraces from the system logs

(As you can tell, I have a vmcore to poke around in, if it would be helpful.)
Last few lines in the ZTS log from the run where it died:

Test: /home/rich/zfs_pipeagain/tests/zfs-tests/tests/functional/cache/cache_011_pos (run as root) [00:13] [PASS]
08:22:41.94 ASSERTION: Remove cache device from pool with spare device should succeed
08:22:42.42 SUCCESS: zpool create testpool /var/tmp/testdir/disk.cache/a /var/tmp/testdir/disk.cache/b /var/tmp/testdir/disk.cache/c cache md21 spare md22
08:22:42.51 SUCCESS: zpool remove testpool md21
08:22:44.66 SUCCESS: display_status testpool
08:22:45.76 SUCCESS: zpool destroy -f testpool
08:22:46.31 SUCCESS: zpool create testpool mirror /var/tmp/testdir/disk.cache/a /var/tmp/testdir/disk.cache/b /var/tmp/testdir/disk.cache/c cache md21 spare md22
08:22:46.49 SUCCESS: zpool remove testpool md21
08:22:48.71 SUCCESS: display_status testpool
08:22:49.06 SUCCESS: zpool destroy -f testpool
08:22:49.47 SUCCESS: zpool create testpool raidz /var/tmp/testdir/disk.cache/a /var/tmp/testdir/disk.cache/b /var/tmp/testdir/disk.cache/c cache md21 spare md22
08:22:49.57 SUCCESS: zpool remove testpool md21
08:22:51.75 SUCCESS: display_status testpool
08:22:52.34 SUCCESS: zpool destroy -f testpool
08:22:52.80 SUCCESS: zpool create testpool raidz2 /var/tmp/testdir/disk.cache/a /var/tmp/testdir/disk.cache/b /var/tmp/testdir/disk.cache/c cache md21 spare md22
08:22:53.00 SUCCESS: zpool remove testpool md21
08:22:55.45 SUCCESS: display_status testpool
08:22:55.71 SUCCESS: zpool destroy -f testpool
08:22:55.71 Remove cache device from pool with spare device should succeed
08:22:55.71 NOTE: Performing local cleanup via log_onexit (cleanup)

kgdb:

Unread portion of the kernel message buffer:
panic: VERIFY3(0 == vdev_obsolete_sm_object(vd, &obsolete_sm_object)) failed (0 == 97)

cpuid = 1
time = 1622982251
KDB: stack backtrace:
#0 0xffffffff80c5ad85 at kdb_backtrace+0x65
#1 0xffffffff80c0cf31 at vpanic+0x181
#2 0xffffffff8213ea5a at spl_panic+0x3a
#3 0xffffffff8224a4d4 at spa_sync+0x904
#4 0xffffffff8226361c at txg_sync_thread+0x3ac
#5 0xffffffff80bcb0ce at fork_exit+0x7e
#6 0xffffffff81069ece at fork_trampoline+0xe
Uptime: 7h44m58s
Dumping 920 out of 8150 MB:..2%..11%..21%..32%..42%..51%..61%..72%..82%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c0cb26 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff80c0cfa0 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff8213ea5a in spl_panic (file=<optimized out>, func=<optimized out>, line=<unavailable>, fmt=<unavailable>) at os/freebsd/spl/spl_misc.c:107
#5  0xffffffff8224a4d4 in vdev_indirect_state_sync_verify (vd=0xfffffe00d0cfc000) at zfs/spa.c:8903
#6  spa_sync_condense_indirect (spa=0xfffffe00d070d000, tx=0xfffff801e9306200) at zfs/spa.c:8999
#7  spa_sync (spa=<optimized out>, spa@entry=0xfffffe00d070d000, txg=<optimized out>, txg@entry=4642) at zfs/spa.c:9241
#8  0xffffffff8226361c in txg_sync_thread (arg=arg@entry=0xfffff801aa5bd000) at zfs/txg.c:595
#9  0xffffffff80bcb0ce in fork_exit (callout=0xffffffff82263270 <txg_sync_thread>, arg=0xfffff801aa5bd000, frame=0xfffffe00cdd6fc00) at /usr/src/sys/kern/kern_fork.c:1083
#10 <signal handler called>
(kgdb) up
#2  0xffffffff80c0cb26 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
486                     doadump(TRUE);
(kgdb) up
#3  0xffffffff80c0cfa0 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:919
919             kern_reboot(bootopt);
(kgdb) up
#4  0xffffffff8213ea5a in spl_panic (file=<optimized out>, func=<optimized out>, line=<unavailable>, fmt=<unavailable>) at os/freebsd/spl/spl_misc.c:107
107             vpanic(fmt, ap);
(kgdb) up
#5  0xffffffff8224a4d4 in vdev_indirect_state_sync_verify (vd=0xfffffe00d0cfc000) at zfs/spa.c:8903
8903            ASSERT0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
(kgdb) p *vd
$4 = {vdev_id = 0, vdev_guid = 16273198774165295950, vdev_guid_sum = 16273198774165295950, vdev_orig_guid = 0, vdev_asize = 129499136, vdev_min_asize = 129499136, vdev_max_asize = 129499136, vdev_ashift = 12, vdev_logical_ashift = 9,
  vdev_physical_ashift = 9, vdev_state = 7, vdev_prevstate = 0, vdev_ops = 0xffffffff82441628, vdev_spa = 0xfffffe00d070d000, vdev_tsd = 0xfffff80004e4ab30, vdev_top = 0xfffffe00d0cfc000, vdev_parent = 0xfffffe00d11eb000,
  vdev_child = 0x0, vdev_children = 0, vdev_stat = {vs_timestamp = 4642410712080, vs_state = 0, vs_aux = 0, vs_alloc = 430080, vs_space = 117440512, vs_dspace = 117440512, vs_rsize = 0, vs_esize = 0, vs_ops = {0, 10, 145, 0, 0, 0},
    vs_bytes = {0, 495616, 2588672, 0, 0, 0}, vs_read_errors = 0, vs_write_errors = 0, vs_checksum_errors = 10, vs_initialize_errors = 0, vs_self_healed = 0, vs_scan_removing = 0, vs_scan_processed = 0, vs_fragmentation = 0,
    vs_initialize_bytes_done = 0, vs_initialize_bytes_est = 0, vs_initialize_state = 0, vs_initialize_action_time = 0, vs_checkpoint_space = 0, vs_resilver_deferred = 0, vs_slow_ios = 0, vs_trim_errors = 0, vs_trim_notsup = 0,
    vs_trim_bytes_done = 0, vs_trim_bytes_est = 0, vs_trim_state = 0, vs_trim_action_time = 0, vs_rebuild_processed = 0, vs_configured_ashift = 0, vs_logical_ashift = 0, vs_physical_ashift = 0}, vdev_stat_ex = {vsx_active_queue = {0,
      0, 0, 0, 0, 0, 0, 0, 0}, vsx_pend_queue = {0, 0, 0, 0, 0, 0, 0, 0, 0}, vsx_queue_histo = {{0 <repeats 12 times>, 5, 5, 0 <repeats 23 times>}, {0 <repeats 12 times>, 10, 20, 1, 0 <repeats 22 times>}, {0 <repeats 37 times>}, {
        0 <repeats 12 times>, 5, 24, 13, 19, 24, 18, 11, 0 <repeats 18 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}}, vsx_total_histo = {{
        0 <repeats 37 times>}, {0 <repeats 15 times>, 3, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0}, {0 <repeats 14 times>, 1, 28, 67, 32, 17, 0 <repeats 18 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {
        0 <repeats 37 times>}, {0 <repeats 37 times>}}, vsx_disk_histo = {{0 <repeats 37 times>}, {0 <repeats 15 times>, 4, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0}, {0 <repeats 12 times>, 1, 5, 20, 61, 54, 4,
        0 <repeats 19 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}, {0 <repeats 37 times>}}, vsx_ind_histo = {{0 <repeats 12 times>, 3, 3, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0}, {0 <repeats 12 times>, 12,
        7, 0, 0, 8, 4, 0, 0, 0, 0, 0, 0, 0}, {0 <repeats 25 times>}, {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 63, 1, 0 <repeats 11 times>}, {0 <repeats 25 times>}, {0 <repeats 25 times>}, {0 <repeats 25 times>}, {0 <repeats 25 times>}, {
        0 <repeats 25 times>}}, vsx_agg_histo = {{0 <repeats 25 times>}, {0 <repeats 25 times>}, {0 <repeats 25 times>}, {0 <repeats 13 times>, 27, 10, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0}, {0 <repeats 25 times>}, {0 <repeats 25 times>}, {
        0 <repeats 25 times>}, {0 <repeats 25 times>}, {0 <repeats 25 times>}}}, vdev_expanding = 0, vdev_reopening = 0, vdev_nonrot = 1, vdev_load_error = 0, vdev_open_error = 0, vdev_validate_error = 0, vdev_open_thread = 0x0,
  vdev_validate_thread = 0x0, vdev_crtxg = 4, vdev_ms_array = 64, vdev_ms_shift = 24, vdev_ms_count = 7, vdev_mg = 0xfffff8000d6a2000, vdev_log_mg = 0xfffff801aa1cec00, vdev_ms = 0xfffff8016d3bd180, vdev_pending_fastwrite = 0,
  vdev_ms_list = {tl_lock = {lock_object = {lo_name = 0xffffffff8242dcfb "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 1784, tl_spa = 0xfffffe00d070d000, tl_head = {0x0, 0x0, 0x0, 0x0}},
  vdev_dtl_list = {tl_lock = {lock_object = {lo_name = 0xffffffff8242dcfb "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 12496, tl_spa = 0xfffffe00d070d000, tl_head = {0x0, 0x0, 0x0,
      0x0}}, vdev_txg_node = {tn_next = {0x0, 0x0, 0x0, 0x0}, tn_member = "\000\000\000"}, vdev_remove_wanted = 0, vdev_probe_wanted = 0, vdev_config_dirty_node = {list_next = 0x0, list_prev = 0x0}, vdev_state_dirty_node = {
    list_next = 0x0, list_prev = 0x0}, vdev_deflate_ratio = 512, vdev_islog = 0, vdev_removing = 0, vdev_ishole = 0, vdev_top_zap = 130, vdev_alloc_bias = VDEV_BIAS_NONE, vdev_checkpoint_sm = 0x0, vdev_initialize_exit_wanted = 0,
  vdev_initialize_state = VDEV_INITIALIZE_NONE, vdev_initialize_node = {list_next = 0x0, list_prev = 0x0}, vdev_initialize_thread = 0x0, vdev_initialize_lock = {lock_object = {lo_name = 0xffffffff82405e1a "vd->vdev_initialize_lock",
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_initialize_cv = {cv_description = 0xffffffff82426e5c "vd->vdev_initialize_cv", cv_waiters = 0}, vdev_initialize_offset = {0, 0, 0, 0},
  vdev_initialize_last_offset = 0, vdev_initialize_tree = 0x0, vdev_initialize_bytes_est = 0, vdev_initialize_bytes_done = 0, vdev_initialize_action_time = 0, vdev_trim_exit_wanted = 0, vdev_autotrim_exit_wanted = 0,
  vdev_trim_state = VDEV_TRIM_NONE, vdev_trim_node = {list_next = 0x0, list_prev = 0x0}, vdev_autotrim_lock = {lock_object = {lo_name = 0xffffffff823dd9f1 "vd->vdev_autotrim_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0},
    sx_lock = 1}, vdev_autotrim_cv = {cv_description = 0xffffffff823da9a1 "vd->vdev_autotrim_cv", cv_waiters = 0}, vdev_autotrim_thread = 0x0, vdev_trim_lock = {lock_object = {lo_name = 0xffffffff8241cb46 "vd->vdev_trim_lock",
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_trim_cv = {cv_description = 0xffffffff823ead82 "vd->vdev_trim_cv", cv_waiters = 0}, vdev_trim_thread = 0x0, vdev_trim_offset = {0, 0, 0, 0},
  vdev_trim_last_offset = 0, vdev_trim_bytes_est = 0, vdev_trim_bytes_done = 0, vdev_trim_rate = 0, vdev_trim_partial = 0, vdev_trim_secure = 0, vdev_trim_action_time = 0, vdev_rebuilding = 0, vdev_rebuild_exit_wanted = 0,
  vdev_rebuild_cancel_wanted = 0, vdev_rebuild_reset_wanted = 0, vdev_rebuild_lock = {lock_object = {lo_name = 0xffffffff823f5b2d "vd->vdev_rebuild_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1},
  vdev_rebuild_cv = {cv_description = 0xffffffff82431415 "vd->vdev_rebuild_cv", cv_waiters = 0}, vdev_rebuild_thread = 0x0, vdev_rebuild_config = {vr_top_vdev = 0x0, vr_scan_msp = 0x0, vr_scan_tree = 0x0, vr_io_lock = {lock_object = {
        lo_name = 0x0, lo_flags = 0, lo_data = 0, lo_witness = 0x0}, sx_lock = 0}, vr_io_cv = {cv_description = 0x0, cv_waiters = 0}, vr_scan_offset = {0, 0, 0, 0}, vr_prev_scan_time_ms = 0, vr_bytes_inflight_max = 0,
    vr_bytes_inflight = 0, vr_pass_start_time = 0, vr_pass_bytes_scanned = 0, vr_pass_bytes_issued = 0, vr_rebuild_phys = {vrp_rebuild_state = 0, vrp_last_offset = 0, vrp_min_txg = 0, vrp_max_txg = 0, vrp_start_time = 0,
      vrp_end_time = 0, vrp_scan_time_ms = 0, vrp_bytes_scanned = 0, vrp_bytes_issued = 0, vrp_bytes_rebuilt = 0, vrp_bytes_est = 0, vrp_errors = 0}}, vdev_initialize_io_lock = {lock_object = {
      lo_name = 0xffffffff823d731e "vd->vdev_initialize_io_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_initialize_io_cv = {cv_description = 0xffffffff823ff55c "vd->vdev_initialize_io_cv",
    cv_waiters = 0}, vdev_initialize_inflight = 0, vdev_trim_io_lock = {lock_object = {lo_name = 0xffffffff823f9295 "vd->vdev_trim_io_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_trim_io_cv = {
    cv_description = 0xffffffff823fbf14 <.L.str.45+8> "vd->vdev_trim_io_cv", cv_waiters = 0}, vdev_trim_inflight = {0, 0, 0}, vdev_indirect_config = {vic_mapping_object = 0, vic_births_object = 0,
    vic_prev_indirect_vdev = 18446744073709551615}, vdev_indirect_rwlock = {lock_object = {lo_name = 0xffffffff8241cb2c "vd->vdev_indirect_rwlock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, sx_lock = 1},
  vdev_indirect_mapping = 0x0, vdev_indirect_births = 0x0, vdev_obsolete_lock = {lock_object = {lo_name = 0xffffffff8243aa4c "vd->vdev_obsolete_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1},
  vdev_obsolete_segments = 0xfffff801aa359400, vdev_obsolete_sm = 0x0, vdev_scan_io_queue_lock = {lock_object = {lo_name = 0xffffffff82405dfd "vd->vdev_scan_io_queue_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0},
    sx_lock = 1}, vdev_scan_io_queue = 0x0, vdev_dtl = {0xfffff80176664c00, 0xfffff80176bcc400, 0xfffff8013ffbac00, 0xfffff80192e1c400}, vdev_dtl_sm = 0x0, vdev_dtl_node = {tn_next = {0x0, 0x0, 0x0, 0x0}, tn_member = "\000\000\000"},
  vdev_dtl_object = 0, vdev_psize = 134217728, vdev_wholedisk = 18446744073709551615, vdev_offline = 0, vdev_faulted = 0, vdev_degraded = 0, vdev_removed = 0, vdev_resilver_txg = 0, vdev_rebuild_txg = 0,
  vdev_path = 0xfffff8016d257de0 "/tmp/tmp-g68hgbzm.zpool", vdev_devid = 0x0, vdev_physpath = 0x0, vdev_enc_sysfs_path = 0x0, vdev_fru = 0x0, vdev_not_present = 0, vdev_unspare = 0, vdev_nowritecache = 0, vdev_has_trim = 1,
  vdev_has_securetrim = 0, vdev_checkremove = 0, vdev_forcefault = 0, vdev_splitting = 0, vdev_delayed_close = 0, vdev_tmpoffline = 0, vdev_detached = 0, vdev_cant_read = 0, vdev_cant_write = 0, vdev_isspare = 0, vdev_isl2cache = 0,
  vdev_copy_uberblocks = 0, vdev_resilver_deferred = 0, vdev_queue = {vq_vdev = 0xfffffe00d0cfc000, vq_class = {{vqc_active = 0, vqc_queued_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848d0 <vdev_queue_timestamp_compare>,
          avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848d0 <vdev_queue_timestamp_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}},
      {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848d0 <vdev_queue_timestamp_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}, {vqc_active = 0, vqc_queued_tree = {avl_root = 0x0,
          avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 672, avl_numnodes = 0, avl_size = 1232}}}, vq_active_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>,
      avl_offset = 672, avl_numnodes = 0, avl_size = 1232}, vq_read_offset_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 696, avl_numnodes = 0, avl_size = 1232},
    vq_write_offset_tree = {avl_root = 0x0, avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 696, avl_numnodes = 0, avl_size = 1232}, vq_trim_offset_tree = {avl_root = 0x0,
      avl_compar = 0xffffffff822848a0 <vdev_queue_offset_compare>, avl_offset = 696, avl_numnodes = 0, avl_size = 1232}, vq_last_offset = 37855232, vq_last_prio = ZIO_PRIORITY_SYNC_READ, vq_ia_active = 0, vq_nia_credit = 0,
    vq_io_complete_ts = 27898133137694, vq_io_delta_ts = 192482, vq_io_search = {io_bookmark = {zb_objset = 0, zb_object = 0, zb_level = 0, zb_blkid = 0}, io_prop = {zp_checksum = ZIO_CHECKSUM_INHERIT,
        zp_compress = ZIO_COMPRESS_INHERIT, zp_complevel = 0 '\000', zp_type = DMU_OT_NONE, zp_level = 0 '\000', zp_copies = 0 '\000', zp_dedup = 0, zp_dedup_verify = 0, zp_nopwrite = 0, zp_encrypt = 0, zp_byteorder = 0,
        zp_salt = "\000\000\000\000\000\000\000", zp_iv = '\000' <repeats 11 times>, zp_mac = '\000' <repeats 15 times>, zp_zpl_smallblk = 0}, io_type = ZIO_TYPE_NULL, io_child_type = ZIO_CHILD_VDEV, io_trim_flags = 0, io_cmd = 0,
      io_priority = ZIO_PRIORITY_SYNC_READ, io_reexecute = 0 '\000', io_state = "\000", io_txg = 0, io_spa = 0x0, io_bp = 0x0, io_bp_override = 0x0, io_bp_copy = {blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {dva_word = {0,
              0}}}, blk_prop = 0, blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 0, blk_fill = 0, blk_cksum = {zc_word = {0, 0, 0, 0}}}, io_parent_list = {list_size = 0, list_offset = 0, list_head = {list_next = 0x0,
          list_prev = 0x0}}, io_child_list = {list_size = 0, list_offset = 0, list_head = {list_next = 0x0, list_prev = 0x0}}, io_logical = 0x0, io_transform_stack = 0x0, io_ready = 0x0, io_children_ready = 0x0, io_physdone = 0x0,
      io_done = 0x0, io_private = 0x0, io_prev_space_delta = 0, io_bp_orig = {blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {dva_word = {0, 0}}}, blk_prop = 0, blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 0, blk_fill = 0,
        blk_cksum = {zc_word = {0, 0, 0, 0}}}, io_lsize = 0, io_abd = 0x0, io_orig_abd = 0x0, io_size = 0, io_orig_size = 0, io_vd = 0x0, io_vsd = 0x0, io_vsd_ops = 0x0, io_metaslab_class = 0x0, io_offset = 21131263, io_timestamp = 0,
      io_queued_timestamp = 0, io_target_timestamp = 0, io_delta = 0, io_delay = 0, io_queue_node = {avl_child = {0x0, 0x0}, avl_pcb = 0}, io_offset_node = {avl_child = {0x0, 0x0}, avl_pcb = 0}, io_alloc_node = {avl_child = {0x0, 0x0},
        avl_pcb = 0}, io_alloc_list = {zal_list = {list_size = 0, list_offset = 0, list_head = {list_next = 0x0, list_prev = 0x0}}, zal_size = 0}, io_flags = 0, io_stage = 0, io_pipeline = 0, io_orig_flags = 0, io_orig_stage = 0,
      io_orig_pipeline = 0, io_pipeline_trace = 0, io_error = 0, io_child_error = {0, 0, 0, 0}, io_children = {{0, 0}, {0, 0}, {0, 0}, {0, 0}}, io_child_count = 0, io_phys_children = 0, io_parent_count = 0, io_stall = 0x0,
      io_gang_leader = 0x0, io_gang_tree = 0x0, io_executor = 0x0, io_waiter = 0x0, io_bio = 0x0, io_lock = {lock_object = {lo_name = 0x0, lo_flags = 0, lo_data = 0, lo_witness = 0x0}, sx_lock = 0}, io_cv = {cv_description = 0x0,
        cv_waiters = 0}, io_allocator = 0, io_cksum_report = 0x0, io_ena = 0, io_tqent = {tqent_task = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000', ta_func = 0x0, ta_context = 0x0},
        tqent_timeout_task = {q = 0x0, t = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000', ta_func = 0x0, ta_context = 0x0}, c = {c_links = {le = {le_next = 0x0, le_prev = 0x0}, sle = {
                sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, c_precision = 0, c_arg = 0x0, c_func = 0x0, c_lock = 0x0, c_flags = 0, c_iflags = 0, c_cpu = 0}, f = 0}, tqent_func = 0x0, tqent_arg = 0x0,
        tqent_id = 0, tqent_hash = {cle_next = 0x0, cle_prev = 0x0}, tqent_type = 0 '\000', tqent_registered = 0 '\000', tqent_cancelled = 0 '\000', tqent_rc = 0}}, vq_lock = {lock_object = {lo_name = 0xffffffff8241cf45 "vq->vq_lock",
        lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}}, vdev_cache = {vc_offset_tree = {avl_root = 0x0, avl_compar = 0xffffffff82272200 <vdev_cache_offset_compare>, avl_offset = 24, avl_numnodes = 0,
      avl_size = 88}, vc_lastused_tree = {avl_root = 0x0, avl_compar = 0xffffffff82272220 <vdev_cache_lastused_compare>, avl_offset = 48, avl_numnodes = 0, avl_size = 88}, vc_lock = {lock_object = {
        lo_name = 0xffffffff824376b2 "vc->vc_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}}, vdev_aux = 0x0, vdev_probe_zio = 0x0, vdev_label_aux = VDEV_AUX_NONE, vdev_leaf_zap = 129, vdev_mmp_pending = 0,
  vdev_mmp_kstat_id = 0, vdev_expansion_time = 0, vdev_leaf_node = {list_next = 0xfffffe00d070e588, list_prev = 0xfffffe00d070e588}, vdev_dtl_lock = {lock_object = {lo_name = 0xffffffff82419581 "vd->vdev_dtl_lock",
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_stat_lock = {lock_object = {lo_name = 0xffffffff82402aaf "vd->vdev_stat_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1},
  vdev_probe_lock = {lock_object = {lo_name = 0xffffffff823da98c "vd->vdev_probe_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, vdev_delay_rl = {start = 0, count = 0,
    burst = 0xffffffff824bab3c <zfs_slow_io_events_per_second>, interval = 1, lock = {lock_object = {lo_name = 0xffffffff8242e58f "rl->lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}}, vdev_deadman_rl = {
    start = 0, count = 0, burst = 0xffffffff824bab3c <zfs_slow_io_events_per_second>, interval = 1, lock = {lock_object = {lo_name = 0xffffffff8242e58f "rl->lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}},
  vdev_checksum_rl = {start = 27898132478393, count = 2, burst = 0xffffffff824bab40 <zfs_checksum_events_per_second>, interval = 1, lock = {lock_object = {lo_name = 0xffffffff8242e58f "rl->lock", lo_flags = 577830912, lo_data = 0,
        lo_witness = 0x0}, sx_lock = 1}}}
(kgdb)

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[behlendorf]

This appears to have been caused by zap_lookup() in vdev_obsolete_sm_object() failing because of a checksum error. In fact, it appears most of the vdev_obsolete_sm_object() callers don't have any error handling for this unlikely case. Reopening.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.