REFERENCE_BY_POINTER when unmounting

[andrewc12]

System information

Type	Version/Name
Distribution Name	Microsoft Windows
Distribution Version	Version 21H2 (OS Build 22000.613)
Kernel Version
Architecture	x86-64
OpenZFS Version	f899f5bced75c2d90489586129a26d1b11ab2d42

Describe the problem you're observing

I had a zfs pool with some folders I was unable to delete.
when I tried to unmount it so I could do a zfs send to a file it crashed.

Describe how to reproduce the problem

create pool
copy university work (with extended attributes) to pool
Try to delete university work
unmount pool

Include any warning/errors/backtraces from the system logs

windbglog

For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff802`046169a0 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff687`1511cc20=0000000000000018
6: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

REFERENCE_BY_POINTER (18)
Arguments:
Arg1: 0000000000000000, Object type of the object whose reference count is being lowered
Arg2: ffffbc0b9cabf760, Object whose reference count is being lowered
Arg3: 0000000000000010, Reserved
Arg4: 0000000000000001, Reserved
	The reference count of an object is illegal for the current state of the object.
	Each time a driver uses a pointer to an object the driver calls a kernel routine
	to increment the reference count of the object. When the driver is done with the
	pointer the driver calls another kernel routine to decrement the reference count.
	Drivers must match calls to the increment and decrement routines. This BugCheck
	can occur because an object's reference count goes to zero while there are still
	open handles to the object, in which case the fourth parameter indicates the number
	of opened handles. It may also occur when the object's reference count drops below zero
	whether or not there are open handles to the object, and in that case the fourth parameter
	contains the actual value of the pointer references count.

Debugging Details:
------------------

Page 1211ad not present in the dump file. Type ".hh dbgerr004" for details

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 2468

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 7456

    Key  : Analysis.Init.CPU.mSec
    Value: 1405

    Key  : Analysis.Init.Elapsed.mSec
    Value: 19245

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 110

    Key  : WER.OS.Branch
    Value: co_release

    Key  : WER.OS.Timestamp
    Value: 2021-06-04T16:28:00Z

    Key  : WER.OS.Version
    Value: 10.0.22000.1


FILE_IN_CAB:  MEMORY.DMP

DUMP_FILE_ATTRIBUTES: 0x1800

BUGCHECK_CODE:  18

BUGCHECK_P1: 0

BUGCHECK_P2: ffffbc0b9cabf760

BUGCHECK_P3: 10

BUGCHECK_P4: 1

BLACKBOXBSD: 1 (!blackboxbsd)


BLACKBOXNTFS: 1 (!blackboxntfs)


BLACKBOXPNP: 1 (!blackboxpnp)


BLACKBOXWINLOGON: 1

PROCESS_NAME:  zfs.exe

STACK_TEXT:  
fffff687`1511cc18 fffff802`046440b5     : 00000000`00000018 00000000`00000000 ffffbc0b`9cabf760 00000000`00000010 : nt!KeBugCheckEx
fffff687`1511cc20 fffff802`248a33e4     : ffffbc0b`ad376b30 00000000`00000002 deadbeef`deadbeef ffffbc0b`9926d080 : nt!ObfReferenceObject+0x13d1b5
fffff687`1511cc60 fffff802`24b1f5db     : 00000000`0000010a ffffbc0b`795cd540 00000000`00000854 fffff802`24b5c29a : OpenZFS!vflush+0x184 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\spl\spl-vnode.c @ 1503] 
fffff687`1511ccf0 fffff802`24b5ff17     : ffffffff`80007178 ffffbc0b`ad376b30 00000000`00000002 00000000`00000001 : OpenZFS!zfs_vfs_unmount+0xeb [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_vfsops.c @ 1609] 
fffff687`1511ce60 fffff802`24b1a20e     : fffff802`24d0f98e 00000004`220d7000 00000001`00008009 00000000`00000000 : OpenZFS!zfs_windows_unmount+0x167 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_vnops_windows_mount.c @ 1592] 
fffff687`1511d340 fffff802`248a8c96     : ffffbc0c`00000001 00000000`00000002 00000000`00000000 00000000`00000000 : OpenZFS!zfs_ioc_unmount+0x7e [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_ioctl_os.c @ 840] 
fffff687`1511d370 fffff802`24b1a015     : ffffbc0b`ad376b30 00000000`00000001 00000000`00000000 ffffbc0b`6000e040 : OpenZFS!zfsdev_ioctl_common+0x816 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\zfs\zfs_ioctl.c @ 7808] 
fffff687`1511d450 fffff802`24afb944     : fffff802`24d34250 ffffbc0b`61bf3a10 00000000`00000001 00000000`00000001 : OpenZFS!zfsdev_ioctl+0x2c5 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_ioctl_os.c @ 790] 
fffff687`1511d540 fffff802`24afb2b1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : OpenZFS!ioctlDispatcher+0x294 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_vnops_windows.c @ 4717] 
fffff687`1511d5f0 fffff802`04504665     : bc0c2342`3e50fff9 72684c37`895d8b8f 00000000`00000000 00000000`00000000 : OpenZFS!dispatcher+0x1e1 [C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\zfs\zfs_vnops_windows.c @ 5625] 
fffff687`1511d6e0 fffff802`04974002     : 00000000`00000001 ffffbc0b`ad376b30 fffff687`1511d781 fffff802`04504423 : nt!IofCallDriver+0x55
fffff687`1511d720 fffff802`04973de2     : ffffbc0c`00000000 fffff687`1511da60 00000000`9c40222f ffffbc0b`ad376b30 : nt!IopSynchronousServiceTail+0x1d2
fffff687`1511d7d0 fffff802`04973146     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xc82
fffff687`1511d900 fffff802`04628f75     : 000000ef`e1afcb84 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtDeviceIoControlFile+0x56
fffff687`1511d970 00007fff`783a3834     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
000000ef`e1afd4f8 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007fff`783a3834


FAULTING_SOURCE_LINE:  C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\spl\spl-vnode.c

FAULTING_SOURCE_FILE:  C:\Users\andre\Downloads\openzfs-windows\openzfs-windows\module\os\windows\spl\spl-vnode.c

FAULTING_SOURCE_LINE_NUMBER:  1503

FAULTING_SOURCE_CODE:  
  1499: 			// while has to start from the top each time. We release
  1500: 			// the node at end of this while.
  1501: 
  1502: 				try {
> 1503: 					Status = ObReferenceObject(fileobject);
  1504: 			//		Status = ObReferenceObjectByPointer(
  1505: 			//		    fileobject,  // fixme, keep this in dvd
  1506: 			//		    0,
  1507: 			//		    *IoFileObjectType,
  1508: 			//		    KernelMode);


SYMBOL_NAME:  OpenZFS!vflush+184

MODULE_NAME: OpenZFS

IMAGE_NAME:  OpenZFS.sys

STACK_COMMAND:  .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET:  184

FAILURE_BUCKET_ID:  0x18_OpenZFS!vflush

OS_VERSION:  10.0.22000.1

BUILDLAB_STR:  co_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {38259c3a-c184-212f-2659-f377b401be79}

Followup:     MachineOwner
---------

cbuf.txt

[andrewc12]

zfs send
https://drive.google.com/file/d/1nRvq_J36Bt2Tq8XsGNW_35tE_K4MsKkZ/view?usp=sharing

[EchterAgo]

This still happens, just the error is slightly different due to a change in the code:

 kd> k
 # Child-SP          RetAddr               Call Site
00 ffffd086`dde6e128 fffff800`15517f82     nt!DbgBreakPointWithStatus
01 ffffd086`dde6e130 fffff800`15517566     nt!KiBugCheckDebugBreak+0x12
02 ffffd086`dde6e190 fffff800`153fd747     nt!KeBugCheck2+0x946
03 ffffd086`dde6e8a0 fffff800`1541ed73     nt!KeBugCheckEx+0x107
04 ffffd086`dde6e8e0 fffff800`21f1e6e8     nt!ObReferenceObjectByPointer+0x21bec3
05 ffffd086`dde6e920 fffff800`22220a3e     OpenZFS!vflush+0x168 [H:\dev\openzfs\module\os\windows\spl\spl-vnode.c @ 1604] 
06 ffffd086`dde6e9b0 fffff800`22268f1f     OpenZFS!zfs_vfs_unmount+0xce [H:\dev\openzfs\module\os\windows\zfs\zfs_vfsops.c @ 1609] 
07 ffffd086`dde6eb40 fffff800`2221c5d5     OpenZFS!zfs_windows_unmount+0x41f [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows_mount.c @ 1581] 
08 ffffd086`dde6f430 fffff800`21f23906     OpenZFS!zfs_ioc_unmount+0x55 [H:\dev\openzfs\module\os\windows\zfs\zfs_ioctl_os.c @ 916] 
09 ffffd086`dde6f470 fffff800`2221c405     OpenZFS!zfsdev_ioctl_common+0x816 [H:\dev\openzfs\module\zfs\zfs_ioctl.c @ 7866] 
0a ffffd086`dde6f550 fffff800`221ff13d     OpenZFS!zfsdev_ioctl+0x2c5 [H:\dev\openzfs\module\os\windows\zfs\zfs_ioctl_os.c @ 866] 
0b ffffd086`dde6f640 fffff800`221fea46     OpenZFS!ioctlDispatcher+0x32d [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 6400] 
0c ffffd086`dde6f710 fffff800`15210665     OpenZFS!dispatcher+0x1e6 [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 7312] 
0d ffffd086`dde6f800 fffff800`1560142c     nt!IofCallDriver+0x55
0e ffffd086`dde6f840 fffff800`15601081     nt!IopSynchronousServiceTail+0x34c
0f ffffd086`dde6f8e0 fffff800`156003f6     nt!IopXxxControlFile+0xc71
10 ffffd086`dde6fa20 fffff800`15410ef8     nt!NtDeviceIoControlFile+0x56
11 ffffd086`dde6fa90 00007fff`6666d0c4     nt!KiSystemServiceCopyEnd+0x28
12 00000085`de71cec8 00007fff`6413591b     ntdll!NtDeviceIoControlFile+0x14
13 00000085`de71ced0 00007fff`64645921     KERNELBASE!DeviceIoControl+0x6b
14 00000085`de71cf40 00007ff7`a5d895d7     KERNEL32!DeviceIoControlImplementation+0x81
15 00000085`de71cf90 00000000`00000000     zpool!zcmd_ioctl_compat+0xe7 [H:\dev\openzfs\lib\libzfs_core\os\windows\libzfs_core_ioctl.c @ 54] 

I encountered this by running multiple scripts in parallel that loop forever on zpool create / zpool destroy.

[sskras]

After several import + export sessions involving my zfs-sample-2 pool, my w10 21H2 also got the similar bug check:

072724-43187-01.dmp.zip

It's 2.2.3rc5, almost the latest release. rc6 seems to include a very specific fix which I think I don't need (please see #384).

Microsoft Windows [Version 10.0.19044.3086]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\system32> zfs version
zfswin-2.2.3rc5
zfs-kmod-zfswin-2.2.3rc5

The pool comes from the short disk image openzfs-sample.img.zip (~300 MiB) that I created on Proxmox 8 using OpenZFS 2.2.2:

saukrs@omn:~$ lsb_release -d
No LSB modules are available.
Description:    Debian GNU/Linux 12 (bookworm)

saukrs@omn:~$ zfs version
zfs-2.2.2-pve1
zfs-kmod-2.2.2-pve1