Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BSOD when AvastAV is installed #97

Closed
zaldre opened this issue Jul 26, 2017 · 27 comments
Closed

BSOD when AvastAV is installed #97

zaldre opened this issue Jul 26, 2017 · 27 comments
Assignees
Labels
Milestone

Comments

@zaldre
Copy link

zaldre commented Jul 26, 2017

Hi there,

As discussed in https://forum.rclone.org/t/rclone-v1-37-release/3194/16 am currently receiving a BSOD error when attempting to access the mount point created by RCLONE in Windows. This fault can be replicated easily in both Windows 7 and in Windows 10 (Replicated fault on 2 different systems)

Error states that there is an issue with module "aswsnx.sys" and the entire system crashes.

Full process workflow:

1> Install rclone 1.3.7 + winfsp + avast AV (latest version of all)
2> Copy existing rclone.conf file from linux system to windows system into default configuration directory (in my case "C:\Users\respe.config\rclone\rclone.conf")
3> Using this config file, Mount an encrypted remote (In my case, The base remote was a google drive backend referenced as google:/ , encrypted mount point cloud:/) -- Full command syntax : rclone.exe mount cloud: g:
4> Mount drive letter appears in Windows Explorer. Navigating to it via WE and attempting to access the contents causes the fault listed above.

As requested, I have obtained minidump files (3 in total inside this archive - https://www.dropbox.com/s/qrv29kalw8s3gzr/Minidump.7z?dl=0)

Please let me know if any further information is required in order to troubleshoot.

Thanks!

Z

@billziss-gh
Copy link
Collaborator

billziss-gh commented Jul 26, 2017

Thank you. I am going on vacation in a couple of days, but I am nevertheless hoping to find some time to look at this soon.

@billziss-gh billziss-gh changed the title rclone mount using WINFSP causes BSOD when AvastAV is installed BSOD when AvastAV is installed Jul 27, 2017
@billziss-gh
Copy link
Collaborator

billziss-gh commented Jul 27, 2017

@zapoklu I finally took a look at the minidumps you have sent.

All minidumps indicate a PAGE_FAULT_IN_NONPAGED_AREA (50) bugcheck within aswSnx. The stack traces are unfortunately incomplete. The faulting process is explorer.exe. WinFsp does not seem to be (directly) responsible.

Unfortunately this is not a lot of information to go by. I list potential things that may help below:

  • Please run the diag.bat file located in \Program Files (x86)\WinFsp\bin and post the results here.

  • Consider running this with the driver verifier enabled for both AvastAV and WinFsp. You can start the driver verifier by typing verifier on the command line.

  • Consider writing a larger memory dump by changing the kind of dump written. Ideal would be the "kernel memory dump". See instructions here: https://support.microsoft.com/en-us/help/254649/overview-of-memory-dump-file-options-for-windows

From my perspective I will see if I can install Avast and try to reproduce the problem. It may also be worthwhile trying to report this problem to the AvastAV product people.

3: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except.
Typically the address is just plain bad or it is pointing at freed memory.
Arguments:
Arg1: ffffa9814bd77f3c, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff8085c2f6b95, If non-zero, the instruction address which referenced the bad memory
	address.
Arg4: 0000000000000002, (reserved)

Debugging Details:
------------------


Could not read faulting driver name

DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  15063.0.amd64fre.rs2_release.170317-1834

SYSTEM_MANUFACTURER:  Microsoft Corporation

SYSTEM_PRODUCT_NAME:  Surface Pro 4

SYSTEM_SKU:  Surface_Pro_4

SYSTEM_VERSION:  D:0B:08F:1C:03P:38

BIOS_VENDOR:  Microsoft Corporation

BIOS_VERSION:  107.1741.768

BIOS_DATE:  06/13/2017

BASEBOARD_MANUFACTURER:  Microsoft Corporation

BASEBOARD_PRODUCT:  Surface Pro 4

DUMP_TYPE:  2

BUGCHECK_P1: ffffa9814bd77f3c

BUGCHECK_P2: 0

BUGCHECK_P3: fffff8085c2f6b95

BUGCHECK_P4: 2

READ_ADDRESS: fffff80012a5d358: Unable to get MiVisibleState
 ffffa9814bd77f3c 

FAULTING_IP: 
aswSnx+6b95
fffff808`5c2f6b95 468b3421        mov     r14d,dword ptr [rcx+r12]

MM_INTERNAL_CODE:  2

CPU_COUNT: 4

CPU_MHZ: 9c0

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 4e

CPU_STEPPING: 3

CPU_MICROCODE: 6,4e,3,0 (F,M,S,R)  SIG: 9E'00000000 (cache) 9E'00000000 (init)

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  CODE_CORRUPTION

BUGCHECK_STR:  AV

PROCESS_NAME:  explorer.exe

CURRENT_IRQL:  0

ANALYSIS_SESSION_HOST:  WINDOWS

ANALYSIS_SESSION_TIME:  07-27-2017 14:22:21.0286

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

TRAP_FRAME:  ffffa981312df230 -- (.trap 0xffffa981312df230)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000068 rbx=0000000000000000 rcx=000000000000003c
rdx=fffff8085c3c2050 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8085c2f6b95 rsp=ffffa981312df3c0 rbp=ffffbb0625fe006a
 r8=000000000000006a  r9=000000000000001c r10=00000000474a9c8c
r11=ffffa981312df451 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
aswSnx+0x6b95:
fffff808`5c2f6b95 468b3421        mov     r14d,dword ptr [rcx+r12] ds:00000000`0000003c=????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff8001281dfb4 to fffff800127e84c0

STACK_TEXT:  
ffffa981`312def98 fffff800`1281dfb4 : 00000000`00000050 ffffa981`4bd77f3c 00000000`00000000 ffffa981`312df230 : nt!KeBugCheckEx
ffffa981`312defa0 fffff800`127092d6 : 00000000`00000000 ffffa981`4bd77f3c ffffa981`312df230 ffffbb06`23bfe640 : nt!MiSystemFault+0x116e84
ffffa981`312df040 fffff800`127f1d72 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!MmAccessFault+0xae6
ffffa981`312df230 fffff808`5c2f6b95 : ffffa981`312df408 00000000`00000000 00000000`00000000 ffffcb8c`9ba85010 : nt!KiPageFault+0x132
ffffa981`312df3c0 ffffa981`312df408 : 00000000`00000000 00000000`00000000 ffffcb8c`9ba85010 00000000`00000000 : aswSnx+0x6b95
ffffa981`312df3c8 00000000`00000000 : 00000000`00000000 ffffcb8c`9ba85010 00000000`00000000 ffffbb06`2a999e00 : 0xffffa981`312df408


STACK_COMMAND:  kb

CHKIMG_EXTENSION: !chkimg -lo 50 -d !nt
    fffff800127093b0 - nt!MmAccessFault+bc0
	[ f6:f3 ]
    fffff8001281dfd9 - nt!MiValidFault+1160f9 (+0x114c29)
	[ f6:f3 ]
2 errors : !nt (fffff800127093b0-fffff8001281dfd9)

MODULE_NAME: memory_corruption

IMAGE_NAME:  memory_corruption

FOLLOWUP_NAME:  memory_corruption

DEBUG_FLR_IMAGE_TIMESTAMP:  0

MEMORY_CORRUPTOR:  LARGE

FAILURE_BUCKET_ID:  MEMORY_CORRUPTION_LARGE

BUCKET_ID:  MEMORY_CORRUPTION_LARGE

PRIMARY_PROBLEM_CLASS:  MEMORY_CORRUPTION_LARGE

TARGET_TIME:  2017-07-24T23:20:06.000Z

OSBUILD:  15063

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2017-07-06 23:06:35

BUILDDATESTAMP_STR:  170317-1834

BUILDLAB_STR:  rs2_release

BUILDOSVER_STR:  10.0.15063.0.amd64fre.rs2_release.170317-1834

ANALYSIS_SESSION_ELAPSED_TIME: 1889

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:memory_corruption_large

FAILURE_ID_HASH:  {e29154ac-69a4-0eb8-172a-a860f73c0a3c}

Followup:     memory_corruption
---------

@MLPAA
Copy link

MLPAA commented Jul 31, 2017

I wonder if disabling the self-defense module in Avast prior to creation would solve the problem ???
Settings > Troubleshooting > uncheck "Enable Avast Self Defense Module". I do this anytime I create a restore point to prevent a possible corruption.

@sws78
Copy link

sws78 commented Aug 29, 2017

I get the same issue when using AVG Anti Virus.

The same PAGE_FAULT_IN_NONPAGED_AREA error is shown on the BSOD but the file is an AVG file and not an AVAST file.

I had AVAST before AVG and swapped because of the error. I do not get this error with neither installed.

@billziss-gh
Copy link
Collaborator

Interesting. I am back from vacation and will install one of these antivirus products and see for myself.

It might be that WinFsp responds to some file system request in a way that confuses these products. BTW, I am wondering why not use Windows Defender (which does not exhibit any such issues)? Do these third party products provide additional protections?

@MLPAA
Copy link

MLPAA commented Sep 2, 2017 via email

@zaldre
Copy link
Author

zaldre commented Sep 5, 2017

@billziss-gh Historically, Windows Defender has been very poor at identifying Windows threats. Avast has been great at doing it (And is free).

@billziss-gh
Copy link
Collaborator

billziss-gh commented Sep 13, 2017

How to reproduce

I am able to reproduce this problem on a Win8 VM with debugger attached and full WinFsp debug logging enabled.

The file system used is MEMFS running under the SYSTEM account. This makes the created drive accessible from all accounts.

memfs-x64.exe -i -F NTFS -m Z:

The commands to reproduce (once the file system is running):

Z:
echo hello>world
dir

Analysis

The (elided) WinFsp log shows just before the crash:

[0] WinFsp!FspCreate(FFFFF98009606EA0, VolU, IRP_MJ_CREATE, FileObject=FFFFFA8002DF6620[0000000000000000:"\"], Flags=0, DesiredAccess=0x100001, ShareAccess=0x7, Options=0x1000021, FileAttributes=0, AllocationSize=0:0, Ea=0000000000000000, EaLength=0) = FSP_STATUS_IOQ_POST[0]
[0] WinFsp!FspFsvolCreateComplete(FFFFF98009606EA0, VolU, IRP_MJ_CREATE, FileObject=FFFFFA8002DF6620[0000000000000000:"\"]) = STATUS_SUCCESS[1]
[0] WinFsp!FspDirectoryControl(FFFFF980153ECEA0, VolK, IRP_MJ_DIRECTORY_CONTROL[IRP_MN_QUERY_DIRECTORY], FileFullDirectoryInformation, FileObject=FFFFFA8002DF6620) = FSP_STATUS_IOQ_POST[0]
[0] WinFsp!FspFsvolDirectoryControlComplete(FFFFF980153ECEA0, VolK, IRP_MJ_DIRECTORY_CONTROL[IRP_MN_QUERY_DIRECTORY], FileFullDirectoryInformation, FileObject=FFFFFA8002DF6620) = STATUS_SUCCESS[78]

This shows the root directory \ being opened and then queried for FileFullDirectoryInformation. WinFsp responds with 78 bytes, which is the correct size for a FILE_FULL_DIR_INFORMATION struct with the Unicode name world contained in it.

The stack trace reported by WinDbg at the time of the crash confirms that this is due to a directory listing operation:

nt!DbgBreakPointWithStatus
nt!KiBugCheckDebugBreak+0x12
nt!KeBugCheck2+0x79f
nt!KeBugCheckEx+0x104
nt! ?? ::FNODOBFM::`string'+0x32e9e
nt!MmAccessFault+0x54f
nt!KiPageFault+0x16e
aswSnx+0x6b95
aswSnx+0x6009
fltmgr!FltpPerformPreCallbacks+0x324
fltmgr!FltpPassThroughInternal+0x8c
fltmgr!FltpPassThrough+0x169
fltmgr!FltpDispatch+0x9e
nt!IovCallDriver+0x3e6
nt!IopSynchronousServiceTail+0x158
nt!NtQueryDirectoryFile+0xce
nt!KiSystemServiceCopyEnd+0x13
ntdll!NtQueryDirectoryFile+0xa    <------- query (list) a directory
KERNELBASE!FindFirstFileExW+0x44f
cmd!FindFirst+0x54
cmd!ExpandAndApplyToFS+0x154
cmd!WalkTree+0x8f
cmd!PrintPatterns+0x1cb
cmd!Dir+0x11f
cmd!eDirectory+0xd
cmd!FindFixAndRun+0x42f
cmd!Dispatch+0xa7
cmd!_chkstk+0x5074
cmd!mystrchr+0x27d
KERNEL32!BaseThreadInitThunk+0x1a
ntdll!RtlUserThreadStart+0x1d

So what happens is that aswSnx crashes after seeing the WinFsp response to a directory listing operation. It is unclear at this time why this is the case.

Speculation

WinFsp may switch the process context from the process issuing the NtQueryDirectoryFile to the file system process. Such IRP's will always be completed in the context of the file system process, which means that IRP fields like UserBuffer will not be valid (as it is a pointer in a different address space). It may be that aswSnx does not expect such a situation, blindly accesses the UserBuffer and crashes.

While this is a peculiarity of WinFsp I note that Microsoft's own FSD's may not always complete IRP_MJ_DIRECTORY_CONTROL IRP's in the context of the process issuing the NtQueryDirectoryFile. See for example, the FastFat sources here, where it is possible to post such an IRP for later processing (in the "fsp" worker thread).

https://github.com/Microsoft/Windows-driver-samples/blob/96eb96dfb613e4c745db6bd1f53a92fe7e2290fc/filesys/fastfat/dirctrl.c#L385-L392

Update

I now note that this theory may be incorrect because the current process during the bugcheck is actually cmd (i.e. the "originating" process) and not the file system process (according to WinDbg). Furthermore the callstack indicates that the problem happens during the preoperation callback (FltpPerformPreCallbacks ).


Full !analyze -v report

kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except.
Typically the address is just plain bad or it is pointing at freed memory.
Arguments:
Arg1: fffff880e268c32c, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff88006454b95, If non-zero, the instruction address which referenced the bad memory
	address.
Arg4: 0000000000000002, (reserved)

Debugging Details:
------------------


DUMP_CLASS: 1

DUMP_QUALIFIER: 0

BUILD_VERSION_STRING:  9200.16384.amd64fre.win8_rtm.120725-1247

DUMP_TYPE:  0

BUGCHECK_P1: fffff880e268c32c

BUGCHECK_P2: 0

BUGCHECK_P3: fffff88006454b95

BUGCHECK_P4: 2

READ_ADDRESS:  fffff880e268c32c 

FAULTING_IP: 
aswSnx+6b95
fffff880`06454b95 468b3421        mov     r14d,dword ptr [rcx+r12]

MM_INTERNAL_CODE:  2

IMAGE_NAME:  aswSnx.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  599c71d4

MODULE_NAME: aswSnx

FAULTING_MODULE: fffff8800644e000 aswSnx

CPU_COUNT: 1

CPU_MHZ: c1c

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3d

CPU_STEPPING: 4

CPU_MICROCODE: 6,3d,4,0 (F,M,S,R)  SIG: 0'00000000 (cache) 0'00000000 (init)

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  AV

PROCESS_NAME:  cmd.exe

CURRENT_IRQL:  0

ANALYSIS_SESSION_HOST:  WINDOWS

ANALYSIS_SESSION_TIME:  09-13-2017 10:29:51.0738

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

DEVICE_OBJECT: 0000000000000050

TRAP_FRAME:  fffff88005fc9290 -- (.trap 0xfffff88005fc9290)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000044 rbx=0000000000000000 rcx=000000000000003c
rdx=fffff88006520050 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88006454b95 rsp=fffff88005fc9420 rbp=fffffa800393006a
 r8=000000000000006a  r9=000000000000001c r10=0000000085b12407
r11=fffff88005fc94b1 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
aswSnx+0x6b95:
fffff880`06454b95 468b3421        mov     r14d,dword ptr [rcx+r12] ds:00000000`0000003c=????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff801089800ea to fffff8010887f930

STACK_TEXT:  
fffff880`05fc88e8 fffff801`089800ea : 00000000`00000000 00000000`00000050 fffff880`05fc8a50 fffff801`089044b8 : nt!DbgBreakPointWithStatus
fffff880`05fc88f0 fffff801`0897f742 : 00000000`00000003 fffff880`05fc8a50 fffff801`08904ee0 fffff880`05fc8fa0 : nt!KiBugCheckDebugBreak+0x12
fffff880`05fc8950 fffff801`08885144 : fffff801`08b59a80 fffff6fc`008460e8 fffff801`08c1dba0 00000000`00000000 : nt!KeBugCheck2+0x79f
fffff880`05fc9070 fffff801`089f3024 : 00000000`00000050 fffff880`e268c32c 00000000`00000000 fffff880`05fc9290 : nt!KeBugCheckEx+0x104
fffff880`05fc90b0 fffff801`088bfb6f : 00000000`00000000 fffff880`e268c32c fffffa80`047f4080 fffff880`05fc9280 : nt! ?? ::FNODOBFM::`string'+0x32e9e
fffff880`05fc9150 fffff801`08882aee : 00000000`00000000 00000000`dfade190 00000000`00000000 fffff880`05fc9290 : nt!MmAccessFault+0x54f
fffff880`05fc9290 fffff880`06454b95 : fffff880`05fc9468 00000000`00000000 00000000`00000000 00000000`00000001 : nt!KiPageFault+0x16e
fffff880`05fc9420 fffff880`06454009 : 00000000`00000000 fffffa80`04948510 fffffa80`02df6620 00000000`00000000 : aswSnx+0x6b95
fffff880`05fc9500 fffff880`01401844 : fffffa80`039390e8 fffff880`05fc9740 fffffa80`02df6620 fffffa80`0263c102 : aswSnx+0x6009
fffff880`05fc96c0 fffff880`01402a6c : fffff880`05fc9880 fffffa80`0262600c fffffa80`02df6600 fffff980`153f0f00 : fltmgr!FltpPerformPreCallbacks+0x324
fffff880`05fc97d0 fffff880`014012e9 : 00000000`00000000 00000000`0000000c fffff801`08c50900 fffff880`05fc9890 : fltmgr!FltpPassThroughInternal+0x8c
fffff880`05fc9800 fffff880`0140109e : fffffa80`026260b0 fffff801`08c509e8 fffff980`153f1000 fffffa80`026260b8 : fltmgr!FltpPassThrough+0x169
fffff880`05fc9860 fffff801`08e47d26 : fffff980`153f0ea0 00000000`00000002 fffffa80`01d2df00 fffff801`08e47894 : fltmgr!FltpDispatch+0x9e
fffff880`05fc98c0 fffff801`08c509e8 : 00000000`00000002 fffff880`05fc9991 fffffa80`02df6620 fffffa80`02626010 : nt!IovCallDriver+0x3e6
fffff880`05fc9910 fffff801`08c3f92e : 00000000`00000050 00000000`00000000 00000000`00000000 fffffa80`047f4080 : nt!IopSynchronousServiceTail+0x158
fffff880`05fc99e0 fffff801`08884053 : fffffa80`047f4080 000000cf`dfade2a8 fffff880`05fc9aa8 00000000`00000000 : nt!NtQueryDirectoryFile+0xce
fffff880`05fc9a90 000007fc`4c192efa : 000007fc`4924af1b 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
000000cf`dfade058 000007fc`4924af1b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`df000101 : ntdll!NtQueryDirectoryFile+0xa
000000cf`dfade060 000007f6`126111a4 : 00000000`00000000 00000000`00000006 000000cf`dfc93de0 00000000`00000000 : KERNELBASE!FindFirstFileExW+0x44f
000000cf`dfade410 000007f6`12623424 : 000000cf`dfc92254 000007f6`12627060 000000cf`dfc93de0 000000cf`dfc8be20 : cmd!FindFirst+0x54
000000cf`dfade460 000007f6`1262350f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : cmd!ExpandAndApplyToFS+0x154
000000cf`dfade750 000007f6`12626b4d : 000000cf`dfc93de0 000000cf`dfc932b0 000000cf`00000006 000000cf`00000000 : cmd!WalkTree+0x8f
000000cf`dfade890 000007f6`12623993 : 00000000`00000000 00000000`00000000 00000000`00000004 00000000`00000020 : cmd!PrintPatterns+0x1cb
000000cf`dfadf170 000007f6`1262385d : 000029be`23d2e840 000000cf`dfc889c0 00000000`0000002a 000000cf`dfc889c0 : cmd!Dir+0x11f
000000cf`dfadf690 000007f6`1261148f : 000000cf`dfc88f20 00000000`00000020 00000000`00000018 000000cf`dfc88f20 : cmd!eDirectory+0xd
000000cf`dfadf6c0 000007f6`12611387 : 00000000`0001fbc0 000000cf`dfc8daa0 00000000`00000001 000007f6`1261417c : cmd!FindFixAndRun+0x42f
000000cf`dfadfb60 000007f6`1263d5a2 : 00000000`00000002 000007f6`1262ddc8 00000000`00000000 00000000`00000000 : cmd!Dispatch+0xa7
000000cf`dfadfc10 000007f6`1262b721 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : cmd!_chkstk+0x5074
000000cf`dfadfc70 000007fc`49aa167e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : cmd!mystrchr+0x27d
000000cf`dfadfcb0 000007fc`4c1ac3f1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1a
000000cf`dfadfce0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d


STACK_COMMAND:  kb

THREAD_SHA1_HASH_MOD_FUNC:  23ff93fec381c26e80d5d2797079f0b24512e3a8

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  a91191f4e5f61b1a1ae3356b10dd6031ee6f526f

THREAD_SHA1_HASH_MOD:  20367eb85e375fb3b100697e65a1e842803cbb30

FOLLOWUP_IP: 
aswSnx+6b95
fffff880`06454b95 468b3421        mov     r14d,dword ptr [rcx+r12]

FAULT_INSTR_CODE:  21348b46

SYMBOL_STACK_INDEX:  7

SYMBOL_NAME:  aswSnx+6b95

FOLLOWUP_NAME:  MachineOwner

BUCKET_ID_FUNC_OFFSET:  6b95

FAILURE_BUCKET_ID:  AV_VRF_aswSnx!Unknown_Function

BUCKET_ID:  AV_VRF_aswSnx!Unknown_Function

PRIMARY_PROBLEM_CLASS:  AV_VRF_aswSnx!Unknown_Function

TARGET_TIME:  2015-11-17T23:17:17.000Z

OSBUILD:  9200

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8

OSEDITION:  Windows 8 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2012-07-26 03:32:43

BUILDDATESTAMP_STR:  120725-1247

BUILDLAB_STR:  win8_rtm

BUILDOSVER_STR:  6.2.9200.16384.amd64fre.win8_rtm.120725-1247

ANALYSIS_SESSION_ELAPSED_TIME: 56d

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_vrf_aswsnx!unknown_function

FAILURE_ID_HASH:  {171ccd04-a7c3-21cb-cc30-5ec40811ff55}

Followup:     MachineOwner
---------

@billziss-gh
Copy link
Collaborator

I have contacted the Avast people with a link to this issue. Hopefully we will hear from them.

@zaldre
Copy link
Author

zaldre commented Sep 15, 2017

Thank you so much @billziss-gh for taking the time to look at this. Much appreciated.

@billziss-gh
Copy link
Collaborator

@zapoklu, you are welcome. Let's hope we will also hear from the Avast people, because we might need their help to figure out why this is failing.

@billziss-gh
Copy link
Collaborator

As of 2017-10-03 I have not received any response from Avast on this. I just emailed them again and plan to also ping them on Twitter. Does anyone know any solid contact that we can use to gain traction on this issue?

@TheMCNerd2017
Copy link

I have no idea of a solid contact for Avast. I personally believe that if a lot of people keep telling them about the issue, they will do something about it.

@MLPAA
Copy link

MLPAA commented Oct 4, 2017

Did you try disabling the Self Defense Module as suggested in my original (first) reply?

@billziss-gh
Copy link
Collaborator

Some good news as I have finally heard from Avast on this. Will keep you posted on the progress.

@billziss-gh
Copy link
Collaborator

@MLPAA I think the intent is to fully solve the problem rather than simply work around it. I personally do not use Avast, but people who do may try your suggestion.

@MLPAA
Copy link

MLPAA commented Oct 4, 2017

@billziss-gh The suggestion isn't to work around it but to solve it. You obviously don't use the product or you would have known why the suggestion was made.

@billziss-gh
Copy link
Collaborator

billziss-gh commented Oct 4, 2017

@MLPAA If you have found that disabling the "Self Defense Module" resolves the problem, then I am happy to promote this as a solution/workaround until we fully resolve the issue.

@billziss-gh
Copy link
Collaborator

billziss-gh commented Oct 5, 2017

@MLPAA FWIW I tried disabling the "Self Defense Module" as per your suggestion and the BSOD still appeared.

@zaldre
Copy link
Author

zaldre commented Oct 5, 2017

Thanks for the effort invested so far @billziss-gh . Really appreciate it mate.

@billziss-gh
Copy link
Collaborator

billziss-gh commented Oct 13, 2017

I finally have an update regarding this issue. Petr Kurtin of Avast has been very helpful in explaining to me some of the internal workings of the Avast filter. Thank you Petr!

During IRP_MJ_DIRECTORY_CONTROL preoperation, the Avast filter may issue an FltQueryDirectoryFile and process the results. Petr has found that under WinFsp he receives "empty" results (i.e. the response buffer he passes to FltQueryDirectoryFile does not get filled). We speculate that this is because FltQueryDirectoryFile fails to copy the Irp->AssociatedIrp.SystemBuffer into Irp->UserBuffer.

It is not clear why the Filter Manager fails to copy the SystemBuffer to the UserBuffer in this particular case. It should be noted that this behavior is different from the default IO Manager behavior (NtQueryDirectoryFile).

It seems clear to both of us that Avast is not at fault here. It uses a documented Filter Manager interface to fetch directory data. Therefore the only place that this can be fixed is WinFsp.

Currently WinFsp uses Irp->AssociatedIrp.SystemBuffer to report an IRP_MJ_DIRECTORY_CONTROL / IRP_MN_QUERY_DIRECTORY result. I intend to change this to use Irp->MdlAddress which should resolve this issue. However the IRP_MN_QUERY_DIRECTORY handler is quite complicated and this change is not trivial. It will take me some time to find the best way to resolve this.

@billziss-gh billziss-gh self-assigned this Oct 13, 2017
@fsgeek
Copy link

fsgeek commented Oct 13, 2017 via email

@billziss-gh
Copy link
Collaborator

billziss-gh commented Oct 13, 2017

WinFsp does not use buffered I/O. It uses "neither".

As you well know, @fsgeek, under the IO Manager if you set the IRP bits IRP_BUFFERED_IO | IRP_DEALLOCATE_BUFFER and also set Irp->AssociatedIrp.SystemBuffer, the IO Manager will dutifully copy the SystemBuffer into Irp->UserBuffer and then deallocate the SystemBuffer during IRP completion (and during the special kernel APC in the context of the originating thread).

[WinFsp sets the IRP_BUFFERED_IO | IRP_DEALLOCATE_BUFFER bits and Irp->AssociatedIrp.SystemBuffer during IRP_MN_QUERY_DIRECTORY processing. FastFat may also do so in some cases (FatBufferUserBuffer) but not for IRP_MN_QUERY_DIRECTORY (I think).]

I am suspecting that FltQueryDirectoryFile does its own completion processing and does not expect an FSD to ever set the SystemBuffer. In a somewhat similar case, discussed here, SRV2 was unable to handle IRP_MN_QUERY_DIRECTORY responses from WinFsp and I had to work around the problem.

So we now have two Windows components that dislike how WinFsp handles IRP_MN_QUERY_DIRECTORY even if the IO Manager is ok with it. This means that I must bite the bullet and change this handling.

@fsgeek
Copy link

fsgeek commented Oct 13, 2017 via email

@billziss-gh
Copy link
Collaborator

billziss-gh commented Oct 20, 2017

Commit 07f15c2 should fix this issue.

This commit passes winfsp-tests locally with and without Avast (see note at the end). Directory listings work correctly with and without Avast. The commit has not been yet tested against the full test suite on AppVeyor.

This is a MAJOR change in how WinFsp handles IRP_MN_QUERY_DIRECTORY. Previously WinFsp used the Irp->AssociatedIrp.SystemBuffer to report results. Now WinFsp uses Irp->MdlAddress for this purpose. This change appears to resolve the BSOD issue when Avast is present. However because IRP_MN_QUERY_DIRECTORY handling is complicated in WinFsp and because the changes are non-trivial, I consider this commit experimental for the time being. More testing will have to be performed to develop confidence in this change.

One of my goals is to incorporate Avast into the WinFsp test suite to avoid such issues in the future. More commits towards this goal will follow. Such commits will appear in the pvt-avast branch.

This change should appear in v1.2B2 when it is released.

NOTE: The winfsp-tests querydir_buffer_overflow_test currently fails under Avast. However this failure happens under both NTFS and MEMFS, therefore it does not appear to be WinFsp specific. Furthermore the querydir_buffer_overflow_test tests a scenario that is unlikely to appear in practice.

querydir_buffer_overflow_test.......... KO
    ASSERT(sizeof(FILE_DIRECTORY_INFORMATION) == IoStatus.Information) failed at
 dirctl-test.c:367:querydir_buffer_overflow_dotest

UPDATE: Latest commit (1a02438) in pvt-avast branch passes all tests on AppVeyor.

@billziss-gh billziss-gh added this to the v1.2 milestone Oct 20, 2017
@billziss-gh
Copy link
Collaborator

Commit fa388e5 adds regular Avast testing in AppVeyor.

@billziss-gh
Copy link
Collaborator

Closing this as it is now resolved. The fix will be included in the v1.2B2 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

6 participants