FuzzerDebug build failure in CI/CD

[shpalani]

Describe the bug

Failed run: https://github.com/microsoft/ebpf-for-windows/actions/runs/4833124330

1. cgroup_count_connect6.o

 44>C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(149,5): error MSB3073: The command "D:\a\ebpf-for-windows\ebpf-for-windows\x64\FuzzerDebug\bpf2c.exe --bpf cgroup_count_connect6.o --dll  >cgroup_count_connect6_dll.c [D:\a\ebpf-for-windows\ebpf-for-windows\x64\FuzzerDebug\cgroup_count_connect6_um\.vcxproj] [D:\a\ebpf-for-windows\ebpf-for-windows\tests\sample\sample.vcxproj]
    44>C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(149,5): error MSB3073: :VCEnd" exited with code -1073740940. [D:\a\ebpf-for-windows\ebpf-for-windows\x64\FuzzerDebug\cgroup_count_connect6_um\.vcxproj] [D:\a\ebpf-for-windows\ebpf-for-windows\tests\sample\sample.vcxproj]
         Done Building Project "D:\a\ebpf-for-windows\ebpf-for-windows\x64\FuzzerDebug\cgroup_count_connect6_um\.vcxproj" (default targets) -- FAILED.
         
         Build FAILED.

44>C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(230,5): error MSB8066: Custom build for 'atomic_instruction_fetch_add.c;bad_map_name.c;bindmonitor.c;bindmonitor_ringbuf.c;bindmonitor_tailcall.c;bpf_call.c;cgroup_count_connect4.c;cgroup_count_connect6.c;cgroup_sock_addr.c;cgroup_sock_addr2.c;decap_permit_packet.c;divide_by_zero.c;droppacket.c;encap_reflect_packet.c;map.c;map_in_map.c;map_in_map_v2.c;map_reuse.c;map_reuse_2.c' exited with code -1. [D:\a\ebpf-for-windows\ebpf-for-windows\tests\sample\sample.vcxproj]
    44>Done Building Project "D:\a\ebpf-for-windows\ebpf-for-windows\tests\sample\sample.vcxproj" (default targets) -- FAILED.

Call stack decode

0:000> k
 # Child-SP          RetAddr               Call Site
00 000000b8`758fbe38 00007ffd`ed2a1e88     ntdll!NtWaitForMultipleObjects+0x14
01 000000b8`758fbe40 00007ffd`ed2a145e     ntdll!WerReportExceptionWorker+0x568
02 000000b8`758fbec0 00007ffd`ed2a0c1b     ntdll!RtlReportExceptionEx+0x80e
03 000000b8`758fbf90 00007ffd`ed2c4014     ntdll!RtlReportException+0x9b
04 000000b8`758fc010 00007ffd`b868eb5e     ntdll!RtlIsZeroMemory+0x154
05 000000b8`758fc040 00007ffd`ed2643af     ucrtbased!_C_specific_handler+0x16e
06 000000b8`758fc0f0 00007ffd`ed1f170e     ntdll!_chkstk+0x12f
07 000000b8`758fc120 00007ffd`ed20f2c3     ntdll!RtlVirtualUnwind2+0x35e
08 000000b8`758fc860 00007ffd`ed2c3fd9     ntdll!RtlRaiseException+0x163
09 000000b8`758fd680 00007ffd`ed2c3fa3     ntdll!RtlIsZeroMemory+0x119
0a 000000b8`758fd6d0 00007ffd`ed2ccdaa     ntdll!RtlIsZeroMemory+0xe3
0b 000000b8`758fd7c0 00007ffd`ed2cd08a     ntdll!_misaligned_access+0x41a
0c 000000b8`758fd7f0 00007ffd`ed2d7709     ntdll!_misaligned_access+0x6fa
0d 000000b8`758fd820 00007ffd`ed26a46d     ntdll!_misaligned_access+0xad79
0e 000000b8`758fd850 00007ffd`ed1d78b1     ntdll!memset+0x432d
0f 000000b8`758fd8a0 00007ffd`ed1db550     ntdll!RtlEnterCriticalSection+0x431
10 000000b8`758fda00 00007ffd`ed1da8c1     ntdll!RtlGetCurrentServiceSessionId+0xbf0
11 000000b8`758fdac0 00007ffd`b69807d9     ntdll!RtlFreeHeap+0x51
12 000000b8`758fdb00 00007ffd`ed22994a     clang_rt_asan_dbg_dynamic_x86_64!_asan_wrap_GlobalSize+0x4d7bd
13 000000b8`758fec30 00007ffd`ed229879     ntdll!RtlRealSuccessor+0x18a
14 000000b8`758fec60 00007ffd`ed2298b8     ntdll!RtlRealSuccessor+0xb9
15 000000b8`758fec90 00007ffd`ed2298b8     ntdll!RtlRealSuccessor+0xf8
16 000000b8`758fecc0 00007ffd`ed2298b8     ntdll!RtlRealSuccessor+0xf8
17 000000b8`758fecf0 00007ffd`ed2979f9     ntdll!RtlRealSuccessor+0xf8
18 000000b8`758fed20 00007ffd`ed237b28     ntdll!LdrInitShimEngineDynamic+0x3319
19 000000b8`758ff0f0 00007ffd`ed25d94d     ntdll!LdrInitializeThunk+0x208
1a 000000b8`758ff170 00007ffd`ed23792e     ntdll!LdrStandardizeSystemPath+0x23d
1b 000000b8`758ff3f0 00000000`00000000     ntdll!LdrInitializeThunk+0xe

OS information

Windows 10, 11

Steps taken to reproduce bug

Seen in CI/CD pipeline intermittently.

Expected behavior

FuzzerDebug build should pass

Actual outcome

FuzzerDebug build fails intermittently

Additional details

No response

[shankarseal]

This is due to microsoft/STL#2908. @shpalani -- please follow up if there is any workarounds.

[shpalani]

05/1/2023: I have reached out to MSVC ASAN Development team by email. Awaiting response.
05/2/2023: As requested, I have provided the crash dump to the ASAN dev team for investigation. Awaiting response on the investigation/findings:

    Directory: \\skyshare\scratch\Users\ebpf-for-windows

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          5/1/2023   4:24 PM       81729463 Build-x64-FuzzerDebug.zip
-a----          5/1/2023   4:25 PM       14662512 Crash-Dumps--x64-FuzzerDebug.zip

05/3/2023: MSVC ASAN and Windows OS Heap team triaged and found a heap corruption and memory from freed allocation with free blocks is causing the issue.
05/4/2023: While the issue is being worked on, workaround provided is to use 'Segment Heap' instead.
05/6/2023: I tried side-by-side assembly manifest. It did not work, as the
new bpf2c.exe.manifest side-by-side assembly manifest isn't being respected because there is already a manifest embedded as a resource in that bpf2c binary (the embedded manifest takes precedence).

[shpalani]

My Analysis/Understandings:

1. Segment Heap:
   To enable Segment Heap for your Win32 application: Application Manifests, assembly version

• Add a heapType element with the value SegmentHeap to your application manifest file. For example:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3">
<assemblyIdentity name="microsoft.ebpf-for-windows.bpf2c" type="win32" version="0.9.0.0"></assemblyIdentity>
<asmv3:application>
  <asmv3:windowsSettings xmlns="http://schemas.microsoft.com/SMI/2020/WindowsSettings">
    <heapType>SegmentHeap</heapType>
  </asmv3:windowsSettings>
</asmv3:application>
</assembly>

2. Image File Execution Options (IFEO) registry keys to enable Segment Heap for your application without modifying the manifest file. For example:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\bpf2c.exe]
"UseSegmentHeap"=dword:00000001

• Note that these methods will enable Segment Heap for all heaps created by your application, including the process heap and any private heaps created by the HeapCreate function. There is currently no official way to enable or disable Segment Heap for individual heaps

---

Segment Heap is a heap implementation that was introduced in Windows 10 for Win32 applications to use memory more efficiently. Segment Heap reduces the overall memory usage of applications by reducing heap fragmentation and improving heap allocation performance.
There are no API changes, if we move the binary to Segment heap from NT Heap usage.

The trade-offs with the NT Heap are as follows:

Aspect	Segment Heap	NT Heap
Memory usage	Lower, especially for applications that make many small and medium-sized allocations	Higher, due to more heap fragmentation
CPU usage	Higher, due to increased overhead of managing segments	Lower, due to simpler allocation algorithm
Availability	Opt-in for Win32 applications, default for Windows apps and some system processes	Default for most Win32 applications
Process	Not supported for 32-bit processes	Supports 32bit and 64bit processes

[shpalani]

Capturing the discussion had in team meeting:

1. ebpf-for-windows application binaries and dlls can move to 'Segment Heap' from 'NT Heap'.
2. Currently, this PR has moved only bpf2c.exe binary to use Segment Heap.
3. Work required to move to Segment heap requires embedding the manifest in their respective .exe/.dll.

Question: Does the external application(s) using ebpf-for-windows dlls/binaries have any dependency/impact by ebpf migrating to segment heap?