Memory Leak #16217

Silvenga · 2023-10-23T15:43:52Z

Windows Terminal version

1.17.11461.0

Windows build number

10.0.19045.0

Other Software

Powershell 5.1.19041.3570

Steps to reproduce

This has occurred a couple times, mostly when I leave docker logs open overnight.

I do have a memory dump - 300MB compressed. I cannot upload the dump publicly though.

Expected Behavior

No response

Actual Behavior

Memory usage continues to climb and Windows starts paging like crazy.

I suspect something is off with the buffer limit (I'm assuming there's a limit somewhere).

zadjii-msft · 2023-10-23T15:46:12Z

Are you using any sort of UIA application - Narrator, Jaws, NVDA, something like that/? (ala #16209)

Silvenga · 2023-10-23T15:57:01Z

@zadjii-msft no "screen readers" are active.

I did see that issue, but wasn't sure what software NVDA was - but now that makes sense after you mentioned Narrator/Jaws.

carlos-zamora · 2023-10-24T20:15:38Z

I do have a memory dump - 300MB compressed. I cannot upload the dump publicly though.

Hey @Silvenga. Any chance you're willing to share the dump privately over email? TIA

Silvenga · 2023-10-24T20:48:51Z

@carlos-zamora of course!

My email is on my GitHub profile (I think anyone logged in can see it). Shoot me an email I can reply to and I'll figure out a place to upload the dump... (likely One Drive)

Silvenga · 2023-10-24T23:21:44Z

Following up via email.

Silvenga · 2023-11-01T14:47:29Z

Also reproduced in v1.18.2822.0. I have another dump if that would help at all (262MB compressed).

zadjii-msft · 2023-11-01T16:20:31Z

I've been digging through the first dump this morning, but I'm honestly not sure I actually know how to take a dump and find the root cause of the leak.

Details

0:000> !heap -s

                                      Process    Total      Total 
                              Global     Heap Reserved  Committed 
    Heap Address  Signature    Flags     List    Bytes      Bytes 
                                        Index      (K)        (K) 

     18221990000   ddeeddee        0        1    67716      36452 
     18223470000   ddeeddee     1000        3  2683012    2020796 
     182234a0000   ddeeddee     1000        4     1156         32 
     1822f430000   ddeeddee     1000        7     1156         28 
     1822f490000   ddeeddee     1000        8     1156         48 
     1822ff13000   ddeeddee     1000        9     1156         36

> heap -s -h 18223470000     

*****************************************************************************************************
                                          Backend Stats
*****************************************************************************************************

Segment Count                      : 0
Bucket Size                        : 4096
Largest possible backend allocation: 524288
Number of buckets                  : 128

_____________________________________________________________________________________________________

   Range    Range     Page    Total     Busy  Committed  Decomitted  Decomitted   Unused     Unused
     Min      Max    Range    Pages    Pages      Pages        Busy        Free    Bytes  Committed
                     Count                                    Pages       Pages               Bytes
_____________________________________________________________________________________________________

       1     4096       39       39       39         39           0           0        0          0
-----------------------------------------------------------------------------------------------------
    4097     8192       35       70       66         70           0           0        0          0
-----------------------------------------------------------------------------------------------------
   12289    16384        8       32       32         30           2           0        0          0
-----------------------------------------------------------------------------------------------------
   28673    32768        9       72       72         47          25           0        0          0
-----------------------------------------------------------------------------------------------------
   40961    45056        2       22        0         11           0          11        0          0
-----------------------------------------------------------------------------------------------------
   65537    69632       12      204      187        154          33          17    45056          0
-----------------------------------------------------------------------------------------------------
   73729    77824        1       19        0          2           0          17        0          0
-----------------------------------------------------------------------------------------------------
  131073   135168        8      264      264        256           8           0    32768          0
-----------------------------------------------------------------------------------------------------
  237569   241664     2598   153282        0          0           0      153282        0          0
-----------------------------------------------------------------------------------------------------
  262145   266240     7802   507130   507130     499286        7844           0 31956992          0
-----------------------------------------------------------------------------------------------------
  376833   380928        1       93       93         93           0           0     2592       1504
-----------------------------------------------------------------------------------------------------
  520193   524288        2     4269        0          0           0        4269        0          0
_____________________________________________________________________________________________________

      Total Count    10517   665496   507883     499988        7912      157596 32037408       1504
_____________________________________________________________________________________________________



************************************************************************************************************************
                                             Lfh allocation stats.
************************************************************************************************************************

________________________________________________________________________________________________________________________

  Bucket    Block      Total      Empty    Block             Busy         Free         Free   Unused     Tail     Tail Affinity Slots (Segments - Blocks)
   Index     Size Subsegment Subsegment    Count            Bytes  Decommitted    Committed    Bytes    Bytes Committed
          (Bytes)      Count      Count                                  Bytes        Bytes                      Bytes
________________________________________________________________________________________________________________________

       1       16          3          0     1752             2181         8192        16800      859        0        0 (   3 -  1752 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 , )
------------------------------------------------------------------------------------------------------------------------
<omitted>
------------------------------------------------------------------------------------------------------------------------
      10      160       7825          0 12757910       1939118440            0        88320 102058840   520608   520608 ( 7825 - 12757910 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 ,   0 -     0 , )
------------------------------------------------------------------------------------------------------------------------
<omitted>
________________________________________________________________________________________________________________________

      Total Count       7900          0 12769072       1939267539       200240       560976 102069245   532576   532112
________________________________________________________________________________________________________________________

So like, I can see there's a leak somewhere in that heap, but finding what's causing that seems hard. The memory in that area isn't super informative... though there is a pattern of repeating 47 23 bytes that's curious

but that's kinda meaningless to me

zadjii-msft · 2023-11-01T16:28:41Z


[0x0]   Microsoft_Terminal_Control!memcpy+0x180   0x3c583ffbd8   0x7ffd8be0880b   
[0x1]   Microsoft_Terminal_Control!memcpy_s+0x43   0x3c583ffbe0   0x7ffd8be0873a   
[0x2]   Microsoft_Terminal_Control!winrt::impl::create_hstring_on_heap+0x36   0x3c583ffc10   0x7ffd8bee273b   
[0x3]   Microsoft_Terminal_Control!winrt::hstring::{ctor}+0xb   0x3c583ffc40   0x7ffd8bf0ed02   
[0x4]   Microsoft_Terminal_Control!winrt::hstring::{ctor}+0xb   0x3c583ffc40   0x7ffd8bf0ed02   
[0x5]   Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::NotifyNewOutput+0x1b   0x3c583ffc40   0x7ffd8bf0ed02   
[0x6]   Microsoft_Terminal_Control!Microsoft::Console::Render::UiaEngine::Present+0x122   0x3c583ffc70   0x7ffd8be21e70   
[0x7]   Microsoft_Terminal_Control!Microsoft::Console::Render::Renderer::_PaintFrameForEngine+0x1f0   0x3c583ffd20   0x7ffd8be21c4e   
[0x8]   Microsoft_Terminal_Control!Microsoft::Console::Render::Renderer::PaintFrame+0x4a   0x3c583ffdc0   0x7ffd8be21ba9   
[0x9]   Microsoft_Terminal_Control!Microsoft::Console::Render::RenderThread::_ThreadProc+0x6d   0x3c583ffdf0   0x7ffeae3d7344   
[0xa]   kernel32!BaseThreadInitThunk+0x14   0x3c583ffe20   0x7ffeb00a26b1   
[0xb]   ntdll!RtlUserThreadStart+0x21   0x3c583ffe50   0x0

well aren't you a suspicious stack frame in the dump.

What we've got here is InteractivityAutomationPeer::NotifyNewOutput allocating a new hstring to copy the substr of the _queuedOutput in UiaEngine. That seems normal.

That hstring then gets copied to the handler we set up in TermControlAutomationPeer:

terminal/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp

Line 85 in 0289cb0

_contentAutomationPeer.NewOutput([this](auto&&, hstring newOutput) { NotifyNewOutput(newOutput); });
- That could probably take an hstring& but whatever
TermControlAutomationPeer::NotifyNewOutput then takes that and sanitizes it

terminal/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp

Line 39 in 0289cb0

static std::wstring Sanitize(std::wstring_view text)
So now in the rest of NotifyNewOutput, we're using a new wstring.
Then, to actually raise the message, we make ANOTHER copy of the string, now as a hstring:

terminal/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp

Line 255 in 0289cb0

dispatcher.RunAsync(Windows::UI::Core::CoreDispatcherPriority::Normal, [weakThis{ get_weak() }, sanitizedCopy{ hstring{ sanitized } }]() {
At this point, we'll pop out of NotifyNewOutput.
- We'll dtor the sanitized that was on the stack.
- We'll pop out of the lambda too, which should decrease the ref count of the newOutput that was passed to it
- that hstring was created in the call to _NewOutputHandlers, so it shouldn't possibly have any outstanding refs

Silvenga · 2023-11-01T16:47:05Z

It's been a while since I've had to find a memory leak in unmanaged code. Last one I debugged was to track down allocations in a CLR profiler. I think we ultimately hooked the native allocations using DotTrace.

So with DotTrace, the most impactful stack that allocated memory that wasn't released:

  90.4%   `winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::<lambda_invoker_cdecl>  •  1,496 KB  •  TerminalConnection.dll!`winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::<lambda_invoker_cdecl>
    90.4%   winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread  •  1,496 KB  •  TerminalConnection.dll!winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread
      90.4%   winrt::impl::invoke<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >  •  1,496 KB  •  TerminalConnection.dll!winrt::impl::invoke<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >
        90.4%   winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_232_> >::Invoke  •  1,496 KB  •  Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_232_> >::Invoke
          90.4%   winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler  •  1,496 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler
            90.4%   Microsoft::Terminal::Core::Terminal::Write  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::Write
              90.4%   Microsoft::Console::VirtualTerminal::StateMachine::ProcessString  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString
                90.4%   Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString'::`2'::<lambda_1> >  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString'::`2'::<lambda_1> >
                  90.4%   Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString
                    90.4%   Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString
                      90.4%   Microsoft::Console::VirtualTerminal::AdaptDispatch::_WriteToBuffer  •  1,496 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::_WriteToBuffer
                        82.6%   Microsoft::Console::Render::UiaEngine::NotifyNewText  •  1,367 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::UiaEngine::NotifyNewText
                          82.6%   std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append  •  1,367 KB  •  Microsoft.Terminal.Control.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append
                            82.6%   std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::_Reallocate_grow_by<`std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append'::`2'::<lambda_1>,wchar_t const *,unsigned __int64>  •  1,367 KB  •  Microsoft.Terminal.Control.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::_Reallocate_grow_by<`std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append'::`2'::<lambda_1>,wchar_t const *,unsigned __int64>
                              82.6%   std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>  •  1,367 KB  •  Microsoft.Terminal.Control.dll!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>
                                82.6%   operator new  •  1,367 KB  •  Microsoft.Terminal.Control.dll!operator new
                                ► 82.6%   _malloc_base  •  1,367 KB  •  ucrtbase.dll!_malloc_base
                      ► 7.81%   til::basic_rle<TextAttribute,unsigned short,til::small_vector<til::rle_pair<TextAttribute,unsigned short>,1> >::_replace_unchecked  •  129 KB  •  Microsoft.Terminal.Control.dll!til::basic_rle<TextAttribute,unsigned short,til::small_vector<til::rle_pair<TextAttribute,unsigned short>,1> >::_replace_unchecked
              ► <0.01%   Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString  •  0.1 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString

About 1.5MB of unreleased native allocations in this 2 minute trace. I'll try and get a longer trace since I'm assuming this is just being stored in the buffer?

zadjii-msft · 2023-11-01T16:55:06Z

terminal/src/renderer/uia/UiaRenderer.cpp

Lines 171 to 182 in 0289cb0

    
           [[nodiscard]] HRESULT UiaEngine::NotifyNewText(const std::wstring_view newText) noexcept 
        
           try 
        
           { 
        
               if (!newText.empty()) 
        
               { 
        
                   _newOutput.append(newText); 
        
                   _newOutput.push_back(L'\n'); 
        
                   _textBufferChanged = true; 
        
               } 
        
               return S_OK; 
        
           } 
        
           CATCH_LOG_RETURN_HR(E_FAIL);

terminal/src/renderer/uia/UiaRenderer.cpp

Lines 224 to 236 in 0289cb0

    
           [[nodiscard]] HRESULT UiaEngine::EndPaint() noexcept 
        
           { 
        
               RETURN_HR_IF(S_FALSE, !_isEnabled); 
        
               RETURN_HR_IF(E_INVALIDARG, !_isPainting); // invalid to end paint when we're not painting 
        
               // Snap this now while we're still under lock 
        
               // so present can work on the copy while another 
        
               // thread might start filling the next "frame" 
        
               // worth of text data. 
        
               std::swap(_queuedOutput, _newOutput); 
        
               _newOutput.clear(); 
        
               return S_OK; 
        
           }

terminal/src/renderer/uia/UiaRenderer.cpp

Lines 253 to 303 in 0289cb0

    
           [[nodiscard]] HRESULT UiaEngine::Present() noexcept 
        
           { 
        
               RETURN_HR_IF(S_FALSE, !_isEnabled); 
        
               // Fire UIA Events here 
        
               if (_selectionChanged) 
        
               { 
        
                   try 
        
                   { 
        
                       _dispatcher->SignalSelectionChanged(); 
        
                   } 
        
                   CATCH_LOG(); 
        
               } 
        
               if (_textBufferChanged) 
        
               { 
        
                   try 
        
                   { 
        
                       _dispatcher->SignalTextChanged(); 
        
                   } 
        
                   CATCH_LOG(); 
        
               } 
        
               if (_cursorChanged) 
        
               { 
        
                   try 
        
                   { 
        
                       _dispatcher->SignalCursorChanged(); 
        
                   } 
        
                   CATCH_LOG(); 
        
               } 
        
               try 
        
               { 
        
                   // The speech API is limited to 1000 characters at a time. 
        
                   // Break up the output into 1000 character chunks to ensure 
        
                   // the output isn't cut off. 
        
                   static constexpr size_t sapiLimit{ 1000 }; 
        
                   const std::wstring_view output{ _queuedOutput }; 
        
                   for (size_t offset = 0; offset < output.size(); offset += sapiLimit) 
        
                   { 
        
                       _dispatcher->NotifyNewOutput(output.substr(offset, sapiLimit)); 
        
                   } 
        
               } 
        
               CATCH_LOG(); 
        
               _selectionChanged = false; 
        
               _textBufferChanged = false; 
        
               _cursorChanged = false; 
        
               _isPainting = false; 
        
               _queuedOutput.clear(); 
        
               return S_OK; 
        
           }

zadjii-msft · 2023-11-01T16:56:28Z

I bet if we're not enabled we just queue up all the text into _newOutput then just keep queueing

Silvenga · 2023-11-01T16:57:32Z

Another top stack (related to UiaEngine::Present as you linked above), about 12MB of allocations when printing enough text to fill the scroll back buffer every few seconds:

  100%   All Calls  •  12,133 KB
    53.4%   Microsoft::Console::Render::RenderThread::_ThreadProc  •  6,484 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::RenderThread::_ThreadProc
      53.4%   Microsoft::Console::Render::Renderer::PaintFrame  •  6,484 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::Renderer::PaintFrame
        53.4%   Microsoft::Console::Render::Renderer::_PaintFrameForEngine  •  6,484 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::Renderer::_PaintFrameForEngine
          53.4%   Microsoft::Console::Render::UiaEngine::Present  •  6,483 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::UiaEngine::Present
            46.1%   winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::NotifyNewOutput  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::NotifyNewOutput
              46.1%   winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring> >::operator()<winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer,winrt::hstring>  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::event<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring> >::operator()<winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer,winrt::hstring>
                46.1%   winrt::impl::invoke<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>,winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer,winrt::hstring>  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::impl::invoke<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>,winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer,winrt::hstring>
                  46.1%   winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>::operator()  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>::operator()
                    46.1%   winrt::impl::delegate<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>,`winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer::TermControlAutomationPeer'::`2'::<lambda_4> >::Invoke  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Windows::Foundation::TypedEventHandler<winrt::Windows::Foundation::IInspectable,winrt::hstring>,`winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer::TermControlAutomationPeer'::`2'::<lambda_4> >::Invoke
                      46.1%   winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer::NotifyNewOutput  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer::NotifyNewOutput
                        46.1%   winrt::impl::consume_Windows_UI_Core_ICoreDispatcher<winrt::Windows::UI::Core::ICoreDispatcher>::RunAsync  •  5,587 KB  •  Microsoft.Terminal.Control.dll!winrt::impl::consume_Windows_UI_Core_ICoreDispatcher<winrt::Windows::UI::Core::ICoreDispatcher>::RunAsync
                          46.1%   Windows::UI::Core::CDispatcher::RunAsync  •  5,587 KB  •  Windows.UI.dll!Windows::UI::Core::CDispatcher::RunAsync
                            46.0%   Windows::UI::Core::CDispatcher::EnqueueAsyncWork  •  5,585 KB  •  Windows.UI.dll!Windows::UI::Core::CDispatcher::EnqueueAsyncWork
                              46.0%   Windows::UI::Core::CoreAsyncInfo_CreateInstance<Windows::UI::Core::CCoreAsyncBase<Windows::Foundation::IAsyncAction,Windows::Foundation::IAsyncActionCompletedHandler,&Windows::UI::Core::AsyncActionName,&Windows::UI::Core::CoreAsyncActionName> >  •  5,585 KB  •  Windows.UI.dll!Windows::UI::Core::CoreAsyncInfo_CreateInstance<Windows::UI::Core::CCoreAsyncBase<Windows::Foundation::IAsyncAction,Windows::Foundation::IAsyncActionCompletedHandler,&Windows::UI::Core::AsyncActionName,&Windows::UI::Core::CoreAsyncActionName> >
                                46.0%   operator new  •  5,585 KB  •  Windows.UI.dll!operator new
                                  46.0%   operator new  •  5,585 KB  •  Windows.UI.dll!operator new
                                    46.0%   malloc  •  5,585 KB  •  msvcrt.dll!malloc
                          ► 0.02%   Windows.UI.dll  •  2.5 KB
          ► 3.78%   winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::SignalTextChanged  •  458 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::SignalTextChanged
          ► 3.60%   winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::SignalCursorChanged  •  437 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::InteractivityAutomationPeer::SignalCursorChanged
        ► <0.01%   Microsoft::Console::Render::DxEngine::Present  •  0.8 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::DxEngine::Present
        ► <0.01%   Microsoft::Console::Render::Renderer::_PaintBufferOutput  •  0.2 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::Renderer::_PaintBufferOutput
    43.6%   `winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::<lambda_invoker_cdecl>  •  5,287 KB  •  TerminalConnection.dll!`winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::<lambda_invoker_cdecl>
      43.6%   winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread  •  5,287 KB  •  TerminalConnection.dll!winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread
        43.6%   winrt::impl::invoke<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >  •  5,287 KB  •  TerminalConnection.dll!winrt::impl::invoke<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >
          43.6%   winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_232_> >::Invoke  •  5,287 KB  •  Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_232_> >::Invoke
            43.6%   winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler  •  5,287 KB  •  Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler
              43.6%   Microsoft::Terminal::Core::Terminal::Write  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::Write
                43.6%   Microsoft::Console::VirtualTerminal::StateMachine::ProcessString  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString
                  43.6%   Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString'::`2'::<lambda_1> >  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString'::`2'::<lambda_1> >
                    43.6%   Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionPrintString
                      43.6%   Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::PrintString
                        43.6%   Microsoft::Console::VirtualTerminal::AdaptDispatch::_WriteToBuffer  •  5,287 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::_WriteToBuffer
                          42.3%   Microsoft::Console::Render::UiaEngine::NotifyNewText  •  5,127 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::Render::UiaEngine::NotifyNewText
                            42.3%   std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append  •  5,127 KB  •  Microsoft.Terminal.Control.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append
                              42.3%   std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::_Reallocate_grow_by<`std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append'::`2'::<lambda_1>,wchar_t const *,unsigned __int64>  •  5,127 KB  •  Microsoft.Terminal.Control.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::_Reallocate_grow_by<`std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::append'::`2'::<lambda_1>,wchar_t const *,unsigned __int64>
                                42.3%   std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>  •  5,127 KB  •  Microsoft.Terminal.Control.dll!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>
                                  42.3%   operator new  •  5,127 KB  •  Microsoft.Terminal.Control.dll!operator new
                                    42.3%   _malloc_base  •  5,127 KB  •  ucrtbase.dll!_malloc_base
                        ► 1.32%   til::basic_rle<TextAttribute,unsigned short,til::small_vector<til::rle_pair<TextAttribute,unsigned short>,1> >::_replace_unchecked  •  160 KB  •  Microsoft.Terminal.Control.dll!til::basic_rle<TextAttribute,unsigned short,til::small_vector<til::rle_pair<TextAttribute,unsigned short>,1> >::_replace_unchecked
                ► <0.01%   Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString  •  0.4 KB  •  Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_ActionPrintString
  ► 2.98%   std::thread::_Invoke<std::tuple<`WindowEmperor::_createNewWindowThread'::`2'::<lambda_1> >,0>  •  362 KB  •  WindowsTerminal.exe!std::thread::_Invoke<std::tuple<`WindowEmperor::_createNewWindowThread'::`2'::<lambda_1> >,0>

zadjii-msft · 2023-11-01T18:50:44Z

Moving a comment:

Possible cause: This doesn't release any memory main/src/renderer/uia/UiaRenderer.cpp#L234

If it's due to that, we could limit to either 4KiB or 2x the previous write size, whatever is larger (as an example):
if (_newOutput.capacity() >= std::max<size_t>(4096, _queuedOutput.size() * 2))
{
    _newOutput = std::wstring{};
    _newOutput.reserve(2048);
}

zadjii-msft · 2023-11-01T18:55:47Z

I suppose the question now is what to do with the output in this case. We disable the UIA renderer when the control loses focus, so presumably the output can just be lost safely, yea? Like, we shouldn't just scream the entire buffer back at UIA once the control regains focus.

Second question would be: Do we filter based on _isEnabled in UiaEngine::NotifyNewText, or do we just always dump the buffer in EndPaint even if we're disabled? I think we have to do the first, because we'll early S_FALSE out of StartPaint if we're disabled. I'll leave that call to @carlos-zamora ☺️

carlos-zamora · 2023-11-01T19:23:54Z

I suppose the question now is what to do with the output in this case. We disable the UIA renderer when the control loses focus, so presumably the output can just be lost safely, yea? Like, we shouldn't just scream the entire buffer back at UIA once the control regains focus.

Correct. I'd modify NotifyNewText() to check _isEnabled at the start. No point in storing the output if we're not enabled.

Second question would be: Do we filter based on _isEnabled in UiaEngine::NotifyNewText, or do we just always dump the buffer in EndPaint even if we're disabled? I think we have to do the first, because we'll early S_FALSE out of StartPaint if we're disabled. I'll leave that call to @carlos-zamora ☺️

Filter based on _isEnabled

Notes in #16217 have the investigation. TL;DR: we'd always buffer text. Even if we're disabled (unfocused). When we're disabled, we'd _never_ clear the buffered text. Oops. Closes #16217

Notes in #16217 have the investigation. TL;DR: we'd always buffer text. Even if we're disabled (unfocused). When we're disabled, we'd _never_ clear the buffered text. Oops. Closes #16217 (cherry picked from commit d14524c) Service-Card-Id: 91033137 Service-Version: 1.18

Notes in #16217 have the investigation. TL;DR: we'd always buffer text. Even if we're disabled (unfocused). When we're disabled, we'd _never_ clear the buffered text. Oops. Closes #16217 (cherry picked from commit d14524c) Service-Card-Id: 91033138 Service-Version: 1.19

Silvenga added Issue-Bug It either shouldn't be doing this or needs an investigation. Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting labels Oct 23, 2023

microsoft-github-policy-service bot added the Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something label Oct 23, 2023

microsoft-github-policy-service bot added Needs-Attention The core contributors need to come back around and look at this ASAP. and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Oct 23, 2023

carlos-zamora added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Attention The core contributors need to come back around and look at this ASAP. labels Oct 24, 2023

microsoft-github-policy-service bot added Needs-Attention The core contributors need to come back around and look at this ASAP. and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Oct 24, 2023

zadjii-msft assigned carlos-zamora Oct 30, 2023

zadjii-msft added the Product-Terminal The new Windows Terminal. label Oct 30, 2023

zadjii-msft added this to the Terminal v1.20 milestone Oct 30, 2023

zadjii-msft mentioned this issue Nov 1, 2023

Severe Memory Leak, Possibly Related to UIA #16209

Closed

zadjii-msft added the Severity-Blocking We won't ship a release like this! No-siree. label Nov 1, 2023

zadjii-msft mentioned this issue Nov 1, 2023

Fix leak in buffering text for UIA when unfocused #16251

Merged

microsoft-github-policy-service bot added the In-PR This issue has a related PR label Nov 1, 2023

zadjii-msft unassigned carlos-zamora Nov 1, 2023

DHowett closed this as completed in #16251 Nov 10, 2023

microsoft-github-policy-service bot added the Needs-Tag-Fix Doesn't match tag requirements label Nov 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory Leak #16217

Memory Leak #16217

Silvenga commented Oct 23, 2023

zadjii-msft commented Oct 23, 2023

Silvenga commented Oct 23, 2023

carlos-zamora commented Oct 24, 2023

Silvenga commented Oct 24, 2023

Silvenga commented Oct 24, 2023

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023 •

edited

Loading

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

carlos-zamora commented Nov 1, 2023

Memory Leak #16217

Memory Leak #16217

Comments

Silvenga commented Oct 23, 2023

Windows Terminal version

Windows build number

Other Software

Steps to reproduce

Expected Behavior

Actual Behavior

zadjii-msft commented Oct 23, 2023

Silvenga commented Oct 23, 2023

carlos-zamora commented Oct 24, 2023

Silvenga commented Oct 24, 2023

Silvenga commented Oct 24, 2023

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023 • edited Loading

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

Silvenga commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023

carlos-zamora commented Nov 1, 2023

zadjii-msft commented Nov 1, 2023 •

edited

Loading