Large number of memfd:doublemapper (deleted) entries #89776
Comments
dotnet-issue-labeler
bot
added
the
needs-area-label
ghost
added
the
vcsjones
added
area-VM-coreclr
and removed
needs-area-label
|
@ayende that's an ifndef - so the doublemapper only deallocates on Linux. It's called from the destructor of |
|
I'm sorry, didn't realize that this was Is there an expectation that this will grow without limit? What sort of reflection would cause this? |
|
Actually, even in the lambda case of capturing context, I expect allocations of a managed object, but not jitting of new objects. Essentially, I expect tiering, loading, and some debugger operations to cause RX -> RW paging. You can use dotnet counters to see if jitting method count increases. I am not sure what's causing the growth in this case. @janvorli, does this count against the real memory usage? Any ideas what might be contributing to this? I thought since the mapping is deleted it becomes free for the process. I do expect it to count against the max_map_count though. |
|
The shared memory that is visible as |
|
Hi, It is going up & down a bit but appears to be mostly stable. Without this flag, we are seeing a lot more mapping, and they are always increasing. My expectation that with |
|
We are currently facing a similar issue, where we found runtime crashes like this: This seems to be the case because we run into the default maximum number of memory maps I wanted to find out why this is happening and by examining $ cat /proc/1/maps | grep doublemapper | wc -l
59588
$ cat /proc/1/maps | wc -l
63878 |
|
We also hit the limit on the # of maps recently in production. I'm not sure how to point a finger, but it absolutely feels like there is a leak here. |
|
We looked in more detail on the `` |
|
@ayende I wonder if it would be possible to run your app with and without W^X enabled for about the same time and then share the |
|
AWS: c5.xlarge In the browser:
** Note: That is a very large file. What happens under the covers is that we import a lot of data into RavenDB. $ cat /proc/$(pidof Raven.Server)/smaps | grep memfd | wc -l We see a very rapid growth to ~2,600 memfd items Here is all the maps here: Output from I tried running: Gave this output: I then killed the RavenDB process: I deleted and re-created the Test database and then: Obviously, there are no I'm adding the full maps from two times, so you can see this over time (without And here is the smaps output for both modes: |
|
I'm currently try to understand why in our application, the process memory (full memory dump) is way bigger, than what we "use" (details see: https://stackoverflow.com/questions/77023695/missmatch-between-expected-memory-size-of-an-dotnet-application-and-real-consume ) So today I checked |
|
Yes, we are also seeing some weirdness around that. |
|
I just tested this on .NET 8.0 RC1, we are seeing (still) an increase in the number of I can reproduce this quite easily:
I'm getting some results from this: I can't get symbols from Running with: |
|
@ayende |
|
Yes, I tried running that, it didn't seem to matter in terms of strace, it did work with lldb, I think, but couldn't get the breakpoint to hit. |
Interestingly, I've seen the symbols both working and not working with strace on the same Ubuntu 22.04 (except that one was in WSL2 - that didn't work, and the other in a docker container - which worked). I am currently trying to figure out what makes it different, since I need to get it working for an investigation I am doing. |
|
@janvorli seeing something similar on our application, over >16k We do generate assemblies at runtime using Microsoft.CodeAnalysis.Scripting.Script, is this known for leaking memory like in #80580? Possibly related: dotnet/roslyn#52217 and dotnet/roslyn#41722 |
|
@theolivenbaum this is not a leak, the number of allocations of regions marked with /memfd:doublemapper are expected to grow when runtime compiles more and more code. If this code is not inside of a collectible AssemblyLoadContext and thus it is not unloadable, this stuff is not freed either. |
Thanks! I'm changing our code to use AssemblyLoadContext, I'll check again in a week to see how it behaves |
|
How's it going ? The grafana shows lots of memory is cache. The result of I created a dump file using
The number of |
|
@stg609 would you be able to share the dump? Unless you prefer a different way, you can share the dump with Microsoft by opening a feedback issue at https://developercommunity.visualstudio.com/, attaching the dump to it (the attachments are not publicly visible), and sharing the feedback issue ID here. |
Yes, I have attached the dump file in the comment. see https://developercommunity.visualstudio.com/t/Many-doublemapper-Images-result-in-OOMKi/10827328#T-N10827379 |
|
@stg609 I am sorry it took me quite some time to investigate your dump. Some of the numbers reported by the !maddress don't make sense. For example, it says that System.Private.CoreLib.dll has several mappings of size around 64MB and 332.67MB in total, while in reality, the size of the whole file is about 12MB. In the end, I have written a windbg script in javascript to dump all blocks managed by the executable memory allocator and got this result:
I don't have any idea why there are so many free blocks. Many blocks on the free list have sizes that are used very often, so I don't see why they wouldn't be reused. Would it be possible for you to take two dumps at two different time points of the same process while the memory consumption is growing so that I can check the trend in these numbers? |
Description
When looking into our process, we noticed a large number of entries like this:
The process has been running for about 8 hours, and we have:
That number is not stable and grows over time, but we are just now loading data into the system, not yet trying to stress it.
Looking at the code, I found the source of that here:
runtime/src/coreclr/minipal/Unix/doublemapping.cpp
Line 63 in bd84336
But looking at where this freed, I see:
runtime/src/coreclr/minipal/Unix/doublemapping.cpp
Line 99 in bd84336
This looks like this will only actually be freed on MaxOS, and not on Linux?
FWIW, I couldn't find where this is called.
Is this number of entries expected? Should we monitor this value?
I understand that this is related to the way the JIT allocate memory?
Related, but we are seeing a large increase in memory usage in some production systems, which is not seen in .NET 6.0 but very noticeable in .NET 7.0
I noticed this:
#80580
And we are investigating whatever we do a lot of dynamic assembly generation (so far we don't think so, but can't rule it out).
Reproduction Steps
When I started writing this post, I had:
By the time I got here, I had:
So I certainly think that there is something that work here.
Note that at this point, the process in question was running for hours, basically in a big loop. So there should be no change in behavior nor would I expect it to run any JIT tiering or some such.
Expected behavior
Not have the runtime allocate indefinitely memory
Actual behavior
We are seeing additional memory mapping over time
Regression?
Yes, we aren't seeing that in .NET 6.0
Known Workarounds
No response
Configuration
No response
Other information
No response
The text was updated successfully, but these errors were encountered: