-
Notifications
You must be signed in to change notification settings - Fork 9.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM copying a file to memory in Blazor #55694
Comments
Google Chrome. |
The wasm address space for the MVP is 32 bits, that means that all of your assemblies, your runtime, any allocations, and all I/O buffering needs to fit into 2GB. To process large nettrace files you will likely need to parse the nettrace file incrementally and stay within the 2GB rss for Wasm data. There are wasm64 extensions that remove this limit but they are not broadly implemented at this time so we'd love to hear more about your use case. |
Ah I should have read more about wasm before starting this project :( The project is https://verdie-g.github.io/dotnet-event-viewer. A nettrace analyzer tool that could be an alternative to Perfview. Nettrace files can be quite large (e.g. a 20 minutes trace of GCAllocationTick on one my service weights 10 GB). When, a nettrace is loaded on that tool, it's indeed parsed incrementally, but the thing is that decompressing the data can do a 5x on the size. So obviously that 10GB trace wouldn't even fit in memory but that tool would still cover most of the use-cases if it could use all the available memory. |
Also I'm surprised the 2GB limit is reached there. The file is 1GB but what else could use the other GB? |
If wasm uses 32-bits addressing wouldn't the limit be 4 GiB ? 🤔 |
Apologies for the spam but I noticed another issue:
In a non-wasm environment I would expect the GC to be super aggressive to collect objects when the available memory is low. I'm concerned this is not the case here. |
Maybe we're not using the right Emscripten flags for 4 GiB support. Historically, Emscripten didn't support 4 GiB because it did things like signed bitshifts of pointers. Once Emscripten fixed those issues, they added a flag ( https://v8.dev/blog/4gb-wasm-memory Edit: It looks like we did add an |
Looking closer, there are additional issues. First it appears that the sample code assumes that every read fills the 64k buffer completely, with stream apis you always want to assume the amount read can be less than the entire buffer and track the position in the buffer until it is full. In your sample if the amount read is smaller than 64k you end up allocating another 64k buffer and appending the partially full buffer to the list. Second the API in question here BrowserFileStream is part of aspnetcore not runtime and not directly familiar with how it buffers the I/O. |
The file upload is just an example. The issue can be reproduced by just allocating empty buffers const int bufferSize = 64 * 1024;
const int oneGig = 1024 * 1024 * 1024;
List<byte[]> list = new();
for (int i = 0; i < oneGig / bufferSize; i += 1)
{
list.Add(new byte[bufferSize]);
}
The issue was moved from dotnet/aspnetcore. |
@mkArtakMSFT the issue is not solved, could you please reopen the thread |
or @lewing? |
Is there an existing issue for this?
Describe the bug
In a Blazor Wasm app, copying a 1GB file to memory will cause a OOM exception.
I'm aware that, according to https://learn.microsoft.com/en-us/aspnet/core/blazor/file-uploads?view=aspnetcore-8.0#file-uploads, copying to file in memory is a bad practice. In my exact use-case, I'm not directly copying the bytes, but I'm parsing a very large nettrace file for this project https://verdie-g.github.io/dotnet-event-viewer.
Expected Behavior
I would expect the browser to allow allocating 1GB on a 16GB machine.
Steps To Reproduce
dotnet new blazorwasm -n BlazorTest
Home.razor
withcd BlazorTest && dotnet run
File.WriteAllBytes("1gb.bin", new byte[1_000_000_000])
)Exceptions (if any)
When the OOM occurs in
BrowserFileStream.ReadAsync
but it can also occur in the byte array allocation. In that case it's a System.OOM exception.
.NET Version
9.0.100-preview.3.24204.13
The text was updated successfully, but these errors were encountered: