32Bit Processes - maximum usable memory #996
Comments
This has been explained around the internet already, eg here: https://stackoverflow.com/questions/7954781/whats-the-difference-between-working-set-and-commit-size Generally, keep in mind that on 32-bit, there's only a maximum of 2GB user space for 32-bit processes - or 3GB, if "large address aware" is enabled for an executable; however, the "Windows 32Bit" numbers seem to be consistent with it being disabled. Why does "Windows 32Bit + mimalloc 2.1.9" get 400MB less than "default"? "wasm32 emsdk 4.0.0" +"mimalloc 2.1.7" seems to be consistent with the 3GB limit I mentioned above. "wasm32 emsdk 4.0.0" +"default" is actually quite astonishing; it's almost 4GB, so more than should actually be supported by the OS. Maybe there's some stuff like using NUMA allocations happening behind the scenes? |
|
The funny thing is with Windows 32Bit, if I reserve and commit the memory ahead I still run into the problem. OS? The "OS" is the browser, this is 64Bit in an 64Bit OS. 🤔 |
Can you elaborate how you did that?
Ah, that's an important detail, and explains nicely why the ceiling is at about 4GB. On why mimalloc then fails much earlier, I can only speculate as well - maybe it's, again, mimalloc trying to get a chunk of memory that's "too large" compared to the available space. To eliminate as much guesswork as possible, it might be best to actually get some insight into what's mimalloc doing. |
|
Windows 32-Bit was just a test after seeing what happened in WebAssembly, to see if mimalloc behaves differently there.
I did the following: mi_option_enable( mi_option_verbose );
mi_option_enable( mi_option_disallow_os_alloc );
mi_option_enable( mi_option_limit_os_alloc );
mi_reserve_os_memory( ( 1024 * 1024 ) * 1790, true, true );
mi_option_set( mi_option_retry_on_oom, 0 );Output Windows 32Bit with reserved memoryOutput Windows 32Bit without special optionsIn this case it can shoot over the process memory limit, but should mimalloc consider this case and make sure as much of the limited memory is available as possible? WebAssembly - no special optionsIf I add up the reserved amount I should be 3GB so I do not know why the last allocation of 512MByte fails ... 🤔 mimalloc is much quicker in certain cases, if the tradeof is this overhead than this is explanation enough for me.
Sorry I should have started with that. 😅 |
Indeed. "Where does the memory go?" is what we should ask. (And since it applies in all the scenarios you tested, it's perhaps not unlikely the answer(s) are the same for both "native" Windows 32-bit and WebAssembly.) Here are some theories I can think of.
...but as I said, these are theories/guesses. And TBH, none of them feel like they could properly explain the difference in usable memory right now. And, it's also possible none of these are actually what's happening. ¯\_(ツ)_/¯ What to do now? Well, depends on how much further you want to go down the rabbit hole.
But, given there's quite a bit of information and ideas collected in this discussion, maybe it's time to ping @daanx and ask whether he might have some insights 😅 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
How much memory should I be able to use on a 32Bit process with mimalloc?
Given the simple example:
Is this expected?
Is there a way to improve on that?
If I check the process with System Informer on Windows when it runs out of memory I see this:
This issue is a result of this discussion: emscripten-core/emscripten#20936 (comment)
The text was updated successfully, but these errors were encountered: