Workers out of memory during optimization runs #387
Comments
|
The new HFSsizing actor is taking a lot more memory than it used to... it has now the largest memory footprint of all actors: See for example https://github.com/ProjectTorreyPines/FUSE.jl/actions/runs/5742584439/job/15564988399 
This likely points to some type instability (which would also makes it slow) |
|
Ok, but in general why does the allocated memory increase with the number of iterations? Shouldn't we stop this ratcheting effect somehow? Maybe by re-setting all variables after each iteration? |
|
Are you requesting enough memory in the first place? |
|
Some more info. Compare these FUSE.timing outputs from the same Julia process (1218946) at the 1st, 2nd, and 56th iterations. By the last iteration, the total allocations are 1.37 TiB. 1st iteration (includes a lot of pre-compiling that is not repeated in subsequent iterations) 2nd iteration 56th iteration |
|
How about the 28th iteration, if you have it? This looks like a memory increase of 20-30 GB per iteration. Some sort of memory leak not captured by the garbage collector? |
|
28th iteration:
|
|
Indeed, some memory does not get freed. |
|
Note that in this case (which is typical for my optimization runs), there are 256 tasks being run in parallel on 4 Omega worker nodes. So whatever memory leakage is indicated in the above timing.txt files, multiply that by 256/4 = 64 to get the total amount for each node. |
|
I see some issues online related to garbage collection and multithreading, though the issues appear closed, for example, JuliaLang/julia#31923. Perhaps it would be worth trying a single thread on a single core to see what the performance in. It should only require a few iterations to see if the memory footprint is growing. |
|
@daveweisberg see 75089bc I see that you are now writing a FUSE.varinfo(FUSE, all=true, imported=true, recursive=true, sortby=:size, minsize=1024)this will contain an output similar to this (here shown with NOTE: You can now also do to see how memory gets used by your julia process """
plot_memory(memory::Memory, n_big_jumps::Int=5, ignore_first_seconds::Int=0)
Plot the memory usage over time from a `Memory` object.
# Arguments
- `n_big_jumps`: number of significant memory jumps to highlight in the plot.
- `ignore_first_seconds`: number of initial seconds to ignore in the plot. Memory usage will be plotted relative to the memory after this cutoff. Default is `0` (no seconds are ignored).
# Returns
A plot with the following characteristics:
- Time is displayed on the x-axis as delta seconds since the first recorded time (after ignoring the specified initial seconds).
- Memory usage is displayed on the y-axis in MB.
- Memory usage is shown as a scatter plot.
- The `n_big_jumps` largest jumps in memory are highlighted in red with annotations indicating the action causing each jump.
"""
plot(FUSE.memory, 5, 0)
I have not tried, but I suspect that you might be able to gather the memory history from each process and plot them this way: using Distributed
p = plot(FUSE.memory, 5) #Main process
memories = [@fetchfrom worker FUSE.memory for worker in workers()]
for memory in memories
plot!(memory, 5)
end
display(p) |
|
@orso82 Sorry but I don't have the time to help diagnose this issue. Can you please recreate this using my established optimization workflow? |
|
Ok. Let's say whoever gets there first. |
|
@orso82 New runs with the
These runs each have 60 generations, and completed without any memory errors. An on-going run with 180 generations can be found here:
|
|
@orso82 Update: The 180 generation run errored out 64% of the way to completion. I've moved the run directory to this location:
Error messages: |
|
@daveweisberg pointed me to this memtrace: I did some analysis on it and I think the memory leak is occurring in Solovev 
@lstagner could you please take a look and see there's anything obvious? I have been burned by Memoize myself in the past... perhaps this is the case again? |
|
I made sure all uses of Memoize limited the amount of instances it could save at once. If you think its caused by Memoize you can use |
|
OK, thank you for the good insight @lstagner ! @daveweisberg the next time you run, can you please try to set |
|
another thing to try is to periodically call |
|
@orso82 I've finished a new optimization run with It can be found here: |
|
Similar growth, but with an offset, which makes sense, since before we had a finite cache size of 5. 
Next, as @lstagner suggested, could you please try to add GC.gc()
ccall(:malloc_trim, Cvoid, (Cint,), 0)at the bottom of your workflow function that you feed to the |
|
@orso82 New run directories with both of Luke's suggestions implemented can be found here:
|
|
Better, but it's still increasing! 
|
|
@daveweisberg can you please try to setup a run where you do not execute the |
|
I have another memtrace of similar behavior of a TEQUIA & FluxMatcher run on SAGA. Here is the memtrace of the last case that was run: |
|
plot of the memtrace |
I am not sure if it only Solovev as I see the same problem for running with TEQULIA (although of course it could be both) |
|
My conclusion on how the memory leakage is happening relates to the garbage collector being unable to do it's job in a pmap while also being in a try-catch structure... see 5d8df73 for the fix |
@TimSlendebroek can you please try calling |
|
The issue has gotten better, but is not fully resolved. It would be great to have @mbolognaJH and @sjkelly take a look at the issue. |
|
This flag can be used to force Julia to collect at a greater interval: |
|
I'll add the minimal working example
But also add the optimization case for
Memory goes up directly with the number of dd's processed |
|
How do we pass
May be able to find workarounds for |
|
I made an example notebook that runs locally and shows the same problems (without wrapping and gc) @mbolognaJH (don't run the last two cells as this is the optimization it's already showing memory issues with simple load and save):
|
|
@TimSlendebroek is there a specific dd.json file I should be using? Maybe I'm missing part of my setup for this example. |
|
Aah sorry @mbolognaJH I can send you one or you can do the following to create your own json: using FUSE
dd, ini,act = FUSE.init(:KDEMO)
IMAS.json2imas(dd,<your path>) |
|
@mbolognaJH can we add each other on zoom/discord so I can respond a bit quicker if you have questions / explain / send files? |
|
Yes sure thing, we can continue the conversation on Zoom (I haven't tried it before, but we'll see how it works): my email is michael.bologna@juliahub.com |
|
I have been able to reproduce the MWE. By limiting the heap size available to the Julia GC with The root cause of this is the multi-threading and multi-processing design patterns used by FUSE. Currently, FUSE uses multi-processing exclusively via By adding The recommend long-term solution, that will also free up memory overhead and improve performance, would be to use multi-threading within each node as documented here. https://docs.julialang.org/en/v1.9/manual/multi-threading/ |
|
Thanks @mbolognaJH , this makes sense to me. I've been looking for how to implement your long-term solution, but I can't find a good example. How can we launch one process per node that has access to all the cores/threads on that node? |
|
You can use the For example: # Although you won't need the heap-size-hint if you're doing the long-term solution,
# but here's how to do multiple exeflags in one `addprocs()` command.
num_procs = 1
num_threads = 8 # You can use "auto" here as well
addprocs(num_procs, exeflags=["--threads=$num_threads", "--heap-size-hint=2G"]) |
|
Thank you @mbolognaJH ! This solves a major headache!!! @bclyons12 @mbolognaJH I consider this solved now. I'll add the Please don't spend further time on this now. I have more pressing questions for both of you @bclyons12 and @mbolognaJH Thanks again!!!! |
|
I still think there are very good reasons to figure out how to do the hybrid multiprocessing-multithreading implementation. I understand that it's not a priority for now, but we should open a separate issue for a future upgrade if we're going to close this one. |
|
@TimSlendebroek reported that the
@daveweisberg please note that as part of d7c57a5 now by default Thanks again @mbolognaJH ! |
on local with 2G hintsize
|
|
Thank you @TimSlendebroek for the thorough testing!! |
|
Thanks everyone for the hard work in solving this issue! I've started testing this with Julia 1.10 on the Omega cluster, and so far it looks good! A quick question for @TimSlendebroek or @orso82: Should I still include the following in the function I'm running on the distributed workers? Or is this no longer necessary? |
I have been seeing
srun error: Out Of Memoryon FUSE optimization runs that last longer than ~60 iterations. I don't understand why a worker node would be running out of memory, but we need to identify which process is causing this. It is preventing me from efficiently completing optimization runs that require 100s of iterations.The following error corresponds to the optimization run located here:
/fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/playground/weisberg_FPP_opt/nominal/opt_betaN_cost__Solovev_Kr_flattop48_HTS0.1_qpol2.75_A3.5_Zeff2.0/where the last saved run folder is from iteration 56.
Error:
The text was updated successfully, but these errors were encountered: