[help] Performance bottleneck with crew_controller_local() and ~32k dynamic branches

[philiporlando]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm experiencing some performance bottlenecks when trying to use crew_controller_local() on a target with ~32k dynamic branches. My workflow involves downloading ~32k source files from an AWS S3 bucket and tracking changes using format = "file".

The main issue is that RAM usage on my server gradually accumulates over time, which eventually slows down the processing to the point where running everything single-threaded (without a controller) is actually faster. Each individual branch runs fairly efficiently, completing in around 0.2 seconds, with minimal CPU or memory usage per branch, so I suspect the overall RAM consumption is related to overhead from managing such a large number of dynamic branches with {crew}.

To mitigate the issue, I've enabled garbage_collection = TRUE in crew_controller_local(), tar_option_set(), and tar_config_set(), but the RAM buildup problem persists.

I'm aware of the overhead concerns related to creating numerous targets, but in this case, tracking changes for each individual S3 objects and local file is critical, making it difficult to abstract the work into fewer, higher-level targets as recommended here:

Dynamic branching makes it easy to create many targets. Unfortunately, if the number of targets exceeds several thousand, overhead may build up and the package may slow down. Temporary workarounds can avoid overhead in specific cases: for example, the shortcut argument of tar_make(), and choosing a pattern like slice() or head() instead of a full map(). But to minimize overhead at scale, it is better to accomplish the same amount of work with a fewer number of targets. In other words, do more work inside each dynamic branch.

Has anyone else experienced similar memory issues with large-scale dynamic branching or found effective strategies to manage memory consumption in this context? Any suggestions or insights would be greatly appreciated!

---

This issue is similar to #1329, but since I’m not using Slurm, I thought it might be worth creating a separate post. Feel free to close this if you believe it’s more appropriate to address in the existing thread.

[wlandau]

Would you try recording memory/cpu logs as described at https://wlandau.github.io/crew/articles/logging.html#in-targets? I think this could help with #1329 as well.

If it's a memory buildup in the callr process or worker processes, it may have something to do with targets or crew. If it's the dispatcher process, then it could be something to do with mirai or the way crew uses it. https://github.com/wlandau/mirai/actions/runs/11445620182/job/31843064157#step:6:1431 may point to a potential memory leak, it's hard to say because the 6 KB in that small test are only "possibly lost".

[wlandau]

In #1349, I saw the local targets callr process steadily increase in memory consumption. If you use autometric to watch memory consumption, is this what you see too?

[wlandau]

The slowness you may be seeing may not have to do with memory (c.f. #1349). Garbage collection via gc() is really slow. I recommend taking a look at the new documentation of the garbage_collection option in tar_option_set() in development targets, which can now take a non-negative integer to control how often garbage collection happens. The default value is 1000 so gc() runs every 1000 targets.

[philiporlando]

Thanks, Will! I'll take a closer look at everything over the next few days and report back soon. I should mention that I was encountering performance issues even before trying to implement any garbage collection. The first few branches seem to dispatch and complete within about 0.2 seconds, but the process slows down dramatically once the server hits its memory limit. I'm hoping {autometric} will help diagnose the problem further.

[wlandau]

Thanks. And if you have a reproducible example you can share, please let me know. I hit a dead end with my own reprexes in #1352.

[philiporlando]

I had some time to take a closer look, and first off, thank you so much for going above and beyond with the troubleshooting effort in #1352. It's been really helpful to get a better understanding of the performance issues, even if there isn't a clear solution for the memory consumption problem.

Unfortunately, I can't share my actual pipeline because it contains proprietary data, but the reprex you created is an even better version of what I had hoped to provide. I’m still planning to diagnose things with {autometric} and will report back soon, though it might take me a couple of days.

[venpopov]

I just experienced this with a relatively simple case, where running everything sequentially took ~1 min, but using crew_controller_local() with 10 workers took 16 minutes and the parent R process exploded to 71GB RAM. Seems like a memory leak somewhere. I’ll put up a reprex in a bit

Edit: posted it here

[wlandau]

You mentioned in #1352 (reply in thread) that development targets fixed a similar issue. Does targets >= 1.9.0 reduce memory usage here too?

NB crew and crew.cluster have recent updates on CRAN.

[venpopov]

Yes, I was referring to the same issue, sorry for posting it in two places - I first found this thread, then while I was preparing the reprex found the thread about memory issues so I posted there. The slow down was due to the memory leak - as RAM use exploded, memory was getting swapped in and out which lead to massive lags in initiating and closing workers. Everything works super smooth now. And it was definitely targets that made the difference - I ran it a renv environment with the only difference being Targets 1.8 vs the development version.