You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With eb116a5 we added the following code to the sample_level_ibd rule in the sample_qc.smk file so that we could apply the max_time_hr workflow parameter to this rule:
But now we are experiencing issues with the mem_mb not being aggressive enough. A sample size of 110K quickly failed on multiple attempts because of memory issues. So we need to make this more aggressive.
The text was updated successfully, but these errors were encountered:
jaamarks
changed the title
Increase the rate at which memory is allocated for the sample_level_ibd rule
Increase the rate at which memory is allocated for the sample_level_ibd rule
Oct 3, 2024
The previous allocation was set in the genome rule in the plink.smk file at: PLINK_BIG_MEM = {1: 1024 * 4, 2: 1024 * 64, 3: 1024 * 250}
From looking at log files of previous successful runs of a 110K sample, the pipeline failed up until it allocated 256GB at which point it succeeded. The pipeline would fail very quickly (under a minute) if it was a memory issue.
We could implemented a max_mem_mb parameter in the config, and then apply the following approach for any rule that requires a significant amount of memory (like the sample_level_ibdrule):
jaamarks
changed the title
Increase the rate at which memory is allocated for the sample_level_ibd rule
implement max_mem_mb option in the config file to resolve memory issues withsample_level_ibd rule
Oct 3, 2024
With eb116a5 we added the following code to the
sample_level_ibd
rule in thesample_qc.smk
file so that we could apply themax_time_hr
workflow parameter to this rule:But now we are experiencing issues with the
mem_mb
not being aggressive enough. A sample size of 110K quickly failed on multiple attempts because of memory issues. So we need to make this more aggressive.The text was updated successfully, but these errors were encountered: