feat: add max_time_hr option for wall-time allocation (issue #334)
#335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jaamarks
force-pushed
the
issue334/wall-time-option
branch
2 times, most recently
from
f699252 to
c59205a
Compare
- This introduces an optional `max_time_hr` config parameter to limit the execution time for the time-intensive rules `population_level_ibd` and `sample_concordance_plink` - The memory allocation for the `population_level_ibd` rule is also improved so that it now scales dynamically with input file size.
jaamarks
force-pushed
the
issue334/wall-time-option
branch
from
c59205a to
9ef96ef
Compare
jaamarks
added a commit
that referenced
this pull request
Oct 3, 2024
feat: Apply ``max_time_hr`` to sample_level_ibd rule (issue #334) **What** - Apply the ``max_time_hr`` feature to the ``sample_level_ibd`` rule within the ``sample_qc.smk`` sub-workflow. - Update the ``workflow_params.py`` file to state all of the rules that ``max_time_hr`` affects. **Why** The ``sample_level_ibd`` rule is another time-intensive step in the pipeline, and can users have experienced timeout issues at this step when running it on large datasets with the standard time-allocation method. So this update ensures that if a user specifies ``max_time_hr`` in the ``config.yml``, it will allso apply to this rule. This is and extension of PR #335. Fixes #334
jaamarks
added a commit
that referenced
this pull request
Oct 3, 2024
feat: Apply ``max_time_hr`` to sample_level_ibd rule (issue #334) **What** - Apply the ``max_time_hr`` feature to the ``sample_level_ibd`` rule within the ``sample_qc.smk`` sub-workflow. - Update the ``workflow_params.py`` file to state all of the rules that ``max_time_hr`` affects. **Why** - The ``sample_level_ibd`` rule is another time-intensive step in the pipeline, and can users have experienced timeout issues at this step when running it on large datasets with the standard time-allocation method. So this update ensures that if a user specifies ``max_time_hr`` in the ``config.yml``, it will allso apply to this rule. This is an extension of PR #335 Fixes #334
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
This PR introduces a new optional configuration parameter,
max_time_hr. Users can specify an integer value representing the maximum wall-time allocation in hours. This feature specifically applies to the time-intensive rulessample_concordance_plinkandpopulation_level_ibd.When
max_time_hris set, these two rules will receive their maximum time allocation immediately. If not specified, wall-time allocation will follow the default patterns:{1: 10, 2: 48, 3: 96}forsample_concordance_plink{1: 8, 2: 24, 3: 48}forpopulation_level_ibdAdditionally, we improved the memory allocation for the
population_level_ibdrule to a data driven approach, similar to commit f672fba.Why
This update offers two key benefits. First, if a user has a smaller wall-time limit than the default allocations (e.g., 36 hours), jobs may fail if they do not complete on the first attempt. For instance, if the initial allocation for the
sample_concordance_plinkrule is 10 hours and the next attempt is 48 hours, which exceeds the maximum allowed allocation, the job will fail. The new configuration prevents this issue.Second, for users processing relatively large sample sizes (greater than 100K), the default 10-hour allocation for the
sample_concordance_plinkrule may be insufficient. By specifying a larger time allocation upfront, users can avoid the risk of failure due to wall-time limits, potentially saving time by reducing unnecessary retries.Fixes #334