-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new config option to set max wall-time for cluster #334
Comments
feat: add `max_time_hr` option for wall-time allocation (issue #334)
This Also need to update the description in the |
feat: Apply ``max_time_hr`` to sample_level_ibd rule (issue #334) **What** - Apply the ``max_time_hr`` feature to the ``sample_level_ibd`` rule within the ``sample_qc.smk`` sub-workflow. - Update the ``workflow_params.py`` file to state all of the rules that ``max_time_hr`` affects. **Why** - The ``sample_level_ibd`` rule is another time-intensive step in the pipeline, and can users have experienced timeout issues at this step when running it on large datasets with the standard time-allocation method. So this update ensures that if a user specifies ``max_time_hr`` in the ``config.yml``, it will allso apply to this rule. This is an extension of PR #335 Fixes #334
Currently, the pipeline includes two resource-intensive rules that require significant time to execute.
In the subworkflow located at
src/cgr_gwas_qc/workflow/sub_workflows/sample_qc.smk
, thesample_concordance_plink
rule utilizes a dynamic time allocation strategy defined as:In the subworkflow at
src/cgr_gwas_qc/workflow/sub_workflows/subject_qc.smk
, the samesample_concordance_plink
rule has a different dynamic allocation strategy:To enhance user experience, we need to implement an option in the configuration file that allows users to set a maximum time limit for these rules. This will enable users to specify a cap on the time allocated to these rules, ensuring that if their cluster has a maximum time limit that is lower than the potential allocation, they can avoid errors related to time over-allocation.
The text was updated successfully, but these errors were encountered: