Skip to content
This repository has been archived by the owner on Apr 19, 2023. It is now read-only.

[BUG] compute resources labels not correctly applied to some processes #323

Closed
cflerin opened this issue Mar 22, 2021 · 1 comment
Closed
Labels
bug Something isn't working

Comments

@cflerin
Copy link
Member

cflerin commented Mar 22, 2021

Describe the bug
The default compute process label (defined with a glob matching part of the label) incorrectly gets applied to certain processes, overwriting the intended settings.

To Reproduce
Steps to reproduce the behavior:

Using a process with a label:

label 'compute_resources__cpu','compute_resources__24hqueue'

and using this section of the config file for the process labels:

process {
   executor = 'local'
   withLabel:'compute_resources__.*|compute_resources__default' {
      cpus = 2
      memory = '60 GB'
      time = '1h'
      clusterOptions = '-A cluster_account'
   }
   withLabel:compute_resources__minimal {
      cpus = 1
      memory = '1 GB'
   }
   withLabel:compute_resources__mem {
      cpus = 4
      memory = '60 GB'
   }
   withLabel:compute_resources__cpu {
      cpus = 20
      memory = '60 GB'
   }
   withLabel:compute_resources__report {
      maxForks = 2
      cpus = 1
      memory = '60 GB'
   }
   withLabel:compute_resources__24hqueue {
      time = '24h'
   }
   withLabel:compute_resources__bwa_mem {
      executor = 'local'
      cpus = 20
      memory = '60 GB'
      time = '24h'
      maxForks = 8
   }
}

When run, this process only gets assigned 2 cpus instead of the intended 20. Looking at the .nextflow.log it seems that the second label in the process definition (compute_resources__24hqueue) causes the default label to match a second time and remove the compute_resources__cpu setttings:

Mar-18 10:12:26.351 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__.*|compute_resources__default` matches label `compute_resources__cpu` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK
Mar-18 10:12:26.352 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__cpu` matches label `compute_resources__cpu` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK
Mar-18 10:12:26.352 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__.*|compute_resources__default` matches label `compute_resources__24hqueue` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK
Mar-18 10:12:26.352 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__24hqueue` matches label `compute_resources__24hqueue` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK

Expected behavior
Keep the expected compute settings in the process.

Screenshots
N/A

Please complete the following information:

  • OS: CentOS Linux release 7.9.2009
  • Nextflow Version: 20.11.0-edge
  • vsn-pipelines Version: develop branch from 0.25.0

Additional context
N/A

@cflerin cflerin added the bug Something isn't working label Mar 22, 2021
@cflerin
Copy link
Member Author

cflerin commented Mar 22, 2021

This can be resolved by removing the compute_resources__.*| default label from the config file, in which case the .nextflow.log shows the proper label matching:

Mar-22 11:22:09.360 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__cpu` matches label `compute_resources__cpu` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK
Mar-22 11:22:09.360 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:compute_resources__24hqueue` matches label `compute_resources__24hqueue` for process with name atac_preprocess_bap:ATAC_PREPROCESS:BWA_MAPPING_PE:MARK_DUPLICATES_SPARK

Proposed solution:
Move the default directives outside of any label, these will then be applied to all processes, but will still allow specific config labels applied to override the defaults. This leaves the compute_resources__default label, which I think should still be kept in case a user needs to apply any further options. Since it can't be empty, I would keep the time setting here (although it could be something else). This would end up like:

process {
   executor = 'local'
   cpus = 2
   memory = '60 GB'
   clusterOptions = '-A lp_symbiosys'
   withLabel:compute_resources__default {
      time = '1h'
   }
   withLabel:compute_resources__minimal {
      cpus = 1
      memory = '1 GB'
   }
   withLabel:compute_resources__mem {
      cpus = 4
      memory = '60 GB'
   }
   withLabel:compute_resources__cpu {
      cpus = 8
      memory = '60 GB'
   }
[...]
}

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant