How can resource limits of a process be configured?

[totycro]

I found the following example for a resource requirement setting:

- class: CommandLineTool
  id: crop-cl
  hints:
    ResourceRequirement:
      coresMax: 8
      ramMax: 24000

So i adapted the snuggs process to use this here.

Now when trying to start a job with this process, I see the following error message from the calrissian pod.

Is this a bug or do the resource limits have to be set differently?

wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian INFO calrissian 0.11.0-sprint1 (cwltool 3.1.20211004060744)
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian INFO Resolved '/workflow/input-data/wf-8c4f9d6a-c265-11ed-8349-4aa140051a59/tmpb6w12yat_wrapped.cwl#main' to 'file:///workflow/input-data/wf-8c4f9d6a-c265-11ed-8349-4aa140051a59/tmpb6w12yat_wrapped.cwl#main'
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian failed
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian Traceback (most recent call last):
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 946, in resolve_all
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 891, in resolve_all
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 594, in _resolve_idmap
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian schema_salad.exceptions.ValidationException: mapSubject 'ResourceRequirement' value 'None' is not a dict and does not have a mapPredicate.
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian ERROR Tool definition failed validation:
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian (140307817895872) (file:///workflow/input-data/wf-8c4f9d6a-c265-11ed-8349-4aa140051a59/tmpb6w12yat_wrapped.cwl) Validation error in position 2:
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   mapSubject 'ResourceRequirement' value 'None' is not a dict and does not have a mapPredicate.
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian Traceback (most recent call last):
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 946, in resolve_all
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 891, in resolve_all
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 594, in _resolve_idmap
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian schema_salad.exceptions.ValidationException: mapSubject 'ResourceRequirement' value 'None' is not a dict and does not have a mapPredicate.
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian 
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian During handling of the above exception, another exception occurred:
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian 
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian Traceback (most recent call last):
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 1038, in main
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian     skip_schemas=args.skip_schemas,
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "/usr/local/lib/python3.7/site-packages/cwltool/load_tool.py", line 383, in resolve_and_validate_document
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian     document_loader.resolve_all(workflowobj, fileuri)
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   File "schema_salad/ref_resolver.py", line 952, in resolve_all
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian schema_salad.exceptions.ValidationException: (140307817895872) (file:///workflow/input-data/wf-8c4f9d6a-c265-11ed-8349-4aa140051a59/tmpb6w12yat_wrapped.cwl) Validation error in position 2:
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian   mapSubject 'ResourceRequirement' value 'None' is not a dict and does not have a mapPredicate.
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian Starting Cleanup
wf-8c4f9d6a-c265-11ed-8349-4aa140051a59-cbjv7 calrissian Finishing Cleanup

Here's also the calrissian logfile directly, but it's basically the same thing:
snuggsresourcelimited-0-3-08c4f9d6a-c265-11ed-8349-4aa140051a59_0_calrissian.log

[fabricebrito]

I suggest you validate the CWL first with:

cwltool --validate https://raw.githubusercontent.com/totycro/fictional-potato/ades-test/workflow-resource-limited.cwl

which returns:

[2023-03-21 16:49:16] INFO /srv/conda/bin/cwltool 3.1.20221027224946
failed
Traceback (most recent call last):
  File "schema_salad/ref_resolver.py", line 961, in resolve_all
  File "schema_salad/ref_resolver.py", line 906, in resolve_all
  File "schema_salad/ref_resolver.py", line 594, in _resolve_idmap
schema_salad.exceptions.ValidationException: https://raw.githubusercontent.com/totycro/fictional-potato/ades-test/workflow-resource-limited.cwl:42:3: mapSubject
                                                                                                         'ResourceRequirement'
                                                                                                         value 'None' is not
                                                                                                         a dict and does not
                                                                                                         have a mapPredicate.
[2023-03-21 16:49:17] ERROR Tool definition failed validation:
(140372786519104) (https://raw.githubusercontent.com/totycro/fictional-potato/ades-test/workflow-resource-limited.cwl) Validation error in position 1:
https://raw.githubusercontent.com/totycro/fictional-potato/ades-test/workflow-resource-limited.cwl:42:3:   mapSubject
                                                                                                           'ResourceRequirement'
                                                                                                           value 'None' is not
                                                                                                           a dict and does not
                                                                                                           have a mapPredicate.

[totycro]

Thanks! There really was an indention error in the cwl, with a patched version I was now able to trigger a run.

It would be convenient for users to see validation errors right when a process is being deployed.

So in the running process, i now see e.g. a pod for step-1-2, and it has the following resource section:

      resources:
        requests:
          cpu: "1"
          memory: 256Mi

Is this derived from this specification in the cwl?

 - baseCommand: s-expression
  class: CommandLineTool

  id: clt

  hints:
    ResourceRequirement:
      coresMax: 1
      ramMax: 24000

The current resources section ensures that the pod has these resources as minimal requirements. Is there also a way of limiting the resources?

So far I have not found any further options in the documentation in the file 22.1116 ADES - Resource allocation - Google Docs.pdf

The patched complete cwl can be found here.

[fabricebrito]

Users can use cwltool to validate their CWL as shown above.

Your CWL declares

  requirements:
    EnvVarRequirement:
      envDef:
        PATH: /srv/conda/envs/env_app_snuggs/bin:/srv/conda/bin:/srv/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    ResourceRequirement: {}

and

  hints:
    ResourceRequirement:
      coresMax: 1
      ramMax: 24000

Remove the hints and define the ResourceRequirement mapping with the values.

Hope this helps

[totycro]

Ah, ok, the documentation suggests to use format:

cwlVersion: v1.0
$graph:
- class: Workflow
  id: s2-cropper
  ...
- class: CommandLineTool
  id: crop-cl
  hints:
    ResourceRequirement:
      coresMax: 8
      ramMax: 24000

However with your suggestion now it seems to really use the values as provided. Interestingly it now created only a step-1 pod and not one for step-1-2, and it has this resource section:

      resources:
        requests:
          cpu: "1"
          memory: 24000Mi

But it seems to do the processing on this one pod now. However it's not limited to these resources, to the pod could still use amount of resources.