Avoid pulling existing Singularity image #177 #178

adrabent · 2022-10-21T16:57:53Z

This pull requests should allow cwltool or toil to make use of an already existing node_slim.sif image in CWL_SINGULARITY_CAHCE or in the current working directory.

When using Singularity and in case nodejs is not installed on your system this image is pulled from Docker every time. In case of offline processing and for often pull requests to Docker this should be avoided.

aroffringa

Thanks @adrabent ! This would for us be an important feature to have, as we're hitting the pull rate limit because of the node:slim container pulling happening many times.

I have some nit suggestions to make the code a bit more clear -- it is a bit hard to grasp what happens currently.

cwl_utils/sandboxjs.py

aroffringa · 2022-10-22T10:10:59Z

cwl_utils/sandboxjs.py

                        or len(dockerimgs.split("\n")) <= 1
                        or force_docker_pull
-                    ):
-                        # pull node:slim docker container
+                    ):                        # pull node:slim docker container


Did you accidentally move this comment to the previous row?

cwl_utils/sandboxjs.py

aroffringa

Looks good to me! I don't know if a core developer also wants to review this?

lgtm-com · 2022-10-25T12:20:39Z

This pull request introduces 1 alert when merging 61f83ed into 740615f - view on LGTM.com

new alerts:

1 for Syntax error

mr-c

Thank you @adrabent for this , and to @aroffringa for the reviews.

What is the testing plan for this change?

adrabent · 2022-10-26T11:43:33Z

Hi @mr-c
I have tested the changes on a supercomputing facility and on our workflow where we have encountered hitting the pull rate limit. CWL workflow now runs fine without pulling any Singularity images anymore and just uses the existing ones thanks to these changes

Nevertheless, I still encounter an error on the compute nodes which do not have any outbound internet access:

[2022-10-25T17:25:29+0200] [MainThread] [W] [cwltool] jshint process timed out
Traceback (most recent call last):
  File "/p/project/chtb00/htb006/software_new/envs/toil/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 3591, in main
    tool = cwltool.load_tool.make_tool(uri, loading_context)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/load_tool.py", line 619, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 989, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 54, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 108, in __init__
    self.make_workflow_step(
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 155, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 250, in __init__
    self.embedded_tool = load_tool(toolpath_object["run"], loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/load_tool.py", line 641, in load_tool
    return make_tool(uri, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/load_tool.py", line 619, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 989, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 54, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 108, in __init__
    self.make_workflow_step(
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 155, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/workflow.py", line 250, in __init__
    self.embedded_tool = load_tool(toolpath_object["run"], loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/load_tool.py", line 641, in load_tool
    return make_tool(uri, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/load_tool.py", line 619, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 986, in toil_make_tool
    return ToilCommandLineTool(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/command_line_tool.py", line 403, in __init__
    super().__init__(toolpath_object, loadingContext)
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/process.py", line 708, in __init__
    validate_js_expressions(
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/validate_js.py", line 270, in validate_js_expressions
    expression_errors, _ = jshint_js(
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/validate_js.py", line 184, in jshint_js
    dump_jshint_error()
  File "/p/project/chtb00/htb006/software_new/envs/toil/lib/python3.9/site-packages/cwltool/validate_js.py", line 175, in dump_jshint_error
    raise RuntimeError(
RuntimeError: jshint failed to run successfully
returncode: -1
stdout: ""

So still internet access is needed to run any CWL workflow using Javascript. On my supercomputer I can load nodejs as a module which also works on the offline compute nodes. So I guess the pulled node:slim container is somewhat incomplete?

Cheers,
Alex

tetron · 2022-10-26T13:22:05Z

The node:slim container requiring network access is very odd and certainly not intended, do you think you could track down what is happening?

adrabent · 2022-10-27T15:56:53Z

Dear @tetron,

I am not very familiar how these javascript checks in nodejs are actually working. But to my naive understanding the exeception arises again in sandboxjs.py. There the timeout occurs here (line 265 to 271):

        nodejs.poll()

        if nodejs.poll() not in (None, 0):
            if killed:
                returncode = -1
            else:
                returncode = nodejs.returncode
        else:
            returncode = 0

The killed == True value arises from here (line 210 - line 222):

        killed = []

        def terminate() -> None:
            """Kill the node process if it exceeds timeout limit."""
            try:
                killed.append(True)
                nodejs.kill()
            except OSError:
                pass

        timer = threading.Timer(timeout, terminate)
        timer.daemon = True
        timer.start()

Even if nodejs is not really using outgoing connections, but could it be it uses some local (network) protocols for the threading, which might be typically disabled on nodes in computing facilities for security reasons?

-Alex

adrabent · 2022-11-24T14:14:44Z

@mr-c
Any updates regarding this merge request?

mr-c · 2022-11-24T14:54:44Z

@adrabent Can you run make cleanup? (You may need to create & activate a virtualen and run make install-deps first)

That would fix https://github.com/common-workflow-language/cwl-utils/actions/runs/3496853174/jobs/5855251466#step:7:32

adrabent · 2022-11-28T14:19:15Z

@mr-c Done.

codecov · 2022-11-28T14:49:31Z

Codecov Report

Merging #178 (d81fb8a) into main (1dd8bfc) will increase coverage by 0.53%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##             main     #178      +/-   ##
==========================================
+ Coverage   37.30%   37.84%   +0.53%     
==========================================
  Files          27       27              
  Lines       22721    22730       +9     
  Branches     6435     6436       +1     
==========================================
+ Hits         8477     8602     +125     
+ Misses      12380    12249     -131     
- Partials     1864     1879      +15

Impacted Files	Coverage Δ
cwl_utils/sandboxjs.py	`61.14% <81.81%> (+36.75%)`	⬆️
cwl_utils/expression.py	`54.02% <0.00%> (+8.04%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

mr-c · 2022-11-28T17:47:50Z

Thank you @adrabent ! I'll go make a new cwl-utils release now

adrabent added 6 commits October 21, 2022 16:36

Singulariity: avoid hitting Docker pull rate limit

66ec346

When using Singularity and in case nodejs is not installed on your system this image is pulled from Docker every time. In case of offline processing and for often pull requests to Docker this should be avoided.

Update sandboxjs.py

0298762

Update sandboxjs.py

b9d6832

Update sandboxjs.py

edde07e

Update sandboxjs.py

49e2ab9

Update sandboxjs.py

fb7eb51

aroffringa suggested changes Oct 22, 2022

View reviewed changes

Polish code

d663b52

aroffringa approved these changes Oct 25, 2022

View reviewed changes

Merge branch 'main' into patch-1

61f83ed

adrabent added 2 commits October 25, 2022 12:29

Fix syntax error

cb04fcf

Bugfix: import glob

a0989ea

aroffringa approved these changes Oct 25, 2022

View reviewed changes

mr-c reviewed Oct 25, 2022

View reviewed changes

Merge branch 'main' into patch-1

42bc094

make cleanup

6baf383

Merge branch 'main' into patch-1

5b65481

mr-c and others added 3 commits November 28, 2022 15:56

Merge branch 'main' into patch-1

cf32cda

port back expr tests from cwltool

0257db4

add test for CWL_SINGULARITY_CACHE

d81fb8a

mr-c merged commit 6d0f89e into common-workflow-language:main Nov 28, 2022

adrabent deleted the patch-1 branch December 9, 2022 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid pulling existing Singularity image #177 #178

Avoid pulling existing Singularity image #177 #178

adrabent commented Oct 21, 2022

aroffringa left a comment

aroffringa Oct 22, 2022

aroffringa left a comment

lgtm-com bot commented Oct 25, 2022

mr-c left a comment

adrabent commented Oct 26, 2022

tetron commented Oct 26, 2022

adrabent commented Oct 27, 2022 •

edited

Loading

adrabent commented Nov 24, 2022

mr-c commented Nov 24, 2022 •

edited

Loading

adrabent commented Nov 28, 2022

codecov bot commented Nov 28, 2022 •

edited

Loading

mr-c commented Nov 28, 2022

Avoid pulling existing Singularity image #177 #178

Avoid pulling existing Singularity image #177 #178

Conversation

adrabent commented Oct 21, 2022

aroffringa left a comment

Choose a reason for hiding this comment

aroffringa Oct 22, 2022

Choose a reason for hiding this comment

aroffringa left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Oct 25, 2022

mr-c left a comment

Choose a reason for hiding this comment

adrabent commented Oct 26, 2022

tetron commented Oct 26, 2022

adrabent commented Oct 27, 2022 • edited Loading

adrabent commented Nov 24, 2022

mr-c commented Nov 24, 2022 • edited Loading

adrabent commented Nov 28, 2022

codecov bot commented Nov 28, 2022 • edited Loading

Codecov Report

mr-c commented Nov 28, 2022

adrabent commented Oct 27, 2022 •

edited

Loading

mr-c commented Nov 24, 2022 •

edited

Loading

codecov bot commented Nov 28, 2022 •

edited

Loading