Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flaky conformance test with cwltool: simple_simple_scatter #2003

Closed
mr-c opened this issue May 18, 2024 · 0 comments · Fixed by #2051
Closed

flaky conformance test with cwltool: simple_simple_scatter #2003

mr-c opened this issue May 18, 2024 · 0 comments · Fixed by #2051
Assignees

Comments

@mr-c
Copy link
Member

mr-c commented May 18, 2024

This has been failing randomly for over a year

https://github.com/common-workflow-language/cwltool/actions/runs/9138932694/job/25130670885#step:7:471

Tests are being run with the cwltest plugin for pytest:

PYTEST_EXTRA="--cwl-include simple_simple_scatter" bash conformance-test.sh
(possibly with SKIP_INSTALL=1 at the beginning if you are hacking on cwltool` and it is already installed)

which is roughly equivalent to

cwltest --test https://github.com/common-workflow-language/cwl-v1.2/raw/main/conformance_tests.yaml -s simple_simple_scatter -- --parallel

or more directly

cwltool --parallel https://github.com/common-workflow-language/cwl-v1.2/raw/main/tests/scatter/simple-simple-scatter.cwl https://github.com/common-workflow-language/cwl-v1.2/raw/main/tests/scatter/scatter-job.yml

Running any of the above produces a null output for me (which would fail the test) about 1 out of every 4 times.

CWL test execution failed. 
expected: {
    "result": [
        [
            "^1a$",
            "^2a$",
            "^3a$",
            "^4a$"
        ],
        [
            "^1b$",
            "^2b$",
            "^3b$",
            "^4b$"
        ],
        [
            "^1c$",
            "^2c$",
            "^3c$",
            "^4c$"
        ],
        [
            "^1d$",
            "^2d$",
            "^3d$",
            "^4d$"
        ]
    ]
got: [
    [
        "^1a$",
        "^2a$",
        "^3a$",
        "^4a$"
    ],
    [
        "^1b$",
        "^2b$",
        "^3b$",
        "^4b$"
    ],
    [
        "^1c$",
        "^2c$",
        "^3c$",
        "^4c$"
    ],
    null
]
caused by: expected: [
    "^1d$",
    "^2d$",
    "^3d$",
    "^4d$"
]
got: null
Test: job: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/scatter-job.yml
id: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/test-index.yaml#simple_simple_scatter
tool: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/simple-simple-scatter.cwl
output:
  result:
  - - ^1a$
    - ^2a$
    - ^3a$
    - ^4a$
  - - ^1b$
    - ^2b$
    - ^3b$
    - ^4b$
  - - ^1c$
    - ^2c$
    - ^3c$
    - ^4c$
  - - ^1d$
    - ^2d$
    - ^3d$
    - ^4d$
doc: Two level nested scatter
tags:
- workflow
- subworkflow
- scatter
- inline_javascript
- expression_tool
GlassOfWhiskey added a commit that referenced this issue Oct 8, 2024
This commit fixes #2003 by handling spurious, repeated callbacks of the
`receive_scatter_output` method of the `ReceiveScatterOutput` class. The
reason of multiple awakenings has not been investigated deeply, though.
In the future, a thorough examination of the `MultithreadedJobExecutor`
logic may be necessary.
@mr-c mr-c closed this as completed in #2051 Oct 8, 2024
mr-c added a commit that referenced this issue Oct 8, 2024
This commit fixes #2003 by handling spurious, repeated callbacks of the
`receive_scatter_output` method of the `ReceiveScatterOutput` class. The
reason of multiple awakenings has not been investigated deeply, though.
In the future, a thorough examination of the `MultithreadedJobExecutor`
logic may be necessary.

---------

Co-authored-by: Michael R. Crusoe <1330696+mr-c@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants