Added loop support

[GlassOfWhiskey]

This commit adds the previous Loop requirement extension directly into the CWL v1.3 spec.

[GlassOfWhiskey]

The important question here is: do we still need loopSource or we can directly use the source field here?
I remember the problem was that the source field requires a step_name/output_name format for the argument, while the loopSource here requires just the output_name, as the step_name is always the current one.

I think that we can proceed in two ways: either modify the schema_salad to handle both behaviors, or keep the loopSource specific definition.

[mr-c]

Needs loop conformance tests, and needs all the tests to use cwlVersion: v1.3.0-dev1 everywhere except for the mixed versions tests which should be adapted with care.

For testing purposes you could also adjust the CI to use cwltool from your PR. We can revert that bit once there is a cwltool release with v1.3.0-dev1 support.

[GlassOfWhiskey]

Putting this PR in Draft state, as it depends on this one.

[mr-c]

Needs a changelog entry and ideally some tests that don't use InlineJavascriptRequirement

[mr-c]

Needs a changelog entry

This is still true

[tom-tan]

I have a simple question about loop feature.

Is it included as mandatory, or optional (e.g., do we need the LoopRequirement in the requirements field)?

I understand it is essentially needed for several applications.
However, IMO, it would be nice if it is defined as an optional feature with the following reasons:

• There exists many applications that do not need loops. Reqiring all the workflow engines to be implemented a loop feature needs too much effort.
• Making loops as optional makes easier to develop CWL compliant engines than making it as mandatory. By keeping the mandatory spec smaller and simple as possible, we can provide users with more choices for workflow engines.

[mr-c]

@tom-tan We could make it optional but without needing a LoopRequirement, like conditionals are in CWL v1.2: https://www.commonwl.org/v1.2/Workflow.html#Conditional_execution_(Optional)

[tetron]

My comment about "Could we expand the names to last_iteration_output and all_iteration_outputs" still applies, the rest looks good.

[mr-c]

please, also described here :-)

[GlassOfWhiskey]

This format is compliant to the scatterMetod field documented just above

[mr-c]

There was an error in how we were running the self-check (are all the CWL documents used in the conformance tests valid at the schema level) which I fixed in #46 and rebased into here

tests/loop/invalid-no-when.cwl is invalid at the CWL schema level on purpose, which causes that check to fail. I'm not against having it in the conformance tests, but we need a way to easily mark it so we don't do the schema validation on it.

1. Filename pattern; Alas invalid is also used in other conformance tests that are schema-salad compliant. Either we rename those files or find another word to stick in the CWL document filename
2. put schema-breaking tests in tests/invalid folder or similar

[GlassOfWhiskey]

I think the best idea would be to simply remove the test, as we can easily rely on SchemaSalad validation to assess this behaviour, without the need for a specific test

[mr-c]

Suggested change

	run: pip install "cwltest>=2.3" pytest-xdist "cwltool>=3.1.20240909164951"
	run: pip install "cwltest>=2.3" pytest-xdist "cwltool>=3.1.20241007082533"

[mr-c]

Please add a changelog, otherwise I approve!