Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR mainly contains the group submission of htcondor jobs (detailed below). However, there are two smaller features that coming with this PR:
Smaller features
law luigid
sub command: The law configuration file can contain sections that synced with the internal luigi config (via naming sections[luigi_<section>]
), which is a convenience to just maintain a single file. However, when starting the central scheduler withluigid
, these sections are not considered. The newlaw luigid
sub command loads and synchronizes the config and then starts the central scheduler, just like plainluigid
would do.process_resources
in remote workflows: Remote workflows likeHTCondorWorkflow
can now declare the resources that their jobs claim to the central luigi scheduler (if used). During status polling, these resources are freed eagerly so that new tasks relying on the same resources can start as soon as possible.Main feature
So far, htcondor jobs prepared in separate job files, each with their own
queue
command (though, merged into a single submission file for faster interactions). However, since the central job interface also provides grouped jobs (such as used by the CMS-CRAB workflow interface), htcondor jobs can actually make use of that feature, too.The change required a rather large restructuring of how the htcondor itself treats jobs during their creating and - obviously - how the
HTCondorJobFileFactory
creates its outputs.There is a config -
job::htcondor_job_grouping_submit
, defaulting toTrue
- which controls the submission type. Grouping is now the default. Setting this option toFalse
will trigger the old behavior.Requires a port to the release_prep branch.