Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add htcondor group submission. #176

Merged
merged 7 commits into from
Jun 8, 2024
Merged

Conversation

riga
Copy link
Owner

@riga riga commented Jun 8, 2024

This PR mainly contains the group submission of htcondor jobs (detailed below). However, there are two smaller features that coming with this PR:

Smaller features

  1. law luigid sub command: The law configuration file can contain sections that synced with the internal luigi config (via naming sections [luigi_<section>]), which is a convenience to just maintain a single file. However, when starting the central scheduler with luigid, these sections are not considered. The new law luigid sub command loads and synchronizes the config and then starts the central scheduler, just like plain luigid would do.
  2. Interface to declare job resource as luigi's process_resources in remote workflows: Remote workflows like HTCondorWorkflow can now declare the resources that their jobs claim to the central luigi scheduler (if used). During status polling, these resources are freed eagerly so that new tasks relying on the same resources can start as soon as possible.

Main feature

So far, htcondor jobs prepared in separate job files, each with their own queue command (though, merged into a single submission file for faster interactions). However, since the central job interface also provides grouped jobs (such as used by the CMS-CRAB workflow interface), htcondor jobs can actually make use of that feature, too.

The change required a rather large restructuring of how the htcondor itself treats jobs during their creating and - obviously - how the HTCondorJobFileFactory creates its outputs.

There is a config - job::htcondor_job_grouping_submit, defaulting to True - which controls the submission type. Grouping is now the default. Setting this option to False will trigger the old behavior.


Requires a port to the release_prep branch.

@riga riga self-assigned this Jun 8, 2024
@riga riga merged commit c475ebb into master Jun 8, 2024
8 checks passed
@riga riga deleted the feature/htcondor_group_submission branch June 8, 2024 10:17
riga added a commit that referenced this pull request Jun 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant