Skip to content

Commit

Permalink
Document alternate run diretories.
Browse files Browse the repository at this point in the history
  • Loading branch information
hjoliver committed Aug 29, 2019
1 parent 3dd37ce commit 4393cb4
Showing 1 changed file with 52 additions and 0 deletions.
52 changes: 52 additions & 0 deletions src/running-suites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1571,3 +1571,55 @@ See the ``[suite servers]`` configuration section
such they can often trigger on time even if the suite is delayed to
the point that downstream tasks are late due to their dependence on
previous-cycle tasks that are delayed.
.. _Alternate Run Directories:
Alternate Run Directories
-------------------------
The ``cylc register`` command normally creates a run directory at
the standard location ``~/cylc-run/<WORKFLOW-NAME>/``. With the ``--run-dir``
option it can create the run directory at some other location, with a symlink
from ``~/cylc-run/<WORKFLOW-NAME>`` to allow access via the standard file path.
This may be useful for quick-running :ref:`Sub-Workflows` that generate large
numbers of files - you could put their run directories on fast local disk or
RAM disk, for performance and housekeeping reasons.
.. _Sub-Workflows:
Sub-Workflows
-------------
A single Cylc workflow can configure multiple cycling sequences in the graph,
but cycles can't be nested. If you need *cycles within cycles* - e.g. to
iterate over many files generated by each run of a cycling task - current
options are:
- parameterize the sub-cycles
- this is easy but it makes more tasks-per-cycle, which is the primary
determinant of workflow size and server program efficiency
- run a separate cycling workflow over the sub-cycle, inside a main-workflow
task, for each main-workflow cycle point - i.e. use **sub-workflows**
- this is very efficient, but monitoring and run-directory housekeeping may
be more difficult because it creates multiple workflows and run directories
Sub-workflows must be started with ``--no-detach`` so that the containing task
does not finish until the sub-workflow does, and they should be non-cycling
or have a ``final cycle point`` so they don't keep on running indefinitely.
Sub-workflow names should normally incorporate the main-workflow cycle point (use
``$CYLC_TASK_CYCLE_POINT`` in the ``cylc run`` command line to start the
sub-workflow), so that successive sub-workflows can run concurrently if necessary and
do not compete for the same run directory. This will generate a new sub-workflow
run directory for every main-workflow cycle point, so you may want to put
housekeeping tasks in the main workflow to extract the useful products from each
sub-workflow run and then delete the sub-workflow run directory.
For quick-running sub-workflows that generate large numbers of files, consider
using :ref:`Alternate Run Directories` for better performance and easier housekeeping.

0 comments on commit 4393cb4

Please sign in to comment.