cylc clean proposal

[oliver-sanders]

An outline of where we could go with the cylc clean command, very much open to ideas.

The current (Cylc7) approach to meddling with workflow files is a confusing trio of:

• rose suite-clean
• rose_prune
• rose_arch

These all have overlapping functionalities, they (ideally) should all:

• Locate workflow installations on different platforms (as required/configured).
• Have cycle awareness.
• Be able to perform cycle arithmetic.
• Be able to flush out files from within workflows

We wont get cylc clean up to rose_arch standards right away, nor should we, but we should be able to get some of the simpler cases out the way by early versions of Cylc8.

I would be keen for the development of a light-weight scaffold with plugins providing filetype support and archive functionality doing the heavy-lifting.

[dpmatthews]

I think a common use case is to want to to revert your run directory back to how it was after you ran cylc install. I think the command to do that would cylc clean myflow --rm log:work:share which seems OK to me. Maybe add this as an example?

[dpmatthews]

Just to note that, as discussed in cylc/cylc-flow#3887, we'll need a --force option to handle the case where the remote clean cannot be completed.

[dpmatthews]

I think I'm happy with the detail of everything targetted for 8.0 (--local, --remote, --rm, --mv).
I haven't worried too much about the detail of the later stuff but the ideas looks good.

[hjoliver]

Looks great to me.

[hjoliver]

Just to play devil's advocate: maybe this is somewhat overkill for housekeeping of workflow files - is there really much need for MIME type-specific plugins, for instance, when 99% (?) of use would be just to delete or compress and/or archive files by location, file extension, and cycle point. (Is the file type content really relevant here?)

[oliver-sanders]

MIME is overkill, really just suggesting that if we add "tar" support we do so via an interface which allows support for other types (or use on if it exists). MIME is just a cheap way of getting file-type association.

[oliver-sanders]

Just considering one final related use-case which is the transfer of files between platforms.

• Internally (e.g. remote_init)
• By users (e.g. to move data where it is needed)

Obviously not a priority ATM but worth ensuring the interface is compatible with this in future.

The utilities outlined here provide most of the framework for this, in particular the ability to find what platform a task has run on and to pick a host on the same filesystem where the data can be extracted. This is very hard for users to do, especially in a portable way.

Something like this:

# copy "somefile" from one platforma to platformb
$ cylc clean myflow \
  --scp share/cycle/1234/somefile platforma \
  --platform platformb

# copy "somefile" from the platform where 1234/foo ran (most recently) to the scheduler host
$ cylc clean myflow//1234/foo \
    --scp share/cycle/<cycle>/somefile \
    --platform=<scheduler>

# internal job log retrieval would look like this
$ cylc clean myflow//1234/mytask/* \
    --rsync log/job/<cycle>/<task>/<job>/ <scheduler>

# remote-init would look like this
$ cylc clean myflow \
    --scp=./ myplatform \
    --exclude='*' \
    --include=...

Would need a couple of subtle change to the above to fix. Also perhaps better to split into cylc clean and cylc sync (as two interfaces to the same functionality) for clarity.

[hjoliver]

We should get this merged, so that the proposal document is can be found easily.

[dpmatthews]

Let's get this merged.
Detail of initial changes looks fine.
Future stuff can be debated further later.