cylc clean - initial implementation #3961

MetRonnie · 2020-11-24T12:41:55Z

These changes partially address #3887

Remove stopped workflows on the local scheduler filesystem

Implements a new command

$ cylc clean workflow_name

Note: Cleaning remote directories will come in a later PR.

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included (unit and functional).
Appropriate change log entry included.
Documentation will be handled by Document cylc clean cylc-doc#256
No dependency changes.

And prevent symlink attack

- Ensure passed "reg" is not a path that points to the cylc-run dir or higher - Ensure symlinks to be deleted have expected form i.e. `something/cylc-run/reg/log` etc

MetRonnie · 2020-11-24T12:53:10Z

Should the docs update come later as part of the 7-to-8 guide?

cylc/flow/scripts/clean.py

cylc/flow/suite_files.py

oliver-sanders · 2020-11-24T13:54:51Z

cylc/flow/suite_files.py

+            if os.path.isdir(target_top_parent):
+                remove_dir(target_top_parent)
+    run_dir_top_parent = get_suite_run_dir(Path(reg).parts[0])
+    remove_dir(run_dir_top_parent)


Yikes, we don't want to remove the entire hierarchy in the process of removing something a the bottom of the hierarchy.

For example here are two flows, a/b/c and a/d:

$ tree a a |-- b | `-- c | `-- suite.rc `-- d `-- suite.rc

If removing one flow, the other should remain:

$ cylc clean a/b/c $ tree a a `-- d `-- suite.rc

Though we can remove any empty dirs between the REG and ~/cylc-run:

$ cylc clean a/d $ tree a

Aside: We can extend this at some point to discover flows in a hierarchy using the suite detecting "scan" logic already implemented:

$ tree a a |-- b | `-- c | `-- suite.rc `-- d `-- suite.rc $ cylc clean a $ tree a -r

Both in the run dir and symlink dirs.

oliver-sanders · 2020-11-30T09:11:41Z

cylc/flow/suite_files.py

+    """
+    _validate_reg(reg)
+    reg = os.path.normpath(reg)
+    if reg.startswith('.'):


BTW the suite name validator has you covered here:

cylc-flow/cylc/flow/unicode_rules.py

Line 190 in 80bb050

not_starts_with(r'\.', r'\-'),

The difference is I'm checking after normalising the path, because you could have cylc clean foo/../.. which normalises to cylc clean .., i.e. rm $HOME. However, I guess I could change the order to

reg = os.path.normpath(reg) _validate_reg(reg)

I don't think we should accept path manipulations, though I'm unaware of a simple method to block them (testing for .. doesn't work for all cases).

Probs good enough to ensure that the flow exists within the cylc-run dir:

reg = Path(reg) reg.resolve() try: reg.relative_to(run_dir_path) except ValueError: raise Exception('Flow not found')

Anyway, not a problem for this PR.

oliver-sanders · 2020-11-30T11:01:35Z

cylc/flow/suite_files.py

+                raise WorkflowFilesError(
+                    f'Invalid Cylc symlink directory {path} -> {target}\n'
+                    f'Target is not a directory')
+            expected_end = str(Path('cylc-run', reg, name))


str(Path( 😆

oliver-sanders · 2020-11-30T11:02:39Z

tests/functional/cylc-clean/00-basic.t

+sed -i '$d' "${TEST_NAME}.stdout"
+# Note: backticks need to be escaped in the heredoc
+cmp_ok "${TEST_NAME}.stdout" << __TREE__
+${TEST_DIR}/sym-cycle


Nice human readable test.

datamel

I have read code, checked this branch out and run tests. I have one comment that needs addressing, other than that I'm pretty sure it is good to go!

datamel · 2020-11-30T12:41:38Z

tests/functional/cylc-clean/00-basic.t

+${TEST_DIR}/sym-log
+\`-- cylc-run
+    \`-- cylctb-${CYLC_TEST_TIME_INIT}
+        |-- f


These tests will fail when running tests using the functional directory name, rather than the f one.

This has been fixed.

oliver-sanders · 2020-11-30T16:56:29Z

tests/functional/cylc-clean/00-basic.t

@@ -62,23 +67,23 @@ cmp_ok "${TEST_NAME}.stdout" << __TREE__
 ${TEST_DIR}/sym-cycle
 \`-- cylc-run
    \`-- cylctb-${CYLC_TEST_TIME_INIT}
-        \`-- f
+        \`-- ${FUNCTIONAL_DIR}


Just realised that if another test gets added to this directory (i.e. 01-<name>.t) then this test will break.

I tried copying another test file into tests/functional/cylc-clean/01-something.t and 00-basic.t still works

If 01-something creates a workflow and the two tests are run together then 01-something will change the output of tree based on the order in which the tests are run.

I tried making a copy of 00-basic.t called 01-foo.t that has a slightly different expected tree (no symlink work dir or leave-me-alone dir) and both pass? Doesn't purge take care of removing the old stuff?

It does, but only once the test has passed but tests run in parallel.

Hopefully b35b10b prevents that

datamel

I'm happy with these changes. Thanks @MetRonnie!

MetRonnie added 5 commits November 17, 2020 19:57

Tidy

1e7724b

Create cylc clean command

cc54307

cylc clean: ensure the right dirs are removed

dc9683b

And prevent symlink attack

cylc clean & register: unified "reg" validation

62222f7

cylc clean: better validation

118d825

- Ensure passed "reg" is not a path that points to the cylc-run dir or higher - Ensure symlinks to be deleted have expected form i.e. `something/cylc-run/reg/log` etc

MetRonnie added this to the cylc-8.0.0 milestone Nov 24, 2020

MetRonnie requested review from oliver-sanders and datamel November 24, 2020 12:41

MetRonnie self-assigned this Nov 24, 2020

MetRonnie added 3 commits November 24, 2020 12:49

Unit tests for cylc clean & helpers

5abf13d

Fix cylc clean for nested suite name e.g. foo/bar/baz

411c335

Functional test for cylc clean

6413632

MetRonnie force-pushed the cylc-clean branch from 19d062d to 6413632 Compare November 24, 2020 12:51

MetRonnie commented Nov 24, 2020

View reviewed changes

cylc/flow/scripts/clean.py Show resolved Hide resolved

oliver-sanders reviewed Nov 24, 2020

View reviewed changes

MetRonnie marked this pull request as draft November 24, 2020 14:21

MetRonnie added 3 commits November 24, 2020 14:29

Make sure tree command is installed for functional test

2a05203

cylc clean: ensure workflow siblings are not removed

caab8b8

Both in the run dir and symlink dirs.

cylc clean: do not raise error if specified workflow doesn't exist

7c6b385

MetRonnie force-pushed the cylc-clean branch from 8d1ec14 to 7c6b385 Compare November 24, 2020 18:38

MetRonnie marked this pull request as ready for review November 25, 2020 11:01

oliver-sanders reviewed Nov 30, 2020

View reviewed changes

oliver-sanders approved these changes Nov 30, 2020

View reviewed changes

datamel reviewed Nov 30, 2020

View reviewed changes

Make cylc clean functional test less flaky

8c7e368

oliver-sanders reviewed Nov 30, 2020

View reviewed changes

MetRonnie added 2 commits December 1, 2020 12:13

Prevent cylc-clean functional test clashing

b35b10b

Update changelog

743d719

datamel approved these changes Dec 2, 2020

View reviewed changes

oliver-sanders merged commit eece5ac into cylc:master Dec 2, 2020

MetRonnie deleted the cylc-clean branch December 2, 2020 10:38

MetRonnie modified the milestones: cylc-8.0.0, cylc-8.0a3 Dec 2, 2020

MetRonnie linked an issue Dec 10, 2020 that may be closed by this pull request

cylc clean #3887

Open

MetRonnie mentioned this pull request Dec 11, 2020

cylc clean #3887

Open

hjoliver modified the milestones: cylc-8.0a3, cylc-8.0b0 Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cylc clean - initial implementation #3961

cylc clean - initial implementation #3961

MetRonnie commented Nov 24, 2020 •

edited

Loading

MetRonnie commented Nov 24, 2020 •

edited

Loading

oliver-sanders Nov 24, 2020

oliver-sanders Nov 30, 2020

MetRonnie Nov 30, 2020 •

edited

Loading

oliver-sanders Nov 30, 2020

oliver-sanders Nov 30, 2020

oliver-sanders Nov 30, 2020

datamel left a comment

datamel Nov 30, 2020

oliver-sanders Dec 1, 2020

oliver-sanders Nov 30, 2020

MetRonnie Nov 30, 2020

oliver-sanders Dec 1, 2020

MetRonnie Dec 1, 2020

oliver-sanders Dec 1, 2020

MetRonnie Dec 1, 2020

datamel left a comment

cylc clean - initial implementation #3961

cylc clean - initial implementation #3961

Conversation

MetRonnie commented Nov 24, 2020 • edited Loading

MetRonnie commented Nov 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MetRonnie Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datamel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datamel left a comment

Choose a reason for hiding this comment

MetRonnie commented Nov 24, 2020 •

edited

Loading

MetRonnie commented Nov 24, 2020 •

edited

Loading

MetRonnie Nov 30, 2020 •

edited

Loading