cylc clean 2: remote clean #4017

MetRonnie · 2020-12-24T13:13:02Z

These changes partially address #3887

Remove stopped workflows on all filesystems

Clean on remote hosts in addition to local filesystem. This is done by getting the platforms from the workflow database and then looking up the install targets for those platforms from global.cylc.

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included (unit and functional).
Appropriate change log entry included.
Documentation will be handled by Document cylc clean cylc-doc#256
No dependency changes.

until all platforms for that install target have been exhausted

- Ensure DB is closed using a try...finally statement in a couple of places where it was lacking - Refactor checking of the DB compatibility slightly

Fixes a bug where if log and share symlink dirs were the same, but share/cycle was different, then share/cycle's target wouldn't get removed

Plus: in CylcSuiteDAO, make db_file_name arg non-optional, as it will raise an exception anyway if not given

MetRonnie · 2020-12-24T15:36:17Z

tests/f/cylc-clean/01-remote.t .. skipped: 'tree' command not available on remote host _remote_background_indep_tcp

@oliver-sanders tree is installed on the GitHub Actions runner but not on the remote host. Is there a way to install tree on the host? (I take it this is to do with Docker?)

cylc/flow/rundb.py

cylc/flow/suite_files.py

Move initial checking of whether a workflow can be cleaned to its own function

oliver-sanders · 2021-01-05T15:15:32Z

Will push later, but still need to figure out how to get the remote functional test to run on GH Actions: #4017 (comment)

If you need the tree command installed on the remote platform (so you can ssh <host> tree <dir>) then this diff aught to do it:

diff --git a/dockerfiles/cylc-dev/Dockerfile b/dockerfiles/cylc-dev/Dockerfile
index 257dd9e70..dee4f4914 100644
--- a/dockerfiles/cylc-dev/Dockerfile
+++ b/dockerfiles/cylc-dev/Dockerfile
@@ -33,7 +33,7 @@ COPY "$CYLC_FLOW_DIR" "cylc"
 RUN apt-get update && \
     # build-deps: build-essential
     # run deps: procps, rsync
-    apt-get -qq -y install build-essential procps rsync && \
+    apt-get -qq -y install build-essential procps rsync tree && \
     # install conda stuff
     conda init bash && \
     . ./usr/local/etc/profile.d/conda.sh && \

MetRonnie · 2021-01-05T16:49:09Z

tests/f/cylc-clean/01-remote.t is passing now on Actions

wxtim

Add myself as a reviewer. GH broken.

oliver-sanders

Looks good, no problems in testing.

The logging is a little odd, the "Removing directory" notices come through as warnings, looks like I get one per install-target but no information as to where it is coming from.

$ cylc clean generic
2021-01-06T15:46:21Z INFO - Cleaning on install target: _remote_background_indep_poll
	(platform: _remote_background_indep_poll)
2021-01-06T15:46:21Z INFO - Cleaning on install target: _remote_background_indep_tcp
	(platform: _remote_background_indep_tcp)
2021-01-06T15:46:21Z WARNING - 2021-01-06T15:46:21Z INFO - Removing directory:
	/root/cylc-run/generic
	
2021-01-06T15:46:21Z WARNING - 2021-01-06T15:46:21Z INFO - Removing directory:
	/root/cylc-run/generic
	
2021-01-06T15:46:21Z INFO - Removing directory: /Users/oliver/cylc-run/generic

We will probably want to clean multiple platforms in parallel for 8.0.0 so reporting remote logging is tricky, can just log in the event of error.

CHANGES.md

cylc/flow/scripts/clean.py

cylc/flow/suite_files.py

cylc/flow/scripts/clean.py

plus 1 or 2 other code review suggestions

wxtim

Read code - Nothing obvious stands out, minor comments.
Read tests.
Checked-out and poked at. Satisfied it works in the desired way with some of my test workflows.

oliver-sanders · 2021-01-11T13:30:57Z

I've noticed too on exvcylcdev that everything that comes back from the Popen subprocess is in stderr instead of stdout for some reason, so it gets logged as a warning.

Hmmm, what does the ret-code say? Would be good to include that in the logging (note we can use the debug level).

MetRonnie · 2021-01-12T10:54:04Z

ret_code is 0 but I've noticed that the LOG.info gets added to the bottom of the blurb in err:

\nWelcome to the Met Office\n\nWARNING: exvcylcdev.......in accordance with the Monitoring policy.\n\n2021-01-12T10:49:59Z INFO - Removing directory: ~/cylc-run/temp5\n

cylc/flow/suite_files.py

oliver-sanders · 2021-01-12T13:37:46Z

If out and error are coming out funny might be worth checking the function that constructs the SSH command. You may need to tell it to capture/return stdout/err.

MetRonnie · 2021-01-12T18:29:39Z

Turns out LOG.info() always ends up in stderr! E.g. try

cylc run --no-detach my_flow > /dev/null

and all the logging will still get printed, or try

cylc run --no-detach my_flow 2> /dev/null

and none of the logging will get printed

By default, no destination is set for any logging messages. You can specify a destination (such as console or file) by using basicConfig() as in the tutorial examples. If you call the functions debug(), info(), warning(), error() and critical(), they will check to see if no destination is set; and if one is not set, they will set a destination of the console (sys.stderr) and a default format for the displayed message before delegating to the root logger to do the actual message output.

from https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial

MetRonnie · 2021-01-13T13:36:42Z

More on this:

cylc-flow/cylc/flow/option_parsers.py

Lines 253 to 271 in 500a7ba

    
           # Set up stream logging for CLI. Note: 
        
           # 1. On choosing STDERR: Log messages are diagnostics, so STDERR is the 
        
           #    better choice for the logging stream. This allows us to use STDOUT 
        
           #    for verbosity agnostic outputs. 
        
           # 2. Suite server programs will remove this handler when it becomes a 
        
           #    daemon. 
        
           if options.debug or options.verbose: 
        
               LOG.setLevel(logging.DEBUG) 
        
           else: 
        
               LOG.setLevel(logging.INFO) 
        
           # Remove NullHandler before add the StreamHandler 
        
           RSYNC_LOG.setLevel(logging.INFO) 
        
           while LOG.handlers: 
        
               LOG.handlers[0].close() 
        
               LOG.removeHandler(LOG.handlers[0]) 
        
           errhandler = logging.StreamHandler(sys.stderr) 
        
           errhandler.setFormatter(CylcLogFormatter( 
        
               timestamp=options.log_timestamp)) 
        
           LOG.addHandler(errhandler)

hjoliver · 2021-01-13T18:59:06Z

# 1. On choosing STDERR: Log messages are diagnostics, so STDERR is the 
#    better choice for the logging stream. This allows us to use STDOUT 
#    for verbosity agnostic outputs.

That's correct, IMO. (Although I had kind of forgotten about it!)

Increase default timeout to 2 mins

MetRonnie · 2021-01-14T11:25:22Z

Ok - logging verbosity, remote timeout and concurrency of ssh commands should be addressed now

oliver-sanders · 2021-01-14T11:46:47Z

Looking good.

Should demote the missing database warning as a missing database is not an error (e.g. flow not run or remote flow installation) suggest:

INFO: No database - only this platform will be cleaned

oliver-sanders · 2021-01-14T11:52:04Z

@hjoliver I think this is good to go now if you want to take a poke.

MetRonnie · 2021-01-14T11:56:00Z

Should demote the missing database warning as a missing database is not an error (e.g. flow not run or remote flow installation)

Updated the latest commit

MetRonnie · 2021-01-14T13:39:24Z

The tests have passed, however test (ubuntu-latest, 3.7, tests/f tests/k, 1/1, _remote_background_indep_poll _remote_at_indep_poll) took a fair bit longer than usual so that run just timed out before the coverage was uploaded. Might have to increase the GH workflow timeout

cylc-flow/.github/workflows/test_functional.yml

Line 8 in c2f6942

timeout-minutes: 35

oliver-sanders

Might have to increase the GH workflow timeout

Dammit, just when I though things were stable, will keep an eye out.

Looks good to me. Going to leave for @hjoliver to merge. Will need to rebase squash pre-merge.

MetRonnie · 2021-01-15T12:03:35Z

Will need to rebase squash pre-merge.

How come? I tried to keep my commits logically grouped. I can try to squash them down more before merging if needed, but I think it would pay to keep some of the changes separate

oliver-sanders · 2021-01-15T13:15:59Z

Tis how we work, most of the commits here document the process of how this branch came to be, this information doesn't really have any value going forward. We aren't going to go back and revert one commit and knowing that one feature was developed slightly earlier than another isn't useful info to anyone. The extra commits do, however, bloat the repo and make bisects longer.

Some (I think predominantly agile) projects use "squash and merge" for all PRs to strip out unnecessary commits, makes sense as the most important information to preserve the is the link to the PR, the history and purpose of the branch should be documented there. We don't because our PRs often do multiple things (we aren't very agile).

(btw we don't use commits as a measure of work or play number games)

MetRonnie · 2021-01-15T14:33:05Z

preserve the is the link to the PR, the history and purpose of the branch should be documented there

I don't know why that didn't occur to me before 👍

hjoliver

Looks great 🎉

~~Please merge after tidying the commit history as suggested by @oliver-sanders~~ (duh - he suggested squash and merge 🤦 )

MetRonnie added 12 commits December 17, 2020 16:22

cylc clean: Ensure run dir is removed if broken symlink

80572d0

Tidy

550340e

cylc clean: remote platforms

3972780

cylc clean: keep retrying with next platform in event of errors

41d1920

until all platforms for that install target have been exhausted

cylc clean: slight refactor & unit test for getting install targets map

4cd1e54

Tidy

4e679e7

Platforms: add unit test for get_install_target_from_platform()

6b3c72b

Suite files: Add test case to test_validate_reg()

0751972

SuiteDatabaseManager improvements

2763e64

- Ensure DB is closed using a try...finally statement in a couple of places where it was lacking - Refactor checking of the DB compatibility slightly

cylc clean: refactor and add/update unit tests

6a7aab7

cylc clean: fix order of removing symlink dirs

6d38cbc

Fixes a bug where if log and share symlink dirs were the same, but share/cycle was different, then share/cycle's target wouldn't get removed

Tidy

a41c66c

Plus: in CylcSuiteDAO, make db_file_name arg non-optional, as it will raise an exception anyway if not given

MetRonnie added this to the cylc-8.0b0 milestone Dec 24, 2020

MetRonnie self-assigned this Dec 24, 2020

MetRonnie changed the title ~~cylc clean II: attack of the cleans~~ cylc clean 2: remote clean Dec 24, 2020

oliver-sanders reviewed Dec 29, 2020

View reviewed changes

cylc/flow/rundb.py Outdated Show resolved Hide resolved

oliver-sanders reviewed Dec 29, 2020

View reviewed changes

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

MetRonnie commented Jan 4, 2021

View reviewed changes

cylc/flow/suite_files.py Show resolved Hide resolved

cylc clean: refactor

8fb8763

Move initial checking of whether a workflow can be cleaned to its own function

cylc clean: functional test for remote clean

7741ac0

MetRonnie force-pushed the cylc-clean branch from aad0d1c to 678b2bf Compare January 5, 2021 16:24

This comment has been minimized.

Sign in to view

MetRonnie marked this pull request as ready for review January 5, 2021 18:10

wxtim reviewed Jan 6, 2021

View reviewed changes

wxtim self-requested a review January 6, 2021 17:02

oliver-sanders reviewed Jan 6, 2021

View reviewed changes

CHANGES.md Outdated Show resolved Hide resolved

cylc/flow/scripts/clean.py Outdated Show resolved Hide resolved

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

cylc/flow/scripts/clean.py Outdated Show resolved Hide resolved

cylc clean: replace remote-clean cmd with an opt for normal clean

b36ddeb

plus 1 or 2 other code review suggestions

Merge branch 'master' into cylc-clean

bb4463b

wxtim approved these changes Jan 11, 2021

View reviewed changes

oliver-sanders reviewed Jan 12, 2021

View reviewed changes

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

MetRonnie added 2 commits January 14, 2021 10:48

cylc clean: make remote clean subprocs more concurrent

ecbc0f4

cylc clean: Add timeout option for remote clean

add6e8b

Increase default timeout to 2 mins

MetRonnie force-pushed the cylc-clean branch from 0516317 to dfb28c0 Compare January 14, 2021 11:55

Reduce verbosity of cylc clean logging

c24244b

MetRonnie force-pushed the cylc-clean branch from dfb28c0 to c24244b Compare January 14, 2021 12:32

oliver-sanders approved these changes Jan 15, 2021

View reviewed changes

hjoliver approved these changes Jan 20, 2021

View reviewed changes

hjoliver merged commit 8aba48f into cylc:master Jan 20, 2021

MetRonnie deleted the cylc-clean branch January 20, 2021 10:22

MetRonnie linked an issue Jan 20, 2021 that may be closed by this pull request

cylc clean #3887

Open

MetRonnie mentioned this pull request Mar 16, 2021

cylc clean #3887

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cylc clean 2: remote clean #4017

cylc clean 2: remote clean #4017

MetRonnie commented Dec 24, 2020 •

edited

Loading

MetRonnie commented Dec 24, 2020

oliver-sanders commented Jan 5, 2021

MetRonnie commented Jan 5, 2021

This comment has been minimized.

wxtim left a comment

oliver-sanders left a comment

wxtim left a comment •

edited

Loading

oliver-sanders commented Jan 11, 2021

MetRonnie commented Jan 12, 2021

oliver-sanders commented Jan 12, 2021

MetRonnie commented Jan 12, 2021

MetRonnie commented Jan 13, 2021

hjoliver commented Jan 13, 2021

MetRonnie commented Jan 14, 2021

oliver-sanders commented Jan 14, 2021

oliver-sanders commented Jan 14, 2021

MetRonnie commented Jan 14, 2021

MetRonnie commented Jan 14, 2021

oliver-sanders left a comment

MetRonnie commented Jan 15, 2021

oliver-sanders commented Jan 15, 2021 •

edited

Loading

MetRonnie commented Jan 15, 2021

hjoliver left a comment •

edited

Loading

cylc clean 2: remote clean #4017

cylc clean 2: remote clean #4017

Conversation

MetRonnie commented Dec 24, 2020 • edited Loading

MetRonnie commented Dec 24, 2020

oliver-sanders commented Jan 5, 2021

MetRonnie commented Jan 5, 2021

This comment has been minimized.

wxtim left a comment

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

wxtim left a comment • edited Loading

Choose a reason for hiding this comment

oliver-sanders commented Jan 11, 2021

MetRonnie commented Jan 12, 2021

oliver-sanders commented Jan 12, 2021

MetRonnie commented Jan 12, 2021

MetRonnie commented Jan 13, 2021

hjoliver commented Jan 13, 2021

MetRonnie commented Jan 14, 2021

oliver-sanders commented Jan 14, 2021

oliver-sanders commented Jan 14, 2021

MetRonnie commented Jan 14, 2021

MetRonnie commented Jan 14, 2021

oliver-sanders left a comment

Choose a reason for hiding this comment

MetRonnie commented Jan 15, 2021

oliver-sanders commented Jan 15, 2021 • edited Loading

MetRonnie commented Jan 15, 2021

hjoliver left a comment • edited Loading

Choose a reason for hiding this comment

MetRonnie commented Dec 24, 2020 •

edited

Loading

wxtim left a comment •

edited

Loading

oliver-sanders commented Jan 15, 2021 •

edited

Loading

hjoliver left a comment •

edited

Loading