-
Notifications
You must be signed in to change notification settings - Fork 168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove pslot dir from local archive folder on CI case completion #2961
base: develop
Are you sure you want to change the base?
Remove pslot dir from local archive folder on CI case completion #2961
Conversation
…A/global-workflow into ci_clean_loc_arch
Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in |
Experiment C48_S2SW FAILED on Hera in Build# 1 in |
Experiment C48_ATM FAILED on Hera in Build# 1 in |
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in |
Experiment C96_atm3DVar FAILED on Hera in Build# 1 in |
Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in |
Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in |
Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in |
CI Failed on Hera in Build# 1
|
…Jenkins file to cleanup experiment
CI Passed on Hera in Build# 2
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will still need to add removal of ATARDIR
/LOCALDIR
in a future PR.
@WalterKolczynski-NOAA Please put the specifics to those in an Issue Task please. |
…ars work in cleanup_experiment
OK, ready for merge. |
PSLOT_PATH="$1" | ||
pslot=$(basename "${PSLOT_PATH}") | ||
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]"' || true) || true | ||
eval HOMEDIR="${HOMEDIR}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this eval
doing? Isn't it already defined on the prior line? eval
should be avoided if at all possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The piping left some outer quotes around some strings so that some of the variables where not resolving without it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few other directories that contain data, though none are very large:
${ARCHIVEDIR}/metplus_data/by_VSDB/{grid2grid,grid2obs,precip}/*/*/${pslot}
${HOME}/.rocoto/<rocoto version>/${pslot} # These are typically empty directories on success with just a log and .lock file otherwise
EDIT: changed ${HOMEDIR}
to ${HOME}
for the rocoto directory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think is is unsafe to mess with user's Rocoto hidden state/conf directory. I would rather not add this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright, that's fine. What do you think about the metplus_data
directories?
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]"' || true) || true | ||
eval HOMEDIR="${HOMEDIR}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try this
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]"' || true) || true | |
eval HOMEDIR="${HOMEDIR}" | |
local regex="^export HOMEDIR=(.*)" | |
local lines | |
mapfile -t lines < "${PSLOT_PATH}/config.base" | |
local line | |
for line in "${lines[@]}"; do | |
if [[ "${line}" =~ ${regex} ]]; then | |
HOMEDIR=$(echo "${BASH_REMATCH[1]}" | envsubst) | |
fi | |
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow looks strong, yeah I'll try that.
Description
This PR adds an cleanup_experiment function in the BASH utils to clean up after a CI case is ran.
This consolidates these actions and also adds the feature to clean the local archive folder specified by
$HOMEDIR/archive/$PSLOT
.Resolves #2954
-->
Type of change
Change characteristics
How has this been tested?
Checklist