Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploy cron job for syncing geps forecast data #19

Closed
4 tasks
tlogan2000 opened this issue Dec 2, 2020 · 2 comments
Closed
4 tasks

Deploy cron job for syncing geps forecast data #19

tlogan2000 opened this issue Dec 2, 2020 · 2 comments
Assignees
@tlvu @tlogan2000

Comments

@tlogan2000
Copy link
Collaborator

Relted to Ouranosinc/raven#313
and #18

  • Deploy on server boreas(?)

  • Create conda env or use docker?

  • Merged output netcdfs should go in to birdhouse/eccc/forecasts/geps

  • 'Current forecast' ncml should go into datasets/forecasts/eccc_geps
This was referenced Dec 17, 2020
Merged
Merged
tlvu added a commit that referenced this issue Dec 18, 2020
@tlvu
Merge pull request #21 from Ouranosinc/self-contain-geps-forecasts-ru…
747cff7 
…ntime

Run convert_grib2_to_nc.py inside a Docker container for reproducible runtime environment.

Added the only missing `cfgrib` package to our PAVICS Jupyter image and used our PAVICS Jupyter image as the runtime environment so other users can also perform the same task as Travis on our Jupyter if needed.

The Jupyter image is always present on Boreas so re-using it means no extra space required for the runtime of the script (except the extra 159MB for `cfgrib`).

@tlogan2000 Note that I had to repeat the the 3 paths (inpath, outpath, threddspath) to the wrapper script as well so if you ever need to modify those paths, you should modify in both scripts.  I've left comments in the code as reminder that there are duplicates.

@huard For your Intake catalog project, you can write a script similar to `convert_grib2_to_nc.py` and I'll automate its deployment and execution the same way.

Automated deployment and update is handled by PR bird-house/birdhouse-deploy-ouranos#2.

Requires PR Ouranosinc/PAVICS-e2e-workflow-tests#56 for the runtime environment.

Part of issue #19 and issue Ouranosinc/raven#313.
tlvu added a commit to bird-house/birdhouse-deploy-ouranos that referenced this issue Dec 18, 2020
@tlvu
Merge pull request #2 from bird-house/add-geps-forecasts-scheduler-jobs
b5e3d05 
Add geps forecasts scheduler jobs.

Matching PR to automate deployment and future updates of the PR Ouranosinc/pavics-vdb#21.  Requires PR bird-house/birdhouse-deploy#110.  Part of issue Ouranosinc/pavics-vdb#19 and issue Ouranosinc/raven#313.

Add a cronjob (`retrieve_geps_forecasts.env`) and a config file (`retrieve_geps_forecasts.yml`) that defines which git repo to pull from and what command to run to handle the geps forecasts.

Leverage the script I did earlier that can deploy any files from any git repos to a PAVICS host.

Activating the cronjob via `env.local` is like so https://github.com/bird-house/birdhouse-deploy/blob/2bb9ce6756cb317c47e28a3a65d95d62312c02fd/birdhouse/env.local.example#L156-L158

@tlogan2000 If you change the code in the `pavics-vdb` repo, it'll be live the next time the cronjob runs.  If you want to change the cronjob frequency, edit `retrieve_geps_forecasts.env` and it autodeploy itself.

Deployed to our staging env Medus: http://medus.ouranos.ca:8083/twitcher/ows/proxy/thredds/catalog/birdhouse/eccc/forecasts/geps/catalog.html.

@huard I'll apply this exact same recipe for your Intake catalog project.

@matprov This is an example showing you how CRIM can have custom cronjobs, all source controlled, centrally tracked by `env,local`, with automated deployment and update.

Sample cronjob logs from Medus:
[retrieve_geps_forecasts.log](https://github.com/bird-house/birdhouse-deploy-ouranos/files/5716267/retrieve_geps_forecasts.log)

```
==========
datadeploy START_TIME=2020-12-18T05:37:02+0000
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ CHECKOUT_CACHE=
+ '[' -z  ]
+ CHECKOUT_CACHE=/data/deploy_data_cache/retrieve_geps_forecasts
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ GIT_SSH_IDENTITY_FILE=
+ '[' -z  ]
+ GIT_SSH_IDENTITY_FILE=/home/mourad/.ssh/id_rsa_pavics-vdb-ro
+ '[' '!' -z /home/mourad/.ssh/id_rsa_pavics-vdb-ro ]
+ export 'GIT_SSH_COMMAND=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=/home/mourad/.ssh/id_rsa_pavics-vdb-ro'
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ GIT_REPO_URLS=git@github.com:Ouranosinc/pavics-vdb.git
+ ensure_not_empty git@github.com:Ouranosinc/pavics-vdb.git
+ '[' -z git@github.com:Ouranosinc/pavics-vdb.git ]
+ REPO_NUM=0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ GIT_BRANCH=origin/minor-enhancements-to-convert_grib2_to_nc.py
+ ensure_not_empty origin/minor-enhancements-to-convert_grib2_to_nc.py
+ '[' -z origin/minor-enhancements-to-convert_grib2_to_nc.py ]
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ GIT_CHECKOUT_NAME=pavics-vdb-for-geps-forecasts
+ ensure_not_empty pavics-vdb-for-geps-forecasts
+ '[' -z pavics-vdb-for-geps-forecasts ]
+ CLONE_DEST=/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ '[' '!' -d /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts ]
+ echo 'refresh repo '"'"'/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts'"'"' on branch '"'"'origin/minor-enhancements-to-convert_grib2_to_nc.py'"'"
refresh repo '/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts' on branch 'origin/minor-enhancements-to-convert_grib2_to_nc.py'
+ cd /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ git remote -v
origin	git@github.com:Ouranosinc/pavics-vdb.git (fetch)
origin	git@github.com:Ouranosinc/pavics-vdb.git (push)
+ git clean -fdx
+ git fetch --prune --all
Fetching origin
Warning: Permanently added 'github.com,140.82.114.3' (RSA) to the list of known hosts.
From github.com:Ouranosinc/pavics-vdb
 + 3e94dfb...a64a444 minor-enhancements-to-convert_grib2_to_nc.py -> origin/minor-enhancements-to-convert_grib2_to_nc.py  (forced update)
   a6e4c2b..a64a444  self-contain-geps-forecasts-runtime -> origin/self-contain-geps-forecasts-runtime
+ git checkout --force origin/minor-enhancements-to-convert_grib2_to_nc.py
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  3e94dfb convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs

If you want to keep it by creating a new branch, this may be a good time
to do so with:

 git branch <new-branch-name> 3e94dfb

HEAD is now at a64a444 convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ SRC_DIRS=
+ DIR_NUM=0
+ yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ POST_ACTIONS_LENGTH=1
+ '[' -z 1 ]
+ expr 1 - 1
+ POST_ACTIONS_LENGTH_FOR_SEQ=0
+ seq 0 0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ POST_ACTION='RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ ensure_not_empty 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ '[' -z 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc' ]
+ echo 'executing post_action '"'"'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'"'"
executing post_action 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ eval 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH}' 'RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH}' 'RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH}' 'CONVERT_GRIB2_TO_NC_PROGRESSBAR=false' ECCC-datamart_sync/run_convert_grib2_to_nc
+ RUN_CONVERT_GRIB2_TO_NC_INPATH= RUN_CONVERT_GRIB2_TO_NC_OUTPATH= RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc
==========
run_convert_grib2_to_nc START_TIME=2020-12-18T05:37:25+0000
+ id -u
+ USER_ID=0
+ id -g
+ GROUP_ID=0
+ mkdir -p /data/tmp/geps_forecast/grib2
+ mkdir -p /data/tmp/geps_forecast/netcdf
+ docker run --rm --name run_convert_grib2_to_nc --volume /data/tmp/geps_forecast/grib2:/data/tmp/geps_forecast/grib2:rw --volume /data/tmp/geps_forecast/netcdf:/data/tmp/geps_forecast/netcdf:rw --volume /data/datasets/eccc/forecasts/geps:/data/datasets/eccc/forecasts/geps:rw --volume /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:ro --env 'CONVERT_GRIB2_TO_NC_INPATH=/data/tmp/geps_forecast/grib2' --env 'CONVERT_GRIB2_TO_NC_OUTPATH=/data/tmp/geps_forecast/netcdf' --env 'CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps' --env CONVERT_GRIB2_TO_NC_PROGRESSBAR --user 0:0 pavics/workflow-tests:201214 python /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py
Downloading grib2 files : attempt 1 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 193 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 2 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 3 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
coverting to netcdf ....
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P159_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P162_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P165_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P168_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P036_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P039_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P042_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P045_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P096_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P099_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P102_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P105_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P099_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P102_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P105_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P108_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P294_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P300_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P306_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P312_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P240_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P246_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P252_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P258_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P027_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P030_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P033_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P036_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P222_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P228_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P234_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P240_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P168_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P171_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P174_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P177_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P051_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P054_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P057_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P060_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P048_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P051_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P054_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P057_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P120_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P123_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P126_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P129_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P312_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P318_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P324_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P330_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P135_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P138_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P141_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P144_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P060_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P063_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P066_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P069_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P132_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P135_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P138_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P141_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P360_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P366_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P372_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P378_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P087_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P090_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P093_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P096_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P342_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P348_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P354_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P360_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P288_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P294_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P300_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P306_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P063_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P066_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P069_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P072_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P318_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P324_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P330_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P336_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P192_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P198_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P204_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P210_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P111_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P114_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P117_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P120_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P366_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P372_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P378_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P384_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P264_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P270_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P276_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P282_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P171_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P174_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P177_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P180_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P000_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P003_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P006_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P009_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P084_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P087_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P090_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P093_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P384_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P147_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P150_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P153_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P156_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P012_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P015_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P018_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P021_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P108_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P111_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P114_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P117_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P123_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P126_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P129_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P132_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P024_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P027_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P030_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P033_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P072_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P075_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P078_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P081_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P336_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P342_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P348_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P354_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P075_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P078_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P081_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P084_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P270_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P276_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P282_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P288_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P216_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P222_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P228_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P234_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P015_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P018_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P021_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P024_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P198_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P204_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P210_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P216_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P144_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P147_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P150_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P153_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P039_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P042_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P045_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P048_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P246_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P252_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P258_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P264_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P180_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P183_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P186_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P189_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P003_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P006_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P009_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P012_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P183_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P186_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P189_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P192_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P156_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P159_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P162_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P165_allmbrs.grib2
done coverting  GEPS . It took :  222.2973051071167 seconds
2020121400 : no action needed for combining variables and timesteps
2020121412 : no action needed for combining variables and timesteps
2020121500 : no action needed for combining variables and timesteps
2020121512 : no action needed for combining variables and timesteps
2020121600 : no action needed for combining variables and timesteps
2020121612 : no action needed for combining variables and timesteps
2020121700 : no action needed for combining variables and timesteps
2020121712 : no action needed for combining variables and timesteps
2020121800 : combining variables and timesteps ...
CMC_geps-raw_latlon0p5x0p5_2020121800_allP_allmbrs.nc
Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 153, in validate_ncml
    assert ds.reftime.values == pd.to_datetime(start_date, format='%Y%m%d%H')
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 336, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 141, in main
    if validate_ncml(opendap_latest, latest_date):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 157, in validate_ncml
    raise Exception("can't read ncml opendap link")
Exception: can't read ncml opendap link
+ cleanup_on_exit
+ set +x

run_convert_grib2_to_nc finished START_TIME=2020-12-18T05:37:25+0000
run_convert_grib2_to_nc finished   END_TIME=2020-12-18T05:47:11+0000
+ expr 0 + 1
+ REPO_NUM=1
+ cleanup_on_exit
+ set +x

datadeploy finished START_TIME=2020-12-18T05:37:02+0000
datadeploy finished   END_TIME=2020-12-18T05:47:12+0000
==========
datadeploy START_TIME=2020-12-18T06:37:06+0000
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ CHECKOUT_CACHE=
+ '[' -z  ]
+ CHECKOUT_CACHE=/data/deploy_data_cache/retrieve_geps_forecasts
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ GIT_SSH_IDENTITY_FILE=
+ '[' -z  ]
+ GIT_SSH_IDENTITY_FILE=/home/mourad/.ssh/id_rsa_pavics-vdb-ro
+ '[' '!' -z /home/mourad/.ssh/id_rsa_pavics-vdb-ro ]
+ export 'GIT_SSH_COMMAND=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=/home/mourad/.ssh/id_rsa_pavics-vdb-ro'
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ GIT_REPO_URLS=git@github.com:Ouranosinc/pavics-vdb.git
+ ensure_not_empty git@github.com:Ouranosinc/pavics-vdb.git
+ '[' -z git@github.com:Ouranosinc/pavics-vdb.git ]
+ REPO_NUM=0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ GIT_BRANCH=origin/minor-enhancements-to-convert_grib2_to_nc.py
+ ensure_not_empty origin/minor-enhancements-to-convert_grib2_to_nc.py
+ '[' -z origin/minor-enhancements-to-convert_grib2_to_nc.py ]
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ GIT_CHECKOUT_NAME=pavics-vdb-for-geps-forecasts
+ ensure_not_empty pavics-vdb-for-geps-forecasts
+ '[' -z pavics-vdb-for-geps-forecasts ]
+ CLONE_DEST=/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ '[' '!' -d /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts ]
+ echo 'refresh repo '"'"'/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts'"'"' on branch '"'"'origin/minor-enhancements-to-convert_grib2_to_nc.py'"'"
refresh repo '/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts' on branch 'origin/minor-enhancements-to-convert_grib2_to_nc.py'
+ cd /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ git remote -v
origin	git@github.com:Ouranosinc/pavics-vdb.git (fetch)
origin	git@github.com:Ouranosinc/pavics-vdb.git (push)
+ git clean -fdx
+ git fetch --prune --all
Fetching origin
Warning: Permanently added 'github.com,140.82.114.4' (RSA) to the list of known hosts.
+ git checkout --force origin/minor-enhancements-to-convert_grib2_to_nc.py
HEAD is now at a64a444 convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ SRC_DIRS=
+ DIR_NUM=0
+ yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ POST_ACTIONS_LENGTH=1
+ '[' -z 1 ]
+ expr 1 - 1
+ POST_ACTIONS_LENGTH_FOR_SEQ=0
+ seq 0 0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ POST_ACTION='RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ ensure_not_empty 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ '[' -z 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc' ]
+ echo 'executing post_action '"'"'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'"'"
executing post_action 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ eval 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH}' 'RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH}' 'RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH}' 'CONVERT_GRIB2_TO_NC_PROGRESSBAR=false' ECCC-datamart_sync/run_convert_grib2_to_nc
+ RUN_CONVERT_GRIB2_TO_NC_INPATH= RUN_CONVERT_GRIB2_TO_NC_OUTPATH= RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc
==========
run_convert_grib2_to_nc START_TIME=2020-12-18T06:37:30+0000
+ id -u
+ USER_ID=0
+ id -g
+ GROUP_ID=0
+ mkdir -p /data/tmp/geps_forecast/grib2
+ mkdir -p /data/tmp/geps_forecast/netcdf
+ docker run --rm --name run_convert_grib2_to_nc --volume /data/tmp/geps_forecast/grib2:/data/tmp/geps_forecast/grib2:rw --volume /data/tmp/geps_forecast/netcdf:/data/tmp/geps_forecast/netcdf:rw --volume /data/datasets/eccc/forecasts/geps:/data/datasets/eccc/forecasts/geps:rw --volume /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:ro --env 'CONVERT_GRIB2_TO_NC_INPATH=/data/tmp/geps_forecast/grib2' --env 'CONVERT_GRIB2_TO_NC_OUTPATH=/data/tmp/geps_forecast/netcdf' --env 'CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps' --env CONVERT_GRIB2_TO_NC_PROGRESSBAR --user 0:0 pavics/workflow-tests:201214 python /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py
Downloading grib2 files : attempt 1 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 2 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 3 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
coverting to netcdf ....
done coverting  GEPS . It took :  2.776029586791992 seconds
2020121400 : no action needed for combining variables and timesteps
2020121412 : no action needed for combining variables and timesteps
2020121500 : no action needed for combining variables and timesteps
2020121512 : no action needed for combining variables and timesteps
2020121600 : no action needed for combining variables and timesteps
2020121612 : no action needed for combining variables and timesteps
2020121700 : no action needed for combining variables and timesteps
2020121712 : no action needed for combining variables and timesteps
2020121800 : no action needed for combining variables and timesteps
Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 153, in validate_ncml
    assert ds.reftime.values == pd.to_datetime(start_date, format='%Y%m%d%H')
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 336, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 141, in main
    if validate_ncml(opendap_latest, latest_date):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 157, in validate_ncml
    raise Exception("can't read ncml opendap link")
Exception: can't read ncml opendap link
+ cleanup_on_exit
+ set +x

run_convert_grib2_to_nc finished START_TIME=2020-12-18T06:37:30+0000
run_convert_grib2_to_nc finished   END_TIME=2020-12-18T06:37:48+0000
+ expr 0 + 1
+ REPO_NUM=1
+ cleanup_on_exit
+ set +x

datadeploy finished START_TIME=2020-12-18T06:37:06+0000
datadeploy finished   END_TIME=2020-12-18T06:37:48+0000
```
@huard
Copy link
Contributor

huard commented Jan 11, 2021

@tlogan2000 @tlvu This can be closed, right ?

@tlvu
Copy link
Collaborator

tlvu commented Jan 11, 2021

Pretty sure yes, I'll let Travis make the final call if he has something else in mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlvu tlvu

@tlogan2000 tlogan2000

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@huard @tlvu @tlogan2000