Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker: new build to get updates and add cfgrib #56

Merged
merged 2 commits into from
Jan 12, 2021
Merged

docker: new build to get updates and add cfgrib #56

merged 2 commits into from
Jan 12, 2021

Conversation

tlvu
Copy link
Contributor

@tlvu tlvu commented Dec 17, 2020 •
edited
Loading

cfgrib added for PR Ouranosinc/pavics-vdb#21.

Jenkins build with known error only: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-with-cfgrib/2/console

Corresponding test fix Ouranosinc/pavics-sdi#195.

Corresponding PR to deploy to PAVICS bird-house/birdhouse-deploy#112.

Relevant changes:

>   - cfgrib=0.9.8.5=pyhd8ed1ab_0

<   - clisops=0.3.1=pyh32f6830_1
>   - clisops=0.4.0=pyhd3deb0d_0

<   - dask=2.30.0=py_0
>   - dask=2020.12.0=pyhd8ed1ab_0

<   - owslib=0.20.0=py_0
>   - owslib=0.21.0=pyhd8ed1ab_0

<   - xarray=0.16.1=py_0
>   - xarray=0.16.2=pyhd8ed1ab_0

<   - xclim=0.21.0=py_0
>   - xclim=0.22.0=pyhd8ed1ab_0

<   - jupyter_conda=3.4.1=pyh9f0ad1d_0
>   - jupyter_conda=4.1.0=hd8ed1ab_1

Full conda env export diff:
201111-201214-conda-env-export.diff.txt

Full new conda env export:
201214-conda-env-export.yml.txt

@tlvu tlvu requested a review from tlogan2000 December 17, 2020 03:46
@tlvu
Copy link
Contributor Author

tlvu commented Dec 17, 2020

@tlogan2000 @Zeitsperre note dask version format change. You guys aware?

@tlvu tlvu mentioned this pull request Dec 17, 2020
Merged
@Zeitsperre
Copy link
Collaborator

Nope. That's news to me! I imagine that it's not a huge issue given that 2020 > 2 ?

@tlvu
Copy link
Contributor Author

tlvu commented Dec 17, 2020

Nope. That's news to me! I imagine that it's not a huge issue given that 2020 > 2 ?

I am not worried about the upgrade not working.

I wonder if this signals some kind of fundamental changes in their release stability. Recall with simver, if minor version change, we expect backward compat changes. If just patch version change, it's probably very safe to not even need to test extensively.

This "convention" is out of the window with rolling release using date as version.

tlogan2000
tlogan2000 approved these changes Dec 17, 2020
View reviewed changes
Copy link
Collaborator

@tlogan2000 tlogan2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks ok. Thanks for the heads-up of new dask 's new {year}.{month} numbering of releases

@tlvu
Copy link
Contributor Author

tlvu commented Dec 17, 2020

Does anyone have time to invetiagte the extetn of changes in the latest dask release vs older version?

dask/dask@2.30.0...2020.12.0

Hopefully by eyeballing the commits description will give you guys some hints.

@aulemahal
Copy link
Collaborator

I had to fight dask in the last week so I took time to quickly read the changelog. Nothing changed that would directly impact pavics I think. Some things might open new possibilities for xarray, but we'll have to wait before it comes up to us.

tlvu added a commit to Ouranosinc/pavics-vdb that referenced this pull request Dec 18, 2020
@tlvu
Merge pull request #21 from Ouranosinc/self-contain-geps-forecasts-ru…
747cff7 
…ntime

Run convert_grib2_to_nc.py inside a Docker container for reproducible runtime environment.

Added the only missing `cfgrib` package to our PAVICS Jupyter image and used our PAVICS Jupyter image as the runtime environment so other users can also perform the same task as Travis on our Jupyter if needed.

The Jupyter image is always present on Boreas so re-using it means no extra space required for the runtime of the script (except the extra 159MB for `cfgrib`).

@tlogan2000 Note that I had to repeat the the 3 paths (inpath, outpath, threddspath) to the wrapper script as well so if you ever need to modify those paths, you should modify in both scripts.  I've left comments in the code as reminder that there are duplicates.

@huard For your Intake catalog project, you can write a script similar to `convert_grib2_to_nc.py` and I'll automate its deployment and execution the same way.

Automated deployment and update is handled by PR bird-house/birdhouse-deploy-ouranos#2.

Requires PR Ouranosinc/PAVICS-e2e-workflow-tests#56 for the runtime environment.

Part of issue #19 and issue Ouranosinc/raven#313.
@tlvu tlvu mentioned this pull request Jan 12, 2021
Merged
tlvu added a commit to Ouranosinc/pavics-sdi that referenced this pull request Jan 12, 2021
@tlvu
Merge pull request #195 from Ouranosinc/catalog_search.ipynb-fix-jenk…
e50335e 
…ins-failure

catalog_search.ipynb: fix help() output change that broke Jenkins

The change is 'None' to None (without the surrounding quote).

Jenkins error due to upgrade from pavics/workflow-tests:201111 to pavics/workflow-tests:201214 (PR Ouranosinc/PAVICS-e2e-workflow-tests#56)

```
  _____ pavics-sdi-master/docs/source/notebooks/catalog_search.ipynb::Cell 0 _____
  Notebook cell execution failed
  Cell 0: Cell outputs differ

  Input:
  import collections
  from birdy import WPSClient
  url = "https://pavics.ouranos.ca/twitcher/ows/proxy/catalog/wps"
  wps = WPSClient(url)
  help(wps.pavicsearch)

  Traceback:
   mismatch 'stdout'

   assert reference_output == test_output failed:

    'Help on meth...h result.\n\n' == 'Help on meth...h result.\n\n'
    Skipping 66 identical leading characters in diff, use -v to show
    - ch(facets=None, shards='*', offset=0, limit=0, fields='*', format='application/solr+json', query='*', distrib=False, type='Dataset', constraints=None, esgf=False, list_type='opendap_url') method of birdy.client.base.WPSClient instance
    ?               ^                                                                                                                                      ^
    + ch(facets='None', shards='*', offset=0, limit=0, fields='*', format='application/solr+json', query='*', distrib=False, type='Dataset', constraints='None', esgf=False, list_type='opendap_url') method of birdy.client.base.WPSClient instance
    ?           +    ^^                                                                                                                                  +    ^^
          Search the PAVICS database and return a catalogue of matches.

          Parameters
          ----------
          facets : string
              Comma separated list of facets; facets are searchable indexing terms in the database.
          shards : string
              Shards to be queried
```
tlvu added a commit to bird-house/birdhouse-deploy that referenced this pull request Jan 12, 2021
@tlvu
jupyter: update to version 201214
5df809c 
Relevant changes:

```diff
>   - cfgrib=0.9.8.5=pyhd8ed1ab_0

<   - clisops=0.3.1=pyh32f6830_1
>   - clisops=0.4.0=pyhd3deb0d_0

<   - dask=2.30.0=py_0
>   - dask=2020.12.0=pyhd8ed1ab_0

<   - owslib=0.20.0=py_0
>   - owslib=0.21.0=pyhd8ed1ab_0

<   - xarray=0.16.1=py_0
>   - xarray=0.16.2=pyhd8ed1ab_0

<   - xclim=0.21.0=py_0
>   - xclim=0.22.0=pyhd8ed1ab_0

<   - jupyter_conda=3.4.1=pyh9f0ad1d_0
>   - jupyter_conda=4.1.0=hd8ed1ab_1
```

See PR Ouranosinc/PAVICS-e2e-workflow-tests#56 for more details.
@tlvu tlvu mentioned this pull request Jan 12, 2021
Merged
@tlvu tlvu merged commit 746abf3 into master Jan 12, 2021
@tlvu tlvu deleted the new-docker-build-with-cfgrib branch January 12, 2021 19:26
tlvu added a commit to bird-house/birdhouse-deploy that referenced this pull request Jan 14, 2021
@tlvu
Merge pull request #112 from bird-house/update-jupyter-env
06f6319 
jupyter: update to version 201214

Matching PR to deploy the new Jupyter env in PR Ouranosinc/PAVICS-e2e-workflow-tests#56 to PAVICS.

Relevant changes:

```diff
>   - cfgrib=0.9.8.5=pyhd8ed1ab_0

<   - clisops=0.3.1=pyh32f6830_1
>   - clisops=0.4.0=pyhd3deb0d_0

<   - dask=2.30.0=py_0
>   - dask=2020.12.0=pyhd8ed1ab_0

<   - owslib=0.20.0=py_0
>   - owslib=0.21.0=pyhd8ed1ab_0

<   - xarray=0.16.1=py_0
>   - xarray=0.16.2=pyhd8ed1ab_0

<   - xclim=0.21.0=py_0
>   - xclim=0.22.0=pyhd8ed1ab_0

<   - jupyter_conda=3.4.1=pyh9f0ad1d_0
>   - jupyter_conda=4.1.0=hd8ed1ab_1
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlogan2000 tlogan2000 tlogan2000 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tlvu @Zeitsperre @aulemahal @tlogan2000