jupyter docker: upgrade birdy, pin pandas and geopandas for Era5 Raven notebooks #37

tlvu · 2020-03-10T22:41:22Z

This is to productize all the manual pip install --user we had to do to demo Era5 Raven notebook PR: Ouranosinc/raven#200

birdy updated for Raven, then for Finch.

Pinning of pandas and geopandas is due to salem not supporting geopandas >= 0.7 as in this issue fmaussion/salem#152 so can unpin once there are new release of salem.

scikit-image was also need by Raven notebook gridded_data_subset.ipynb.

The ipyleaflet marker do not display in Raven Region_selection.ipynb but can be moved and its coordinate can be extracted.

Not sure why, maybe relate to jupyter-widgets/ipyleaflet#91. Possibly also due to the freshly released jupyterlab-2.0 but downgrading jupyterlab means downgrading all the other jupyter extensions we used, too much work to just fix one broken extension so did not follow that road.

The marker was working fine in the previous image pavics/workflow-tests:200120 which had jupyterlab-1.2.5 which further support the theory that jupyterlab-2.0 broke it.

Noticeable Jupyter env changes:

<   - birdy=v0.6.6=py_0
>     - birdhouse-birdy==0.6.9  (install from Pypi instead of Conda to fix DockerHub failure !)

<   - geopandas=0.7.0=py_1
>   - geopandas=0.6.2=py_0 

<   - pandas=1.0.1=py37hb3f55d8_0                                                                                                                     
>   - pandas=0.25.3=py37hb3f55d8_0

>   - scikit-image=0.16.2=py37hb3f55d8_0

Full conda env export Diff:
200309-200310-conda-env-export.diff.txt

200309-200312-conda-env-export.diff.txt

200310-200312-conda-env-export.diff.txt

Full new conda env export:
200310-conda-env-export.yml.txt

200312-conda-env-export.yml.txt

…ed_data_subset.ipynb scikit-image need by that same notebook as well. This is most likely because salem do not work with geopandas >=0.7. Can try to un-pin geopandas and pandas with new salem release.

…upyterlab For Raven Region_selection.ipynb.

…as well, too much work Step 9/18 : RUN jupyter lab build ---> Running in 9593c434adfe [LabBuildApp] JupyterLab 1.2.5 [LabBuildApp] Building in /usr/local/envs/birdy/share/jupyter/lab [LabBuildApp] Building jupyterlab assets (build:prod:minimize) [LabBuildApp] WARNING | "nbdime-jupyterlab@2.0.0" is not compatible with the current JupyterLab Conflicting Dependencies: JupyterLab Extension Package >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/apputils >=3.2.0 <3.3.0 >=4.0.0 <5.0.0 @jupyterlab/coreutils >=1.2.2 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/notebook >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/rendermime >=4.2.0 <4.3.0 >=5.0.0 <6.0.0 @jupyterlab/services Removing intermediate container 9593c434adfe ---> 16c7a0cdd721 Step 10/18 : RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager && jupyter serverextension enable voila --sys-prefix && jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leaflet ---> Running in 4f18a273aab9 Building jupyterlab assets (build:prod:minimize) "nbdime-jupyterlab@2.0.0" is not compatible with the current JupyterLab Conflicting Dependencies: JupyterLab Extension Package >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/apputils >=3.2.0 <3.3.0 >=4.0.0 <5.0.0 @jupyterlab/coreutils >=1.2.2 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/notebook >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/rendermime >=4.2.0 <4.3.0 >=5.0.0 <6.0.0 @jupyterlab/services Enabling: voila - Writing config: /usr/local/envs/birdy/etc/jupyter - Validating... voila 0.1.21 OK Building jupyterlab assets (build:prod:minimize) "nbdime-jupyterlab@2.0.0" is not compatible with the current JupyterLab Conflicting Dependencies: JupyterLab Extension Package >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/apputils >=3.2.0 <3.3.0 >=4.0.0 <5.0.0 @jupyterlab/coreutils >=1.2.2 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/notebook >=1.2.1 <1.3.0 >=2.0.0 <3.0.0 @jupyterlab/rendermime >=4.2.0 <4.3.0 >=5.0.0 <6.0.0 @jupyterlab/services Removing intermediate container 4f18a273aab9

tlvu · 2020-03-10T22:46:21Z

@huard Looks like the new birdy broke Finch: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/install-birdy-from-source/2/console

18:22:45  _________ pavics-sdi-master/docs/source/notebooks/finch.ipynb::Cell 2 __________
18:22:45  Notebook cell execution failed
18:22:45  Cell 2: Cell execution caused an exception
18:22:45  
18:22:45  Input:
18:22:45  tasmin = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/flyingpigeon/cmip3/tasmin.sresa2.miub_echo_g.run1.atm.da.nc"
18:22:45  resp = wps.frost_days(tasmin)
18:22:45  
18:22:45  Traceback:
18:22:45  
18:22:45  ---------------------------------------------------------------------------
18:22:45  ValueError                                Traceback (most recent call last)
18:22:45  <ipython-input-3-b7c63d8c88b4> in <module>
18:22:45        1 tasmin = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/flyingpigeon/cmip3/tasmin.sresa2.miub_echo_g.run1.atm.da.nc"
18:22:45  ----> 2 resp = wps.frost_days(tasmin)
18:22:45  
18:22:45  </src/birdhouse-birdy/birdy/client/base.py-39> in frost_days(self, tasmin, freq)
18:22:45  
18:22:45  /src/birdhouse-birdy/birdy/client/base.py in _execute(self, pid, **kwargs)
18:22:45      281     def _execute(self, pid, **kwargs):
18:22:45      282         """Execute the process."""
18:22:45  --> 283         wps_inputs = self._build_inputs(pid, **kwargs)
18:22:45      284         wps_outputs = [
18:22:45      285             (o.identifier, "ComplexData" in o.dataType)
18:22:45  
18:22:45  /src/birdhouse-birdy/birdy/client/base.py in _build_inputs(self, pid, **kwargs)
18:22:45      251 
18:22:45      252                     # Guess the mimetype of the input value
18:22:45  --> 253                     mimetype, encoding = guess_type(value, supported_mimetypes)
18:22:45      254 
18:22:45      255                     if encoding is None:
18:22:45  
18:22:45  /src/birdhouse-birdy/birdy/utils.py in guess_type(url, supported)
18:22:45      166     else:
18:22:45      167         if mime not in supported:
18:22:45  --> 168             raise ValueError(f"mimetype {mime} not in supported mimetypes {supported}.")
18:22:45      169 
18:22:45      170     return mime, enc
18:22:45  
18:22:45  ValueError: mimetype application/x-ogc-dods not in supported mimetypes ['application/x-netcdf'].

huard · 2020-03-10T23:33:28Z

Ok, this will be fixed in Finch itself. Will make a new release this week.

tlvu · 2020-03-10T23:49:16Z

Ok, this will be fixed in Finch itself. Will make a new release this week.

Reference to Finch issue bird-house/finch#99, thanks David !

huard

New birdy version (on pypi) should fix the Finch problem, but I'll update Finch as well to make sure.

docker/environment.yml

Noticed one passing DockerHub build since we installed from source so maybe using pip to install might help. This is just a shoot in the dark.

tlvu · 2020-03-12T16:52:45Z

Reference to birdy fix bird-house/birdy#175

tlvu · 2020-03-12T17:13:59Z

@huard Jenkins test failed against the new Finch: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/install-birdy-from-source/4/console

There is this error, easy to fix, just want to confirm it's expected:

12:46:06  _________ pavics-sdi-master/docs/source/notebooks/finch.ipynb::Cell 1 __________
12:46:06  Notebook cell execution failed
12:46:06  Cell 1: Cell outputs differ
12:46:06  
12:46:06  Input:
12:46:06  help(wps.frost_days)
12:46:06  
12:46:06  Traceback:
12:46:06   mismatch 'stdout'
12:46:06  
12:46:06   assert reference_output == test_output failed:
12:46:06  
12:46:06    'Help on meth...cess run.\n\n' == 'Help on meth...ut files.\n\n'
12:46:06    Skipping 284 identical leading characters in diff, use -v to show
12:46:06    - /x-netcdf`
12:46:06    + /x-netcdf`, :mimetype:`application/x-ogc-dods`
12:46:06              NetCDF Files or archive (tar/zip) containing netCDF files.
12:46:06          freq : {'YS', 'MS', 'QS-DEC', 'AS-JUL'}string
12:46:06              Resampling frequency
12:46:06          
12:46:06          Returns
12:46:06          -------
12:46:06          output_netcdf : ComplexData:mimetype:`application/x-netcdf`
12:46:06              The indicator values computed on the original input grid.
12:46:06          output_log : ComplexData:mimetype:`text/plain`
12:46:06              Collected logs during process run.
12:46:06    +     ref : ComplexData:mimetype:`application/metalink+xml; version=4.0`
12:46:06    +         Metalink file storing all references to output files.

Then there is this real error:

12:46:06  _________ pavics-sdi-master/docs/source/notebooks/finch.ipynb::Cell 2 __________
12:46:06  Notebook cell execution failed
12:46:06  Cell 2: Cell outputs differ
12:46:06  
12:46:06  Input:
12:46:06  tasmin = "https://lvupavics-lvu.pagekite.me/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/flyingpigeon/cmip3/tasmin.sresa2.miub_echo_g.run1.atm.da.nc"
12:46:06  resp = wps.frost_days(tasmin)
12:46:06  
12:46:06  Traceback:
12:46:06  Unexpected output fields from running code: {'stdout'}

With Traceback from the Finch container:

INFO:PYWPS:Request: execute        
INFO:PYWPS:Validating OPeNDAP; Mode: 0
INFO:PYWPS:Computing the output netcdf
Traceback (most recent call last): 
  File "/opt/conda/envs/finch/lib/python3.7/site-packages/pywps/app/Process.py", line 243, in _run_process
    self.handler(wps_request, wps_response)  # the user must update the wps_response.
  File "/code/finch/processes/wps_base.py", line 34, in _handler_wrapper
    return self.wrapped_handler(request, response)
  File "/code/finch/processes/wps_xclim_indices.py", line 97, in _handler
    filename = _make_unique_drs_filename(out, [f.name for f in output_files])
  File "/code/finch/processes/wps_xclim_indices.py", line 134, in _make_unique_drs_filename
    filename = drs_filename(ds)       
  File "/code/finch/processes/utils.py", line 186, in drs_filename
    raise Exception(f"Unknown project: {ds.attrs['project_id']}")                                         
Exception: Unknown project: IPCC Fourth Assessment                                   
ERROR:PYWPS:Process error: method=wps_xclim_indices.py._handler, line=97, msg=Unknown project: IPCC Fourth Assessment
INFO:PYWPS:Removing temporary working directory: /tmp/pywps_process_yl1duk2g

tlvu · 2020-03-12T17:17:22Z

Add reference to Finch update in PAVICS: bird-house/birdhouse-deploy#32

huard · 2020-03-12T17:51:23Z

The first is expected.
The second is probably an easy fix. @davidcaron

tlvu · 2020-03-13T01:42:41Z

Latest birdy 0.6.9 is working flawlessly with existing old Finch 0.2.7 and existing recently released Raven 0.10.0. See Jenkins run http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/install-birdy-from-source/9/console for Finch testing. Raven test done manually.

So I'll release this Jupyter env so at least the Raven notebooks works without manually hacking the Jupyter env. Old Finch is not broken anymore so no regression there.

See PR Ouranosinc/PAVICS-e2e-workflow-tests#37 or commit Ouranosinc/PAVICS-e2e-workflow-tests@8bbb5b4 for more info. Noticeable Jupyter env changes: ```diff < - birdy=v0.6.6=py_0 > - birdhouse-birdy==0.6.9 (install from Pypi instead of Conda to fix DockerHub failure !) < - geopandas=0.7.0=py_1 > - geopandas=0.6.2=py_0 < - pandas=1.0.1=py37hb3f55d8_0 > - pandas=0.25.3=py37hb3f55d8_0 > - scikit-image=0.16.2=py37hb3f55d8_0 ```

tlvu added 5 commits March 10, 2020 14:21

docker: install birdy from source to avoid waiting for a full release

533a3ce

docker: pin old working geopandas and pandas for Raven notebook gridd…

5b1806e

…ed_data_subset.ipynb scikit-image need by that same notebook as well. This is most likely because salem do not work with geopandas >=0.7. Can try to un-pin geopandas and pandas with new salem release.

docker: pin old jupyterlab for ipyleaf marker not displaying in new j…

15cf5b2

…upyterlab For Raven Region_selection.ipynb.

release: update to use image pavics/workflow-tests:200310

56e4b1d

tlvu requested a review from huard March 10, 2020 22:41

huard reviewed Mar 11, 2020

View reviewed changes

docker/environment.yml Outdated Show resolved Hide resolved

docker/environment.yml Show resolved Hide resolved

tlvu added 3 commits March 12, 2020 12:32

docker: install birdy 0.6.9 from pip to try fix DockerHub build

cf81f13

Noticed one passing DockerHub build since we installed from source so maybe using pip to install might help. This is just a shoot in the dark.

docker: note condition to unpin geopandas and pandas

46a6fc6

release: update to use image pavics/workflow-tests:200312

e097769

huard mentioned this pull request Mar 12, 2020

drs_filename creates unncessary failures bird-house/finch#103

Closed

tlvu mentioned this pull request Mar 12, 2020

Update finch to v0.5.1 bird-house/birdhouse-deploy#32

Merged

tlvu merged commit 8bbb5b4 into master Mar 13, 2020

tlvu deleted the install-birdy-from-source branch March 13, 2020 01:48

tlvu mentioned this pull request Mar 13, 2020

jupyter: update to pavics/workflow-tests:200312 for Raven notebooks bird-house/birdhouse-deploy#34

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jupyter docker: upgrade birdy, pin pandas and geopandas for Era5 Raven notebooks #37

jupyter docker: upgrade birdy, pin pandas and geopandas for Era5 Raven notebooks #37

tlvu commented Mar 10, 2020 •

edited

Loading

tlvu commented Mar 10, 2020

huard commented Mar 10, 2020

tlvu commented Mar 10, 2020

huard left a comment

tlvu commented Mar 12, 2020

tlvu commented Mar 12, 2020

tlvu commented Mar 12, 2020

huard commented Mar 12, 2020

tlvu commented Mar 13, 2020

jupyter docker: upgrade birdy, pin pandas and geopandas for Era5 Raven notebooks #37

jupyter docker: upgrade birdy, pin pandas and geopandas for Era5 Raven notebooks #37

Conversation

tlvu commented Mar 10, 2020 • edited Loading

tlvu commented Mar 10, 2020

huard commented Mar 10, 2020

tlvu commented Mar 10, 2020

huard left a comment

Choose a reason for hiding this comment

tlvu commented Mar 12, 2020

tlvu commented Mar 12, 2020

tlvu commented Mar 12, 2020

huard commented Mar 12, 2020

tlvu commented Mar 13, 2020

tlvu commented Mar 10, 2020 •

edited

Loading