C3S_34e Meeting with Ouranos and CRIM: 26th August 2021

[agstephens]

Agenda - C3S_34e Meeting with Ouranos and CRIM: 26th August 2021

1. Minutes and actions from previous meeting

Previous minutes: #101

Previous actions

• AS: Put regridding higher up the agenda next month
• AS: move meetings back to 4-weekly from today - and edit meeting settings in Zoom
• ES/AS: Run the cf-checker on example files generated by Xarray to see if 1D coordinate variables with _FillValue set will fail cf-checks. Do we need to take action? - Ongoing
• ES: Create an issue on the xarray repository to explain the _FillValue in 1D coordinate variables, and that we believe these contravene the CF-Conventions (See "coordinate variable" section under: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#terminology). See: _FillValue added with output to netcdf pydata/xarray#5448
• MS: Raise an xESMF issue to discuss haloes

2. Project update: c3s_34e [Ag]

• quiet times over the summer:

  • still some xarray/numpy/dask memory errors on our service
  • working on deployment of fix for decadal (DCPP) runs for C3S

• other projects:

  • STAC prototype still under development at CEDA
  • putting PRs in for changes to enable "free text search" and "discoverable facets"

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos (DL):

CRIM (TL):

4. Discussion:

5. Agree next steps and next meeting

Next meeting date: Thursday 30th September (pushed back 1 week)

[agstephens]

Agenda - C3S_34e Meeting with Ouranos and CRIM: 26th August 2021

ACTIONS

• TS: Share details of the seg fault problem seen when using clisops in raven WPS.
• AS: Run the cf-checker on example files generated by Xarray to see if 1D coordinate variables with _FillValue set will fail cf-checks. Do we need to take action?

1. Minutes and actions from previous meeting

Previous minutes: #101

Previous actions

• AS: Put regridding higher up the agenda next month
• AS: move meetings back to 4-weekly from today - and edit meeting settings in Zoom
• ES/AS: Run the cf-checker on example files generated by Xarray to see if 1D coordinate variables with _FillValue set will fail cf-checks. Do we need to take action? - Ongoing
• ES: Create an issue on the xarray repository to explain the _FillValue in 1D coordinate variables, and that we believe these contravene the CF-Conventions (See "coordinate variable" section under: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#terminology). See: _FillValue added with output to netcdf pydata/xarray#5448
• MS: Raise an xESMF issue to discuss haloes

2. Project update: c3s_34e [Ag]

• quiet times over the summer:

  • still some xarray/numpy/dask memory errors on our service
  • working on deployment of fix for decadal (DCPP) runs for C3S

• PyWPS:

  • Security fix has been rolled out in v4.4

• other projects:

  • STAC prototype still under development at CEDA
  • putting PRs in for changes to enable "free text search" and "discoverable facets"

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos (DL):

• new collaborators - Uni College London; CERFACS:
  • xclim & icclim functionality
• regridding: small fixes
• improving bias-correction methods
• data portal dev:
  • implementing WFS functionality within cf_xarray
  • implements GeoJSON to Xarray
  • to work with station data - initially with point features
  • Possible server implementing WFS: https://www.ncei.noaa.gov/erddap/index.html
  • Possible related code: https://github.com/earthobservations/wetterdienst

CRIM (TL):

• supporting xclim (with Ouranos) to support the climate data portal
• answered OGC call for climate services:
  • machine learning
  • federated architectures
• working on CEDA logs - NLP analysis
• working on cataloguing (including STAC)
• implemented user workspaces
• developing weaver for execution workflow and processes:
  • following the OGC API processing standard
  • following Best Practices for EO applications
• experimenting with EO Data Cubes
• working with Natural Resources Canada - interfacing with EO data

4. Discussion:

TS: Have had to remove clisops from some projects, due to a problem:

• GDAL/geopandas operations in xESMF
• In a WPS implementation, seeing a seg fault
• Looks like an issue in ESMpy or xESMF

5. Agree next steps and next meeting

Next meeting date: Thursday 30th September (pushed back 1 week)

[Zeitsperre]

Thanks again for coordinating the meeting, Ag! Here are some resources to help document this strange behaviour:

Context: Ouranosinc/raven#397 (comment)

Segfault source: Ouranosinc/raven#397 (comment)
Stack trace: https://gist.github.com/Zeitsperre/d5d5f88be82835a47e48172f3bb7be16
Changes that were needed in RavenPy (reverted to a version that did not include libESMF/xESMF): https://github.com/CSHS-CWRA/RavenPy/pull/120/files

[agstephens]

Hi @Zeitsperre thanks for providing the links. We'll take a look.