Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Climate reader functionality #47

Draft
wants to merge 9 commits into
base: remove-lfs
Choose a base branch
from
Draft

Climate reader functionality #47

wants to merge 9 commits into from

Conversation

crvernon
Copy link
Member

@crvernon crvernon commented Jul 18, 2019

TASK:

Create a reader to ingest standard climate data formats in 3D [day, latitude, longitude] to the format and units required by Xanthos [land_cell, month].

COMPLETED:

  • Built functionality to read in raw climate data commonly used and generated by ISIMIP
  • Currently built functionality for pr, tas, tasmin, tasmax
  • Test suite included

USAGE:

To save formatted outputs as datasets:

from xanthos import ClimateToXanthos
ctx = ClimateToXanthos(save_outputs=True)

# for precipitation ('pr')
pr_arr = ctx.format_pr(ncdf="<path to NetCDF file>",
                                      data_variable_name='pr',
                                      start_yr=<int four digit year>,
                                      through_yr=<int four digit year>,
                                      yr_interval=1,
                                      out_file="<path and name to save NPY file to>")

# for `tas`
tas_arr = ctx.format_tas(ncdf="<path to NetCDF file>",
                                         data_variable_name='tas',
                                         out_file="<path and name to save NPY file to>")

@crvernon crvernon requested a review from rplzzz July 18, 2019 16:13
@crvernon crvernon requested a review from FeralFlows July 24, 2019 18:21
Copy link
Contributor

@rplzzz rplzzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The basic approach here looks sound, but it's failing one of its tests. What appears to be happening is that we are producing a tuple where a scalar value is expected:

Traceback (most recent call last):
  File "/home/travis/build/JGCRI/xanthos/xanthos/test/test_equal_outputs.py", line 29, in testOutputs
    res = xth.execute()
  File "/home/travis/build/JGCRI/xanthos/xanthos/model.py", line 94, in execute
    results = config_runner.run()
  File "/home/travis/build/JGCRI/xanthos/xanthos/configurations.py", line 130, in run
    c.hydropower_actual()
  File "/home/travis/build/JGCRI/xanthos/xanthos/components.py", line 427, in hydropower_actual
    HydropowerActual(self.s, self.Avg_ChFlow)
  File "/home/travis/build/JGCRI/xanthos/xanthos/hydropower/actual.py", line 81, in __init__
    self.hydro_sim()
  File "/home/travis/build/JGCRI/xanthos/xanthos/hydropower/actual.py", line 183, in hydro_sim
    self.init_sim()
  File "/home/travis/build/JGCRI/xanthos/xanthos/hydropower/actual.py", line 125, in init_sim
    self.q_states = self.inflow.groupby(self.inflow.index.month).quantile((0, 0.2375, 0.4750, 0.7125, 0.95, 1))
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 1908, in quantile
    interpolation=interpolation,
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 2248, in _get_cythonized_result
    func(**kwargs)  # Call func to modify indexer values in place
  File "pandas/_libs/groupby.pyx", line 692, in pandas._libs.groupby.group_quantile
TypeError: must be real number, not tuple

One thing that might prove useful in future developments along these lines: the ISIMIP files that we use contain a coordinate variable called time_bnds, which gives the start and end time of each time slice in the dataset. For determining the length of the months, time_bnds[:, 1] - time_bnds[:,0] is potentially more convenient than doing a bunch of calendrical calculations, and it avoids making assumptions about the type of calendar in use.

@rplzzz
Copy link
Contributor

rplzzz commented May 11, 2020

@crvernon It looks like you still have me marked as a reviewer on this PR. I'm guessing this is an oversight, though I'm happy to give you my thoughts if you want them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants