Added a recipe for converting a gridded rainfall dataset to zarr #5
Conversation
…nment constraints
| # This file may be used to create an environment using: | ||
| # $ conda create --name <env> --file <this file> | ||
| # platform: linux-64 | ||
| @EXPLICIT |
There was a problem hiding this comment.
Use of a "spec file" in this way is new to me!
It's worth being explicit about intended setup - i naively ran "conda env create -n gear" which threw an error - it's probably better not to call the spec file environment.yml?
@jmarshrossney has put a lot of thought into environment management with complex dependencies as a form of PhD procrastination, and may have suggestions here
There was a problem hiding this comment.
Thanks for tagging me. It is true that I have often gazed into the hellscape that is python packaging as a form of ritual self punishment for my waning interest in the research I was doing.
As far as I know conda list --explicit is good unless you're installed anything using pip, but as @metazool says should probably be in a packages.txt of something instead of an environment.yml.
Beyond that, happy to look at this together at a time convenient to both of us, in a new PR :)
| T, | ||
| ) | ||
|
|
||
| startyear = 1990 |
There was a problem hiding this comment.
Even at a prototype stage I'd start where you want to end up and introduce a config file to hold paths and parameters here.
omegaconf could be suitable
hydra - saw this recently being adopted by the Minerva project which my old team collaborated on. It's not clear to me how one benefits from both!
Either way, a standard-enough, no-thinking-required way of other people managing their configuration separately from your code :D
There was a problem hiding this comment.
If suggestions are welcome I'd say just argparse is probably sufficient for this, and means
- Fewer dependencies to potentially fight among each other
- Fewer non-standard tools to distract from the main learning points
Easy to upgrade to something like omegaconf down the line if the need for something more sophisticated arises!
| # Here we convert some of the variables in the file | ||
| # to coordinate variables so that pangeo-forge-recipes | ||
| # can process them | ||
| print(f'Preprocessing before {ds =}') |
There was a problem hiding this comment.
A fun SO discussion here about "why use logging instead of print"
I know it's just a script and you can probably find print statements in my code without looking very hard :) - use of logging instead is a good habit to build though!
My use of it tends to be basic, with logging.basicConfig(level=logging.INFO) just under the imports, then logging.info wherever i'd print
See NERC-CEH#9