Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Github Actions; Preliminary Documentation #6

Merged
merged 39 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
276e400
test files added
RickytheGuy Aug 3, 2023
1a499b0
environment
RickytheGuy Aug 3, 2023
1e922a6
added simple test github actions
RickytheGuy Aug 3, 2023
09fd515
allow nc files
RickytheGuy Aug 3, 2023
7bc5d94
github tests
RickytheGuy Aug 3, 2023
9d88212
test actions
RickytheGuy Aug 7, 2023
0d4e635
Merge branch 'geoglows:main' into main
RickytheGuy Aug 7, 2023
a54430f
manual dispatch
RickytheGuy Aug 7, 2023
3247edb
Merge branch 'main' of https://github.com/RickytheGuy/rapid-inflows
RickytheGuy Aug 7, 2023
8e10efd
updated env
RickytheGuy Aug 7, 2023
b641c75
fixed env file
RickytheGuy Aug 7, 2023
f4389d3
update
RickytheGuy Aug 7, 2023
db87caf
updated test script
RickytheGuy Aug 7, 2023
440881a
updated yml
RickytheGuy Aug 7, 2023
59c7cd4
updated tests
RickytheGuy Aug 7, 2023
c5dd869
fixed path
RickytheGuy Aug 7, 2023
550ff97
updates
RickytheGuy Aug 7, 2023
7f29baa
completed test comparison
RickytheGuy Aug 7, 2023
c38339b
updated path names
RickytheGuy Aug 7, 2023
b0fc173
formatted
RickytheGuy Aug 7, 2023
aa76c47
updated path
RickytheGuy Aug 7, 2023
767efed
Create README.md
RickytheGuy Aug 7, 2023
b35245f
updated readme
RickytheGuy Aug 7, 2023
b13efe5
removed bioconda, package versions
RickytheGuy Aug 8, 2023
6f0d62c
Update test_compare.py
RickytheGuy Aug 8, 2023
a91e397
Update README.md
RickytheGuy Aug 8, 2023
5327612
Update test.yml, removed bionconda
RickytheGuy Aug 8, 2023
af0ed5a
updated .gitignore
RickytheGuy Aug 8, 2023
927c487
added to readme
RickytheGuy Aug 8, 2023
d569360
provided functinality for multiple wts
RickytheGuy Aug 8, 2023
85932ff
additional tests
RickytheGuy Aug 8, 2023
ec9fecf
more test files
RickytheGuy Aug 8, 2023
c6f24d8
Merge branch 'main' of https://github.com/RickytheGuy/rapid-inflows
RickytheGuy Aug 8, 2023
0476ec0
renamed
RickytheGuy Aug 8, 2023
4a85b1c
check weight table dims vs. xr dims
RickytheGuy Aug 8, 2023
075c092
updated readme for weigh table
RickytheGuy Aug 8, 2023
7797942
removed multiple wts functionality
RickytheGuy Aug 9, 2023
2471ad1
removed old paths
RickytheGuy Aug 9, 2023
7af4d6c
removed push
RickytheGuy Aug 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 3 additions & 11 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,7 @@
name: Test Python Script

on:
push:
branches:
- rapid-inflows
pull_request:
branches:
- rapid-inflows
workflow_dispatch:
branches:
- rapid-inflows
[push, pull_request, workflow_dispatch]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would be ok with on pull request or workflow_dispatch. Do you think we need to do it on every push?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is neccesary. It's nice since it automatically tests every push, but workflow_dispatch could make it obsolete since we can run the test whenever we want. If you want me to remove it I can


jobs:
test:
Expand All @@ -32,11 +24,11 @@ jobs:
uses: conda-incubator/setup-miniconda@v2
with:
mamba-version: "*"
channels: conda-forge,bioconda,defaults
channels: conda-forge,defaults
auto-activate-base: false
activate-environment: inflows
environment-file: envs/environment.yml

- name: Run Tests
run: |
python ./tests/test_compare.py
python ./tests/test_compare.py
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
*.pyc
example.py
test.nc
test*.nc
.DS_store
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# rapid_inflows
Rapid Inflows is a python interface for RAPID that assists to prepare inflow data. Gridded runoff data is taken in, along with lists of basins and their area. The runoff for each basin is calculated for every time step, and the resulting flows are written to an output inflow datset, the main dataset used by RAPID. More information about RAPID can be found [here](http://rapid-hub.org).
Louis R. Rosas, Dr Riley Hales

Rapid Inflows is a python interface for RAPID that assists to prepare inflow data. Gridded runoff data is taken in, along with lists of basins and their area. The runoff for each basin is calculated for every time step, and the resulting flows are written to an output inflow NetCDF file, the main dataset used by RAPID. Inspired by ERDC's [RAPIDpy](https://github.com/erdc/RAPIDpy). More information about RAPID can be found [here](http://rapid-hub.org).

## Inputs
The `create_inflow_file` function takes in four parameters:
Expand All @@ -9,7 +11,7 @@ The `create_inflow_file` function takes in four parameters:
4) an output filename

### Gridded Runoff Datasets
The input netcdf datasets should follow the following conventions. It is expected that the datasets have 3 or 4 dimensions, in the following orders: `time, latitude, longitude` or `time, expver, latitude, longitude` (The 4th dimension is often present in the most recent releases of the ECMWF datasets, which has an 'experimentl version' dimension. Internally, this dimension is flattened and the data summed. the name of this dimension does not matter). The runoff, longitude, and latitude variables may be any of the following, respectively: `'ro', 'RO', 'runoff', 'RUNOFF'`, `'lon', 'longitude', 'LONGITUDE', 'LON'`, `'lat', 'latitude', 'LATITUDE', 'LAT'`. Latitude and longitude is expected in even steps, from 90 to -90 degrees and 0 to 360 degrees repectively. Time is expected as `'time'`. The difference in time between each dataset should be equal (typically a time step onf a day is used), and should be an interger.
The input netcdf datasets should follow the following conventions. It is expected that the datasets have 3 or 4 dimensions, in the following orders: `time, latitude, longitude` or `time, expver, latitude, longitude` (The 4th dimension is often present in the most recent releases of the ECMWF datasets, which has an 'experimental version' dimension. Internally, this dimension is flattened and the data summed. The name of this dimension does not matter). The runoff, longitude, and latitude variables may be any of the following, respectively: `'ro', 'RO', 'runoff', 'RUNOFF'`, `'lon', 'longitude', 'LONGITUDE', 'LON'`, `'lat', 'latitude', 'LATITUDE', 'LAT'`. Latitude and longitude is expected in even steps, from 90 to -90 degrees and 0 to 360 degrees repectively. Time is expected as `'time'`. The difference in time between each dataset should be equal (typically a time step of a day), and should be an interger.

The user may input as many datasets as desired. There is a built in memory check which warns the user if the amount of memory required exceeds 80% of the available RAM and will terminate the process if the amount of memory required exceeds all the available memory.

Expand All @@ -28,9 +30,9 @@ This file can be generated using either [these ArcGIS tools](https://github.com/
An example weight table can be found in `/tests/inputs`.

### Comid csv
A csv which has the unique ids (COMIDs, reach ids, LINKNOs, etc.) of the basins/streams modeled in its first column, sorted in the order in which the user wants RAPID to process. This file can be generated using either [these ArcGIS tools](https://github.com/Esri/python-toolbox-for-rapid) or [these python scripts](https://github.com/geoglows/tdxhydro-rapid).
A csv which has the unique ids (COMIDs, reach ids, LINKNOs, etc.) of the basins/streams modeled in its first column, sorted in the order in which the user wants RAPID to process. Two other columns must be present, labeled `'lat'` and `'lon'`. These columns represent a point related to each river reach. This file can be generated using either [these ArcGIS tools](https://github.com/Esri/python-toolbox-for-rapid) or [these python scripts](https://github.com/geoglows/tdxhydro-rapid).

An example comid csv can be found in `/tests/inputs`.

## Outputs
The `create_inflow_file` outputs a new netcdf 3 (classic) dataset.
The `create_inflow_file` outputs a new netcdf 3 (classic) dataset. It is desgined to be accepted by rapid and pass all of its internal tests. Example output datasets can be found in `/tests/validation`.
19 changes: 9 additions & 10 deletions envs/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,13 @@ name: inflows
channels:
- defaults
- conda-forge
- bioconda
dependencies:
RickytheGuy marked this conversation as resolved.
Show resolved Hide resolved
- dask=2023.6.0
- netcdf4=1.6.2
- numpy=1.25.0
- pandas=1.5.3
- psutil=5.9.0
- pyarrow=11.0.0
- pytest=7.4.0
- python=3.11.4
- xarray=2023.6.0
- dask
- netcdf4
- numpy
- pandas
- psutil
- pyarrow
- pytest
- python
- xarray
Loading