Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDIP Data IO and Graphics (v2) [develop] #83

Merged
merged 94 commits into from
Mar 31, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
c016b31
Setup CI, v16.2, try to fix HDF5 issue, try fix outlined in Troublesh…
Matthew-Boyd Oct 29, 2021
fbce239
Setup CI, v16.3, try to fix HDF5 issue, try fix outlined in Troublesh…
Matthew-Boyd Oct 29, 2021
6e1a5bb
Setup CI, v16.4, try to fix HDF5 issue, try fix outlined in Troublesh…
Matthew-Boyd Oct 29, 2021
2b3a846
Setup CI, v16.5, try to fix HDF5 issue, try fix outlined in Troublesh…
Matthew-Boyd Oct 29, 2021
ccb2655
Setup CI, v16.6, try to fix HDF5 issue, try fix outlined in Troublesh…
Matthew-Boyd Oct 29, 2021
acdf864
Setup CI, v16.7, list installed toolboxes and addons
Matthew-Boyd Oct 29, 2021
9ac032f
Setup CI, v17, uninstall all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
026dcd4
Setup CI, v17.1, uninstall all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
0e01825
Setup CI, v17.2, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
768b788
Setup CI, v17.3, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
c08dd6e
Setup CI, v17.4, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
ae1719e
Setup CI, v17.5, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
e6f557b
Setup CI, v17.6, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
d257e54
Setup CI, v17.7, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
3931196
Setup CI, v17.8, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 29, 2021
36f2cf3
Setup CI, v17.9, toggle off all pre-installed toolboxes
Matthew-Boyd Oct 30, 2021
bf5c6e4
Setup CI, v17.10, list installed addons and toolboxes
Matthew-Boyd Nov 19, 2021
6fbcccb
Setup CI, v17.11, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
b2a4cb7
Setup CI, v17.12, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
5f88f7f
Setup CI, v17.13, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
911d772
Setup CI, v17.14, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
397a942
Setup CI, v17.15, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
1ea23f3
Setup CI, v17.16, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
915c814
Setup CI, v17.17, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
18b8f16
Setup CI, v17.18, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
6a3756c
Setup CI, v17.19, start uninstalling toolboxes
Matthew-Boyd Nov 19, 2021
05a7f7f
Setup CI, v17.20, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
2e6ef19
Setup CI, v17.21, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
4a1dce4
Setup CI, v17.22, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
57a4cc8
Setup CI, v17.23, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
f2e840f
Setup CI, v17.24, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
6e1ac94
Setup CI, v17.25, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
d8d344a
Setup CI, v17.26, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
e244857
Setup CI, v17.27, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
8dae0f6
Setup CI, v17.28, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
554c7cf
Setup CI, v17.29, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
37e48c7
Setup CI, v17.30, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
5468d0a
Setup CI, v17.31, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
7ad0acf
Setup CI, v17.32, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
3e8d32d
Setup CI, v17.33, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
351bc78
Setup CI, v17.34, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
b681055
Setup CI, v17.35, start uninstalling toolboxes
Matthew-Boyd Nov 20, 2021
5ce0479
Setup CI, v18, simplify to most basic test
Matthew-Boyd Nov 23, 2021
c6b43fa
Setup CI, v19, install mhkit toolbox
Matthew-Boyd Nov 23, 2021
6931cab
Setup CI, v20, run a single test that uses mhkit
Matthew-Boyd Nov 23, 2021
5025e57
Setup CI, v21, run a single different test that uses mhkit
Matthew-Boyd Nov 23, 2021
7dc9f9e
Setup CI, v21.1, run a single different test that uses mhkit
Matthew-Boyd Nov 23, 2021
bb95e9a
Setup CI, v21.2, run a single different test that uses mhkit
Matthew-Boyd Nov 23, 2021
15eefe0
Add function to reload python in matlab after python code changes
Matthew-Boyd Nov 24, 2021
0cce6cf
Fix paths to python usgs and noaa functions
Matthew-Boyd Nov 25, 2021
8c0a3f4
Update pierson_moskowitz_spectrum wrapper with additional Hs parameter
Matthew-Boyd Nov 25, 2021
50dba0d
Update create_spectra with additional Hs parameter
Matthew-Boyd Nov 25, 2021
32012fa
Update tests fixing mostly broken relative file paths
Matthew-Boyd Nov 25, 2021
d17c5a3
Merge QC_Test into master (#63)
parangat94 Jan 24, 2022
deedd7e
Bug fix io request data (#78)
Matthew-Boyd Feb 8, 2022
45e15fd
Move CI Unit Tests to GitHub Actions (#80)
H0R5E Feb 10, 2022
8bc87e8
Initial commit, WIP
Matthew-Boyd Jan 22, 2022
45d9a40
More progress, WIP
Matthew-Boyd Jan 23, 2022
b782ea3
Make first complete working version of cdip data collector
Matthew-Boyd Jan 24, 2022
ab54762
Refactor datum_categories function
Matthew-Boyd Jan 24, 2022
9b0a3b4
Rename categories to groups
Matthew-Boyd Jan 24, 2022
5863481
Alphabetize local functions
Matthew-Boyd Jan 24, 2022
f6cc2b6
Fix a couple issues
Matthew-Boyd Feb 9, 2022
a2ca6a9
Finish adding data tests, minus plot tests
Matthew-Boyd Feb 9, 2022
e59c70d
Add autoretry
Matthew-Boyd Feb 9, 2022
03a5ba2
Refactor autoretry
Matthew-Boyd Feb 9, 2022
84150b7
Add compendium plot and associated tests
Matthew-Boyd Feb 10, 2022
3b77eaa
Add box plot function, not needing statistical toolbox
Matthew-Boyd Feb 10, 2022
6906d73
Customize boxplot function to look more like Python plot
Matthew-Boyd Feb 10, 2022
86cc331
Refactor compendium plot
Matthew-Boyd Feb 10, 2022
4ac1228
Add boxplot and associated test
Matthew-Boyd Feb 10, 2022
df53a35
Handle when no times are given
Matthew-Boyd Feb 11, 2022
b038ae8
Make plots not open in new window when in live script
Matthew-Boyd Feb 12, 2022
2207391
Add example live script for CDIP
Matthew-Boyd Feb 12, 2022
ed70624
Make boxplot look more like Python plot
Matthew-Boyd Feb 12, 2022
4a31424
Make compendium plot look more like Python plot
Matthew-Boyd Feb 12, 2022
11d3996
Change plot time parameters from POSIX to datetime
Matthew-Boyd Feb 12, 2022
6d2892e
Remove optional nc object parameter
Matthew-Boyd Feb 12, 2022
05d4b6e
Fix docstring
Matthew-Boyd Feb 12, 2022
97e570d
resolving merge conflict
rpauly18 Feb 18, 2022
923f192
updating matlab toolbox
rpauly18 Feb 18, 2022
421eeba
Warn on non-existant parameters
Matthew-Boyd Feb 18, 2022
578a9d2
Fix spelling errors
Matthew-Boyd Feb 18, 2022
501ab98
Replace a try-catch block
Matthew-Boyd Feb 21, 2022
53f8a67
Merge branch 'cdip_off_develop' of github.com:MHKiT-Software/MHKiT-MA…
rpauly18 Feb 21, 2022
b5662ee
Add low-level netcdf cdip functions for older matlab versions
Matthew-Boyd Feb 25, 2022
9d2da86
Merge commit 'refs/pull/83/head' of github.com:MHKiT-Software/MHKiT-M…
rpauly18 Feb 25, 2022
54eea47
resolving merge conflict
rpauly18 Feb 25, 2022
c79fcda
Merge branch 'develop' into cdip_off_develop
Matthew-Boyd Feb 28, 2022
bd91a88
Merge branch 'cdip_off_develop' of https://github.com/MHKiT-Software/…
Matthew-Boyd Feb 28, 2022
2e66730
Reduce query size of USGS instantaneous data test to avoid query limit
Matthew-Boyd Mar 22, 2022
1cd75f7
Remove USGS instantaneous data test to stop failed CI runs
Matthew-Boyd Mar 25, 2022
dd7c7d1
Remove Python 3.7 CI jobs as it is no longer supported by hosted runners
Matthew-Boyd Mar 25, 2022
4437be7
Remove boxplot legend dependency on data file
Matthew-Boyd Mar 25, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
fail-fast: false
matrix:
os: [macos-latest, ubuntu-latest, windows-latest]
python-version: [3.7, 3.8, 3.9]
python-version: [3.8, 3.9]
runs-on: ${{ matrix.os }}
env:
mhkit-python-dir: 'MHKiT-Python'
Expand Down
Binary file added examples/cdip_example.mlx
Binary file not shown.
Binary file modified mhkit.mltbx
Binary file not shown.
7 changes: 0 additions & 7 deletions mhkit/tests/River_TestIO.m
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,5 @@ function test_request_usgs_data_daily(testCase)
assertEqual(testCase,data.units, struct('Discharge', " cubic feet per second"));
assertEqual(testCase,size(data.Discharge), [10 1]);
end

function test_request_usgs_data_instant(testCase)
data=request_usgs_data("15515500","00060","2009-08-01","2009-08-10","data_type",'Instantaneous');

assertEqual(testCase,data.units, struct('Discharge'," cubic feet per second"));
assertEqual(testCase,size(data.Discharge), [10*24*4 1]);
end
end
end
146 changes: 146 additions & 0 deletions mhkit/tests/Wave_TestIOcdip.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
classdef Wave_TestIOcdip < matlab.unittest.TestCase
%WAVE_TESTIOCDIP Testing CDIP data querying and formatting
% The Coastal Data Information Program (CDIP) measures, analyzes,
% archives and disseminates coastal environment data.
% https://cdip.ucsd.edu/

methods (Test)

function test_get_netcdf_variables_all2Dvars(testCase)
vars2D = {'waveEnergyDensity', 'waveMeanDirection', ...
'waveA1Value', 'waveB1Value', 'waveA2Value', ...
'waveB2Value', 'waveCheckFactor', 'waveSpread', ...
'waveM2Value', 'waveN2Value'};

data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'years', 1996, ...
'all_2D_variables', true);

assertTrue(testCase, all(ismember(vars2D, fieldnames(data.data.wave2D))));
end

function test_get_netcdf_variables_params(testCase)
parameters = {'waveHs', 'waveTp', 'notParam', 'waveMeanDirection'};

data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'years', 1996, ...
'parameters', parameters);

assertTrue(testCase, all(ismember({'waveHs', 'waveTp'}, ...
fieldnames(data.data.wave))));
assertTrue(testCase, all(ismember({'waveMeanDirection'}, ...
fieldnames(data.data.wave2D))));
assertTrue(testCase, all(ismember({'waveFrequency'}, ...
fieldnames(data.metadata.wave))));
end

function test_get_netcdf_variables_time_slice(testCase)
start_date = '1996-10-01';
end_date = '1996-10-31';

data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'start_date', start_date, ...
'end_date', end_date, ...
'parameters', {'waveHs'});

start_dt = datetime(start_date, ...
'InputFormat', 'yyyy-MM-dd', ...
'TimeZone', 'UTC');
end_dt = datetime(end_date, ...
'InputFormat', 'yyyy-MM-dd', ...
'TimeZone', 'UTC');
assertTrue(testCase, data.data.wave.waveTime(end) <= end_dt);
assertTrue(testCase, data.data.wave.waveTime(1) >= start_dt);
end

function test_request_parse_workflow_multiyear(testCase)
station_number = '067';
year1 = 2011;
year2 = 2013;
years = [year1, year2];
parameters = {'waveHs', 'waveMeanDirection', 'waveA1Value'};
data = cdip_request_parse_workflow( ...
'station_number', station_number, ...
'years', years, ...
'parameters', parameters);

expected_index0 = datetime(year1, 1, 1, 'TimeZone', 'UTC');
expected_index_final = datetime(year2, 12, 31, 'TimeZone', 'UTC');

assertEqual(testCase, dateshift( ...
data.data.wave.waveTime(1), 'start', 'day'), ...
expected_index0);
assertEqual(testCase, dateshift( ...
data.data.wave.waveTime(end), 'start', 'day'), ...
expected_index_final);
end

function test_request_parse_workflow_no_times(testCase)
station_number = '100';
data_type = 'historic';
data = cdip_request_parse_workflow( ...
'station_number', station_number, ...
'data_type', data_type);

assertEqual(testCase, data.data.wave.waveTime(1), ...
datetime('30-Jan-2001 00:17:11', 'TimeZone', 'UTC'));
end

function test_plot_boxplot(testCase)
data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'years', 2011, ...
'parameters', {'waveHs'}, ...
'all_2D_variables', false);
fig = plot_boxplot( ...
data.data.wave.waveHs, ...
data.data.wave.waveTime);

filename = fullfile(tempdir, 'wave_plot_boxplot.png');
saveas(fig, filename);
assertTrue(testCase, isfile(filename));
delete(filename);
end

function test_plot_compendium(testCase)
data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'years', 2011, ...
'parameters', {'waveHs', 'waveTp', 'waveDp'}, ...
'all_2D_variables', false);
fig = plot_compendium( ...
data.data.wave.waveHs, ...
data.data.wave.waveTp, ...
data.data.wave.waveDp, ...
data.data.wave.waveTime);

filename = fullfile(tempdir, 'wave_plot_compendium.png');
saveas(fig, filename);
assertTrue(testCase, isfile(filename));
delete(filename);
end

function test_plot_compendium2(testCase)
data = cdip_request_parse_workflow( ...
'station_number', '067', ...
'start_date', '2011-03-03', ...
'end_date', '2011-03-31', ...
'parameters', {'waveHs', 'waveTp', 'waveDp'}, ...
'all_2D_variables', false);
fig = plot_compendium( ...
data.data.wave.waveHs, ...
data.data.wave.waveTp, ...
data.data.wave.waveDp, ...
data.data.wave.waveTime);

filename = fullfile(tempdir, 'wave_plot_compendium2.png');
saveas(fig, filename);
assertTrue(testCase, isfile(filename));
delete(filename);
end
end
end

2 changes: 1 addition & 1 deletion mhkit/tests/Wave_TestResourceMetrics.m
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,7 @@ function test_plot_environmental_contour_multiyear(testCase)

relative_file_name= '../../examples/data/wave/Hm0_Te_46022.json';
full_file_name = fullfile(fileparts(mfilename('fullpath')), relative_file_name);

fid = fopen(full_file_name); % Opening the file
raw = fread(fid,inf); % Reading the contents
str = char(raw'); % Transformation
Expand Down
Loading