Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bnb/dh refactor #220

Merged
merged 433 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
433 commits
Select commit Hold shift + click to select a range
bc6dc6b
some upper -> lower case changes to just reduce some local warnings. …
bnb32 Jun 27, 2024
d8b8f9f
fix: deriver caching test
bnb32 Jun 27, 2024
32ef4de
removing some np.array_equal vs dask warnings
bnb32 Jun 27, 2024
7e53679
tf run eagerly to get accurate pytest-cov report on solar model
bnb32 Jun 27, 2024
6532057
removed duplicate arg
bnb32 Jul 1, 2024
605b6d6
fix: missing doc string
bnb32 Jul 1, 2024
9657848
\b escape for nicer click help message formatting
bnb32 Jul 1, 2024
37d98e9
linting
bnb32 Jul 1, 2024
af3f6f3
removed funky sample counter test
bnb32 Jul 1, 2024
9033b47
safe_serialize method for handling objects like datetimeindex in json…
bnb32 Jul 10, 2024
d56a0ed
need to pass time_index to bias_transforms as list bc it gets convert…
bnb32 Jul 10, 2024
4660b5e
lower casing in docs
bnb32 Jul 10, 2024
f2986f1
fixed n_chunk calc in tests
bnb32 Jul 10, 2024
dea1b60
redudant code: moved cmd log to add_cmd_status.
bnb32 Jul 10, 2024
eb7dc4a
swapping out time_index arg for date_range_kwargs. Time indices can b…
bnb32 Jul 10, 2024
22ebb83
need string rep of timestamp
bnb32 Jul 10, 2024
8d33713
some fwp strat init cleaning
bnb32 Jul 11, 2024
7c5cb36
rename: exo_kwargs -> exo_handler_kwargs.
bnb32 Jul 12, 2024
cc67308
yml linting?
bnb32 Jul 12, 2024
b8e1f92
refact: removed getattr -> fwp strat in fwp, removed output handler a…
bnb32 Jul 12, 2024
e5c7221
refact: split collector classes into collectors dir with nc, base, h5…
bnb32 Jul 12, 2024
2df1c0e
todo flag added for generalizing pool kick off
bnb32 Jul 12, 2024
d27dca4
fix: circular imports from splitting collectors
bnb32 Jul 12, 2024
c6503ae
refact: removed uneeded attrs in ForwardPass
bnb32 Jul 12, 2024
ac3f176
feat: added compute call if dask before fwp generator call
bnb32 Jul 12, 2024
3721cde
feat: skip preflight for fwp strat on head nodes.
bnb32 Jul 12, 2024
4d0fc4d
refact: cleaned up fwp strategy init and setup skip of some init rout…
bnb32 Jul 13, 2024
5b4ba1f
fix: missing time index in input handler when features = []
bnb32 Jul 13, 2024
2651f55
refact: _parse_time_slice
bnb32 Jul 15, 2024
6ab0873
fix: len(strat.node_chunks) -> n_chunks
bnb32 Jul 15, 2024
3f7d7c3
adds: repeat topo over time to be consistently 3d across features.
bnb32 Jul 15, 2024
28ee2f0
refact: simplifying stats compute methods.
bnb32 Jul 15, 2024
d435804
add: __rmul__ for multipliying Sup3rX objects by weights
bnb32 Jul 16, 2024
45383f6
fix: lower exo feature names
bnb32 Jul 16, 2024
10dc3af
add: writing to tmp file first for cacher
bnb32 Jul 16, 2024
e6132b7
fix: pr test actions
bnb32 Jul 16, 2024
31ea00f
stupid gh actions
bnb32 Jul 16, 2024
1d0bada
added `sample` method to quickly access data for batching. updated fr…
bnb32 Jul 17, 2024
eb68749
missed imports for np.random update
bnb32 Jul 17, 2024
60e48f9
removed unused code in accessor and sup3rdataset
bnb32 Jul 17, 2024
3d4a027
fix: height interp mask fails when multiple values match min distance.
bnb32 Jul 18, 2024
dd6a0ee
ensuring np.float32 for height interp
bnb32 Jul 18, 2024
41b9cb1
removed default device from model load (now included in params). some…
bnb32 Jul 19, 2024
ffb4d5c
some linting clean up
bnb32 Jul 19, 2024
53069e4
cyclic import fix
bnb32 Jul 19, 2024
ef76b6e
rebasing on time dependent qdm merge
bnb32 Jul 19, 2024
8fc0600
linting
bnb32 Jul 19, 2024
fbe8e86
fix: using set() put features in the wrong order.
bnb32 Jul 19, 2024
6be9677
trying "synchronize" to trigger tests
bnb32 Jul 19, 2024
7c9b991
default should trigger tests. so heres a check
bnb32 Jul 19, 2024
a58d720
dtype = np.float32 checks for height interp. qdm test fixes after rebase
bnb32 Jul 20, 2024
b1b5bf5
fix: ill defined gen config fixture
bnb32 Jul 20, 2024
d3dd59b
pickle doesnt like un-computed mask from argmin. added compute call. …
bnb32 Jul 21, 2024
43fb3b3
enforcing same rng state across all tests. some logging added to era …
bnb32 Jul 22, 2024
b9e65f4
dont need legacy np.random call
bnb32 Jul 22, 2024
bf21ea6
additional test for dual feature sets
bnb32 Jul 22, 2024
a1624b6
moved normalization to outside of batch iteration. more performant.
bnb32 Jul 23, 2024
76b95c4
stats args removed from batch queue. this is now handled just by stat…
bnb32 Jul 23, 2024
88819d6
normalization fixes
bnb32 Jul 23, 2024
744939c
rebase fixes on presrat
bnb32 Jul 23, 2024
6c7e3cd
methods in bias transforms currently need eager loading for some arra…
bnb32 Jul 24, 2024
e48cc9d
enabled loading of bc factor files with Extracter class. currently co…
bnb32 Jul 24, 2024
9fc1e15
get source type hdf -> h5
bnb32 Jul 24, 2024
9b250fb
unused fixture in qdm tests. date_range_kwargs in presat tests. datas…
bnb32 Jul 24, 2024
f38133c
check for time independence during data extraction
bnb32 Jul 24, 2024
0fa79aa
test fixes
bnb32 Jul 24, 2024
51802a1
moved bias correction to `init_chunk` instead of performing on full d…
bnb32 Jul 25, 2024
68358c8
changed samplers to sample batches using adjacent time slices which a…
bnb32 Jul 27, 2024
54f1828
sampler updates for tests. rex version req bump
bnb32 Jul 27, 2024
6df02bf
reducing loading mem by casting elevation to dask array. moved duplic…
bnb32 Jul 27, 2024
a499420
slight change with cdapi update
bnb32 Jul 27, 2024
3256c64
moved loader name maps and Dimension class to names.py. accounted for…
bnb32 Jul 28, 2024
0dc8928
more efficient era downloader writing. incremental stats - will load …
bnb32 Jul 28, 2024
6120a31
Bump `rex` req
ppinchuk Jul 29, 2024
6fcd406
Deprecate `Regridder` implementation
ppinchuk Jul 29, 2024
f552ddb
Drop regridding test (now in `rex`)
ppinchuk Jul 29, 2024
0d0dda2
Update import
ppinchuk Jul 29, 2024
e1b8f87
Remove unused import
ppinchuk Jul 29, 2024
c72c817
Merge pull request #228 from NREL/pp/rex_regridder
ppinchuk Jul 29, 2024
43e6242
np.asarray replacement for _compute_if_dask in most places. added fwp…
bnb32 Jul 30, 2024
d51d887
Loaders and loader utils used in era_downloader to remove duplicate c…
bnb32 Jul 30, 2024
80dbb5c
-rename: extracter -> rasterizer
bnb32 Jul 31, 2024
de031fe
cleaned up signature and doc parsing utils and tests. gaps version bump
bnb32 Jul 31, 2024
20b4fa2
gaps version bump.
bnb32 Aug 1, 2024
1b18ed4
log_args decorator in era downloader. removed manual logging
bnb32 Aug 1, 2024
53f6968
simplified auto gen of composite signatures and docs. moved some logi…
bnb32 Aug 1, 2024
725c880
sup3rmeta subclasscheck update
bnb32 Aug 1, 2024
bb2c0c1
type hint fix
bnb32 Aug 1, 2024
27f1eae
test fixes
bnb32 Aug 1, 2024
749633b
nc for cc shouldnt actually have base loader arg. fixed order of comp…
bnb32 Aug 1, 2024
250a0fe
test on units persisting through load and temp change to celsius. rem…
bnb32 Aug 3, 2024
8bd542d
rh look up tests for nc cc handler
bnb32 Aug 3, 2024
c045b81
general exo rasterizers for nc and h5 and any feature, with Sup3rX me…
bnb32 Aug 4, 2024
a27110e
added recursive derivation. simplified exo handling: get highest res …
bnb32 Aug 5, 2024
fbde72e
self reference in feature registry leads to infinite recursion - adde…
bnb32 Aug 6, 2024
3c9c72c
sup3rwind citation update
bnb32 Aug 6, 2024
a707019
good error from 3.8 - bad dims in nc writer
bnb32 Aug 6, 2024
33cc102
readme updates and test fixes
bnb32 Aug 6, 2024
9b1f29b
missed parse time slice in fwp strat
bnb32 Aug 6, 2024
a6acb90
fix: broken rename
bnb32 Aug 6, 2024
40e5021
cleaned up `__getitem__` and added `.values` for @grantbuster :) also…
bnb32 Aug 7, 2024
211976f
need to return dataarray when ellipsis in `__getitem__` keys
bnb32 Aug 7, 2024
fe47f3e
more __getitem__ simplifying
bnb32 Aug 7, 2024
71073d2
keys parsing fix in sup3rx getitem
bnb32 Aug 7, 2024
d04e6e7
older xarray doesnt have to_dataarray()
bnb32 Aug 7, 2024
3d4114e
edge case in getitem. spatial only feature caching / regridding
bnb32 Aug 7, 2024
47d1fe7
custom types removed.
bnb32 Aug 7, 2024
d8e0fe2
simplified args for `Sup3rDataset`. Single check for tuples in `Conta…
bnb32 Aug 8, 2024
acea804
bad type check fix
bnb32 Aug 8, 2024
0e32ff5
xr.Dataset.load() method conflict with Loader. change latter to _load()
bnb32 Aug 9, 2024
ca52250
sza derived feature added. wrapped some function from rex class with …
bnb32 Aug 11, 2024
a708443
use dask solar zenith in sza exo rasterizer
bnb32 Aug 11, 2024
b3f8260
out of core writing to netcdf with parallel delayed dask tasks
bnb32 Aug 11, 2024
90ecb90
dont need threadpool for cacher with dask delayed tasks
bnb32 Aug 11, 2024
05166f9
netcdf version req
bnb32 Aug 11, 2024
0b26b69
netcdf4 req added
bnb32 Aug 12, 2024
c5f0dec
with netcd4f4 installed engine='h5netcdf' needs to be explicit. befor…
bnb32 Aug 12, 2024
5805bd4
compute needed in qa for export with RexOutput
bnb32 Aug 12, 2024
bde79ed
test fixes, netcd4 version req cap, and xr open mfdataset wrapper wit…
bnb32 Aug 12, 2024
009edd5
one annoying test remaining
bnb32 Aug 12, 2024
2ae0cbc
delayed dask and schedulers used to replace threadpoolexecutor logic.…
bnb32 Aug 12, 2024
77c36ca
added initial meta / time_index building from only unique spatial / t…
bnb32 Aug 15, 2024
e6ed04a
collection speed up fix. attrs copy update for rex update. rex versio…
bnb32 Aug 16, 2024
a8a7e28
caching h5 with lat / lon instead of /meta/lat and /meta/lon.
bnb32 Aug 16, 2024
6bbdf98
more epochs for training test
bnb32 Aug 16, 2024
9957800
not caching exo data with time if time independent. this can be a hug…
bnb32 Aug 18, 2024
0f34e35
bottleneck dependency for fast ffill operation, and others.
bnb32 Aug 18, 2024
3117212
added method to automatically determine exogenous data steps from lr_…
bnb32 Aug 18, 2024
78e69ce
test fixes
bnb32 Aug 18, 2024
a00fdaf
test fix
bnb32 Aug 19, 2024
ca3b03e
changed masking logic in level interpolation to enable delayed comput…
bnb32 Aug 20, 2024
f619328
auto check on exo features needs model list from solar multi step gan.
bnb32 Aug 21, 2024
d719c59
run caching tests with max_workers=1
bnb32 Aug 21, 2024
dacd1d5
fixes for solar multi step gan + exo data
bnb32 Aug 22, 2024
9453b25
limit on padding removed since data is loaded as ndarray before paddi…
bnb32 Aug 22, 2024
9058109
type checking additions
bnb32 Aug 22, 2024
bafcb8d
test fixes
bnb32 Aug 22, 2024
9e71ed6
solar multistep fix and max_workers -> 1 for tests
bnb32 Aug 22, 2024
fccea44
default max_workers=1 for cacher
bnb32 Aug 22, 2024
7bac634
adjustment for ensemble downloads
bnb32 Aug 27, 2024
79a4602
h5_attrs -> output_attrs. used in enforce_limits in both netcdf and h…
bnb32 Aug 27, 2024
41f82dc
rsds added to output_attrs. other tests will pass when rex qdm_lim br…
bnb32 Aug 27, 2024
9430b02
moved calculation of k factor and tau into separate static methods fo…
grantbuster Jul 27, 2024
0c5f9b9
moved presrat transform into separate method for modular call directl…
grantbuster Jul 27, 2024
d56f70f
bug fix: wrong time index passed to run single and default relative s…
grantbuster Jul 28, 2024
3edbcef
more explicit enforcement of a zero threshold value and passing to QD…
grantbuster Jul 28, 2024
8ee6b2b
protect against divide by zero in calculation of k
grantbuster Jul 29, 2024
e413982
moved ntimesteps and window size to kwargs so easier to modify. Found…
grantbuster Jul 31, 2024
5657774
moved presrat class to its own module, merged new fixes with dh_refac…
grantbuster Aug 1, 2024
6596038
fixed presrat test - zero rate threshold can manipulate values now
grantbuster Aug 1, 2024
ca862eb
added outrange feature to qdm and presrat
grantbuster Aug 5, 2024
4314053
delta denom min should fix the low precip value issues and no need to…
grantbuster Aug 16, 2024
8d8474c
bug fix for daily cs ghi == 0 causing nan values in clearsky ratio
grantbuster Aug 16, 2024
1c1056d
added tmin/tmax to h5 attrs
grantbuster Aug 28, 2024
1fdf415
dont need to derive clearsky_ghi if needed feature is already in load…
bnb32 Aug 28, 2024
a7c4d1d
modfication to SolarCC model to enable running as stand alone forward…
bnb32 Aug 28, 2024
0d198b6
dont need separate status fle write. this is handled in get_node_cmd
bnb32 Aug 30, 2024
661e650
clean up docstrings
grantbuster Aug 30, 2024
83d6475
bump rex version requirement for QDM kwargs
grantbuster Aug 30, 2024
ce76db8
split up attribute collection for meta and time index. significantly …
bnb32 Aug 31, 2024
af2336b
redistribution of unfinished chunks needs some refactoring. reverting…
bnb32 Aug 31, 2024
7e90cb0
added collection test with more chunks
bnb32 Aug 31, 2024
c21ed3c
added time_shift argument to deriver and data handler so that daily d…
bnb32 Sep 4, 2024
a5f42b8
moved output attrs to json file and included in setup package data
grantbuster Sep 12, 2024
ae2324b
Merge branch 'bnb/dh_refactor' into gb/presrat_updates_2
bnb32 Sep 12, 2024
84d3be5
linting fix
bnb32 Sep 12, 2024
c692000
Merge pull request #230 from NREL/gb/presrat_updates_2
bnb32 Sep 12, 2024
7232eb1
regex edit for correctly getting chunk indices from solar module outp…
bnb32 Sep 13, 2024
1920fcd
increase padding for failing test
bnb32 Sep 13, 2024
3808db1
time_shift arg added to run_temporal_chunk so this can be provided in…
bnb32 Sep 14, 2024
e4f009f
time shift added to solar module, so that daily gcm data can be shift…
bnb32 Sep 12, 2024
9042364
12 default shift only meant for current solar module runs.
bnb32 Sep 16, 2024
2aa353b
default time shift as 0.
bnb32 Sep 16, 2024
7c8fe53
era downloader test fixes
bnb32 Sep 16, 2024
711c471
missed era downloader updates from other branch
bnb32 Sep 17, 2024
dd6250a
xr open mfdataset wrapper added
bnb32 Sep 17, 2024
8ae857d
dont need time shift in the solar module. can just shift this in a se…
bnb32 Sep 17, 2024
e4428fd
better node distribution for solar module
bnb32 Sep 19, 2024
7ec885a
removing todo note for better solar module node distributuon
bnb32 Sep 20, 2024
69e8344
removed expanded time dimension in exo rasterizer for time independen…
bnb32 Sep 21, 2024
cf5e466
exo test fix with removal of time dimension expansion for time indepe…
bnb32 Sep 21, 2024
e9b42e9
only use temporal slices to get exo data chunk if exo data is time de…
bnb32 Sep 21, 2024
2ea6206
need to check coords also for pressure derivation. fixed this in base…
bnb32 Sep 24, 2024
e7e44b1
missed default xr kwargs for combining monthly files
bnb32 Sep 24, 2024
92ee0f7
typo in xr_open_mfdataset logging
bnb32 Sep 24, 2024
c3a3309
added height interpolation for the case of just single level data. e.…
bnb32 Sep 26, 2024
b900d0d
wrong conditions on combining lev and var arrays in interpolation rou…
bnb32 Sep 27, 2024
cc9eaae
``.values`` method fix - returns ``as_array`` but loaded into memory.…
bnb32 Oct 14, 2024
71d4000
lowered bottleneck verrsion req
bnb32 Oct 14, 2024
94975f9
removed bottlenexck requirement
bnb32 Oct 15, 2024
9d5f23e
bias calc cli needs to be able to reference QDM and presrat
grantbuster Oct 15, 2024
9ea18ec
try preloading dh data - parallel compute is prohibitivly slow with l…
grantbuster Oct 16, 2024
df9ee4e
docs
grantbuster Oct 16, 2024
2d01ae9
preload kwarg and method for bias calc
grantbuster Oct 16, 2024
79a1e7a
attempted fix on numpy array indexing with preloaded data handler
grantbuster Oct 16, 2024
31a2f38
added test for various slicing operations on sup3rdataset with and wi…
grantbuster Oct 16, 2024
96fc946
check datahandler objects for compute method so rex classes dont quer…
grantbuster Oct 16, 2024
532eb4d
if chunks=None load into memory. This mimics the behavior of xr.open_…
bnb32 Oct 17, 2024
e6f7dc5
Merge pull request #233 from NREL/bnb/compute_on_none_chunks
grantbuster Oct 21, 2024
f864740
Merge pull request #232 from NREL/gb/bc_debug
grantbuster Oct 21, 2024
f31f716
fixed double bias correction of features in bc qdm utility
grantbuster Oct 22, 2024
e89362c
Merge pull request #234 from NREL/gb/bc_util
grantbuster Oct 22, 2024
41bb7e0
Added error catch on integer indexing of features
bnb32 Oct 22, 2024
99e197b
typo: pydata-sphinx-theme didn't reach 15.3 yet
castelao Oct 23, 2024
c84ff5e
cfg, typo: Wrong name of package 'sphinx-book-theme'
castelao Oct 23, 2024
f894e4f
cfg: Adding pydata-sphinx-theme as a conda requirement in pixi
castelao Oct 23, 2024
54fc028
cfg: Updating sphinx while locking minimum viable version
castelao Oct 23, 2024
9aa6cb1
cfg: Updating pixi.lock
castelao Oct 23, 2024
5d6e5d7
added bias future specific kwargs for qdm so time slice can be input …
grantbuster Oct 23, 2024
b4c30a8
fix bias fut kwargs attribute
grantbuster Oct 23, 2024
3ff59d0
Merge pull request #236 from NREL/fix/pydata_sphinx_theme_version
castelao Oct 23, 2024
4f59503
Access pattern changed so ``Sup3rX.__getitem__`` can only accept eith…
bnb32 Oct 23, 2024
6801f54
fixed bad kwargs in presrat test file.
bnb32 Oct 23, 2024
c9450d3
unpacking indices with * only works with python 3.11+
bnb32 Oct 23, 2024
43a04a0
updated indexing for bc cli tests
bnb32 Oct 23, 2024
c4a6a46
qa, rasterizers, and qdm index updates for test fixes
bnb32 Oct 23, 2024
0fff0be
if n_writes request is too big, set to n_chunks instead of raising error
grantbuster Oct 23, 2024
f53b275
Removing some logic from ``Sup3rX.__setitem__`` and ``Sup3rDataset.__…
bnb32 Oct 24, 2024
8a2ca31
changed values method to property to match xarray pattern
bnb32 Oct 24, 2024
cfddcfe
Merge pull request #235 from NREL/bnb/updated_indexing
bnb32 Oct 24, 2024
8a8ca48
Merge pull request #237 from NREL/gb/collect_chunks
grantbuster Oct 24, 2024
8d969f7
Caching with DailyDataHandler objects updated. These handlers have .d…
bnb32 Oct 24, 2024
18c942a
added simple test on cc batching for daily boundaries
grantbuster Oct 24, 2024
d719957
Merge pull request #238 from NREL/gb/check_daily_batching
grantbuster Oct 24, 2024
157c102
Checks added to caching and derivations so that coarsening, time roll…
bnb32 Oct 25, 2024
524dd11
updates to era_downloader from masked_fwp branch
bnb32 Oct 25, 2024
ef17fde
Merge pull request #239 from NREL/bnb/caching_fixes
bnb32 Oct 25, 2024
b03b838
fix: get_coords return 2d coord arrays when lat and lon have a single…
bnb32 Nov 1, 2024
91a37ed
doc string for data handler: added chunks=None loads data into memory…
bnb32 Nov 1, 2024
b8523d8
refactor: interp_method -> interp_kwargs (includes method and run_lev…
bnb32 Nov 2, 2024
0e95307
removed map blocks from lin interp method. was producing the wrong sh…
bnb32 Nov 2, 2024
b4a754e
small tweak in _lin_interp
bnb32 Nov 2, 2024
89fe26a
removed pressure -> level alias in methods
bnb32 Oct 28, 2024
2c82e8c
missed cherry pick for fixed height parsing.
bnb32 Nov 2, 2024
508fadc
height parsing fix
bnb32 Nov 3, 2024
a194f28
cdsapi url update. cachers additions with dataset specific attributes
bnb32 Nov 4, 2024
3f8802f
if the Rasterizer gets chunks = None it will init the Loader with chu…
bnb32 Nov 4, 2024
b8d75ec
got blockwise operations working with _lin_interp. squeaking out a li…
bnb32 Nov 2, 2024
9ac7518
fixed broken attr caching with nested dictionaries
bnb32 Nov 4, 2024
afe7d0b
added delta min kwarg to presrat bias correction fwp function
grantbuster Oct 31, 2024
a37b1e9
added logging statement to k factor in case range is set
grantbuster Nov 1, 2024
5dcc98d
fixed performance issue with data indexing for bias calculation - fas…
grantbuster Nov 1, 2024
f6c7992
added option for delta_range in qdm and presrat bias transform functions
grantbuster Nov 2, 2024
39aeb03
added info log for pre load data
grantbuster Nov 2, 2024
6672d4b
head node shouldnt need to pre-load with chunks=None
grantbuster Nov 2, 2024
ba3b226
bump rex requirement for delta_range kwargs for QDM functions
grantbuster Nov 4, 2024
8085a43
log shape of data being pre loaded into mem by sup3rx
grantbuster Nov 4, 2024
e5df5d0
bug fix for esoteric edge case with missing leap day in source data
grantbuster Nov 5, 2024
6e46d6c
added better exception if QDM/presrat returns NaN values
grantbuster Nov 5, 2024
b8ebeeb
Merge pull request #240 from NREL/gb/bc_kwargs
grantbuster Nov 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .flake8

This file was deleted.

2 changes: 1 addition & 1 deletion .github/linters/.flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[flake8]
ignore = E731,E402,F,W503
ignore = E731,E402,F,W503,E203
exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,bin/tmp/*
max-complexity = 12
4 changes: 3 additions & 1 deletion .github/linters/.python-lint
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ confidence=
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=
no-member,
attribute-defined-outside-init,
arguments-renamed,
unspecified-encoding,
consider-using-f-string,
Expand Down Expand Up @@ -498,4 +500,4 @@ known-third-party=enchant

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
overgeneral-exceptions=builtin.Exception
1 change: 1 addition & 0 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
VALIDATE_PYTHON_ISORT: false
VALIDATE_PYTHON_MYPY: false
VALIDATE_DOCKERFILE_HADOLINT: false
VALIDATE_GITHUB_ACTIONS: false
VALIDATE_JSCPD: false
VALIDATE_JSON: false
VALIDATE_MARKDOWN: false
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pull_request_tests.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Pytests

on:
on:
pull_request:
workflow_dispatch:

Expand Down Expand Up @@ -37,4 +37,4 @@ jobs:
python -m pip install .[test]
- name: Run pytest
run: |
python -m pytest -v --disable-warnings
python -m pytest -v --disable-warnings
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ nosetests.xml
coverage.xml
*.cover
.hypothesis/
coverage/lcov.info
lcov.info
*.png

# Translations
Expand Down Expand Up @@ -119,3 +121,6 @@ tags
# pixi environments
.pixi
*.egg-info

# test dirs
exo_cache
33 changes: 20 additions & 13 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-json
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: flake8
- id: mixed-line-ending
- repo: https://github.com/PyCQA/pylint
- id: check-json
- id: check-yaml
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: flake8
args: [--config, .github/linters/.flake8]
- id: mixed-line-ending
- repo: https://github.com/PyCQA/pylint
rev: v3.1.0
hooks:
- id: pylint
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.4.10
hooks:
- id: ruff
- id: pylint
args:
[
--rcfile,
.github/linters/.python-lint,
]
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.5.3
hooks:
- id: ruff
73 changes: 8 additions & 65 deletions .pylintrc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[MASTER]
[MAIN]

# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
Expand Down Expand Up @@ -55,76 +55,20 @@ confidence=
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=
no-name-in-module,
no-member,
attribute-defined-outside-init,
arguments-renamed,
unspecified-encoding,
consider-using-f-string,
# Defaults
print-statement,
parameter-unpacking,
unpacking-in-except,
old-raise-syntax,
backtick,
import-star-module-level,
raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
apply-builtin,
basestring-builtin,
buffer-builtin,
cmp-builtin,
coerce-builtin,
execfile-builtin,
file-builtin,
long-builtin,
raw_input-builtin,
reduce-builtin,
standarderror-builtin,
unicode-builtin,
xrange-builtin,
coerce-method,
delslice-method,
getslice-method,
setslice-method,
no-absolute-import,
old-division,
dict-iter-method,
dict-view-method,
next-method-called,
metaclass-assignment,
indexing-exception,
raising-string,
reload-builtin,
oct-method,
hex-method,
nonzero-method,
cmp-method,
input-builtin,
round-builtin,
intern-builtin,
unichr-builtin,
map-builtin-not-iterating,
zip-builtin-not-iterating,
range-builtin-not-iterating,
filter-builtin-not-iterating,
using-cmp-argument,
div-method,
idiv-method,
rdiv-method,
exception-message-attribute,
invalid-str-codec,
sys-max-int,
bad-python3-import,
deprecated-string-function,
deprecated-str-translate-call,
deprecated-itertools-function,
deprecated-types-field,
next-method-defined,
dict-items-not-iterating,
dict-keys-not-iterating,
dict-values-not-iterating,
consider-using-f-string,
unspecified-encoding,
# Custom
protected-access,
fixme,
Expand All @@ -145,7 +89,6 @@ disable=
too-many-nested-blocks,
invalid-name,
import-error,
bad-continuation,
try-except-raise,
no-else-raise,
no-else-return,
Expand Down Expand Up @@ -558,4 +501,4 @@ known-third-party=enchant

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
overgeneral-exceptions=builtin.Exception
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#################
*****************
Welcome to SUP3R!
#################
*****************

|Docs| |Tests| |Linter| |PyPi| |PythonV| |Codecov| |Zenodo|

Expand Down
7 changes: 3 additions & 4 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
"""
Documentation config file
"""
Expand All @@ -16,8 +15,8 @@
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sphinx_rtd_theme
import sys

sys.path.insert(0, os.path.abspath('../../'))

# -- Project information -----------------------------------------------------
Expand All @@ -31,6 +30,7 @@
sys.path.append(pkg)

from sup3r import __version__ as v

# The short X.Y version
version = v
# The full version, including alpha/beta/rc tags
Expand Down Expand Up @@ -104,8 +104,7 @@
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
html_theme = 'pydata_sphinx_theme'

# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/solar/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"nsrdb_source_fp": "/scratch/gbuster/sup3r/source_gcm_data/nsrdb_clearsky.h5",
"nsrdb_agg": 625,
"nsrdb_smoothing": 0,
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/trh/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"worker_kwargs": {
"max_workers": 1
}
Expand Down
2 changes: 1 addition & 1 deletion examples/sup3rcc/run_configs/wind/config_fwp.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"input_handler_kwargs": {
"target": [23.2, -129],
"shape": [26, 59],
"temporal_slice": [null, null, null],
"time_slice": [null, null, null],
"worker_kwargs": {
"max_workers": 1
}
Expand Down
4 changes: 2 additions & 2 deletions examples/sup3rwind/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ The Sup3rWind data is also loaded into `HSDS <https://www.hdfgroup.org/solutions
Example Sup3rWind Data Usage
-----------------------------

Sup3rWind data can be used in generally the same way as Sup3rCC data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to the Sup3rCC example notebook `here <https://github.com/NREL/sup3r/tree/main/examples/sup3rcc/using_the_data.ipynb>`_ for usage patterns.
Sup3rWind data can be used in generally the same way as `Sup3rCC <https://nrel.github.io/sup3r/examples/sup3rcc.html>`_ data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to the Sup3rCC example notebook `here <https://github.com/NREL/sup3r/tree/main/examples/sup3rcc/using_the_data.ipynb>`_ for usage patterns.

Running Sup3rWind Models
-------------------------

The process for running the Sup3rWind models is much the same as for Sup3rCC (``sup3r/examples/sup3rcc/README.rst``).
The process for running the Sup3rWind models is much the same as for `Sup3rCC <https://nrel.github.io/sup3r/examples/sup3rcc.html>`_.

#. Download the Sup3rWind models to your hardware using the AWS CLI: ``$ aws s3 cp s3://nrel-pds-wtk/sup3rwind/models/``
#. Download the ERA5 data that you want to downscale from `ERA5-single-levels <https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview/>`_ and/or `ERA5-pressure-levels <https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=overview/>`_.
Expand Down
29 changes: 20 additions & 9 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ classifiers=[
"Programming Language :: Python :: 3.11",
]
dependencies = [
"NREL-rex>=0.2.84",
"NREL-rex>=0.2.87",
"NREL-phygnn>=0.0.23",
"NREL-gaps>=0.6.0",
"NREL-gaps>=0.6.13",
"NREL-farms>=1.0.4",
"dask>=2022.0",
"h5netcdf>=1.1.0",
Expand All @@ -54,11 +54,12 @@ dev = [
]
doc = [
"sphinx>=7.0",
"sphinx_rtd_theme>=2.0",
"sphinx-click>=4.0",
"pydata-sphinx-theme>=15.4"
]
test = [
"pytest>=5.2",
"pytest-env"
]

[project.urls]
Expand Down Expand Up @@ -224,10 +225,10 @@ max-complexity = 12
[tool.ruff.lint.per-file-ignores]
"__init__.py" = [
"F401", # unused-import
]
#"docs/source/conf.py" = [
# "E402", # unused-import
# ]
]
"docs/source/conf.py" = [
"E402", # unused-import
]

[tool.ruff.lint.pylint]
max-args = 5 # (PLR0913) Maximum number of arguments for function / method
Expand Down Expand Up @@ -274,16 +275,17 @@ xarray = ">=2023.0"

[tool.pixi.pypi-dependencies]
NREL-sup3r = { path = ".", editable = true }
NREL-rex = { version = ">=0.2.84" }
NREL-rex = { version = ">=0.2.87" }
NREL-phygnn = { version = ">=0.0.23" }
NREL-gaps = { version = ">=0.6.0" }
NREL-gaps = { version = ">=0.6.13" }
NREL-farms = { version = ">=1.0.4" }

[tool.pixi.environments]
default = { solve-group = "default" }
dev = { features = ["dev", "doc", "test"], solve-group = "default" }
doc = { features = ["doc"], solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
viz = { features = ["viz"], solve-group = "default" }
bnb32 marked this conversation as resolved.
Show resolved Hide resolved

[tool.pixi.tasks]
test = "pytest --pdb --durations=10 tests"
Expand All @@ -294,10 +296,19 @@ sphinx_rtd_theme = ">=2.0"

[tool.pixi.feature.test.dependencies]
pytest = ">=5.2"
pytest-cov = ">=5.0.0"

[tool.pixi.feature.dev.dependencies]
build = ">=0.6"
twine = ">=5.0"
ruff = ">=0.4"
ipython = ">=8.0"
pytest-xdist = ">=3.0"

[tool.pixi.feature.viz.dependencies]
jupyter = ">1.0.0"
bnb32 marked this conversation as resolved.
Show resolved Hide resolved
hvplot = ">0.10.0"
bnb32 marked this conversation as resolved.
Show resolved Hide resolved

[tool.pytest_env]
CUDA_VISIBLE_DEVICES = "-1"
TF_ENABLE_ONEDNN_OPTS = "0"
Loading
Loading