Bump pandas to 0.25.0; test updates #1448

kandersolar · 2022-04-22T23:54:20Z

Closes Bug in pandas assert_*_equal wrapper test causes other tests to pass when they should fail #1447
Closes Output of solarposition.sun_rise_set_transit_ephem depends on installed ephem version #1449
I am familiar with the contributing guidelines
Tests added
~~[ ] Updates entries in docs/sphinx/source/reference for API changes.~~
Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
~~[ ] New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.~~
Pull request is nearly complete and ready for detailed review.
Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

Initially taking the less aggressive approach of only bumping pandas to 0.25.0, but wouldn't mind going straight to 1.1.0 if that's what others would prefer.

don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289

kandersolar

Ready for review. For context, this PR fixes #1447 and cleans up the various issues it was hiding.

I opted to do a hodgepodge of fixes in a single PR so that we don't have broken tests on master. Let me know if this is too annoying to review and I'll divide it into smaller PRs.

kandersolar · 2022-04-26T01:55:47Z

pvlib/solarposition.py

@@ -22,6 +22,7 @@
 import pandas as pd
 import scipy.optimize as so
 import warnings
+import pytz


pytz isn't in the standard library right? Is there some alternative to pytz.UTC that we can use to prevent making this an explicit dependency? I'm not at all familiar with non-pandas timezone code.

pandas is built on pytz, so I don't have a problem with the dependency. Interesting idea, though: could we avoid pytz explicitly and just use pandas instead.

kandersolar · 2022-04-26T01:56:14Z

pvlib/tests/iotools/test_crn.py

@@ -83,7 +83,7 @@ def test_read_crn_problems(testfile_problems, columns_mapped, dtypes):
                              '2020-07-06 13:10:00'],
                             freq=None).tz_localize('UTC')
    values = np.array([
-        [92821, 20200706, 1200, 20200706, 700, '3.0', -80.69, 28.62, 24.9,
+        [92821, 20200706, 1200, 20200706, 700, '3', -80.69, 28.62, 24.9,


I think '3' is the correct value here, see previous discussion #1368 (comment)

mystery solved, thanks!

kandersolar · 2022-04-26T01:59:37Z

pvlib/iotools/crn.py

    data = data.astype(dict(zip(HEADERS, DTYPES)))

+    # finally, replace -999 values with NaN
+    data = data.replace(NAN_DICT, value=np.nan)


I think this reworking of read_crn is more correct in two ways:

It doesn't add .0 to string columns that happen to only contain numeric characters.

It only drops all-blank rows, not rows containing all -999 values. Not sure the latter would ever happen in reality, but if it does then it will return a row of NaN instead of dropping it altogether.

Technically this change does affect the returned values (e.g. '3' vs '3.0'); does it warrant a what's new entry?

Technically this change does affect the returned values (e.g. '3' vs '3.0'); does it warrant a what's new entry?

I don't think so

cwhanse · 2022-04-28T13:58:51Z

pvlib/iotools/crn.py

    data = data.astype(dict(zip(HEADERS, DTYPES)))

+    # finally, replace -999 values with NaN
+    data = data.replace(NAN_DICT, value=np.nan)


Technically this change does affect the returned values (e.g. '3' vs '3.0'); does it warrant a what's new entry?

I don't think so

cwhanse · 2022-04-28T14:00:41Z

pvlib/solarposition.py

@@ -22,6 +22,7 @@
 import pandas as pd
 import scipy.optimize as so
 import warnings
+import pytz


pandas is built on pytz, so I don't have a problem with the dependency. Interesting idea, though: could we avoid pytz explicitly and just use pandas instead.

wholmgren · 2022-04-28T17:16:49Z

pvlib/tests/iotools/test_crn.py

@@ -83,7 +83,7 @@ def test_read_crn_problems(testfile_problems, columns_mapped, dtypes):
                              '2020-07-06 13:10:00'],
                             freq=None).tz_localize('UTC')
    values = np.array([
-        [92821, 20200706, 1200, 20200706, 700, '3.0', -80.69, 28.62, 24.9,
+        [92821, 20200706, 1200, 20200706, 700, '3', -80.69, 28.62, 24.9,


mystery solved, thanks!

pvlib/solarposition.py

Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

kandersolar · 2022-04-28T21:06:58Z

Hmm, bare linux 3.8 and 3.9 have an interesting tracking.py failure: https://dev.azure.com/solararbiter/pvlib%20python/_build/results?buildId=6045&view=logs&j=88437082-9fce-53b3-ffed-ea022e86ea67&t=6b9e297c-f567-5c08-dab9-9c9bfc583be2&l=191

Maybe related to numpy's adoption of SIMD? It doesn't seem to happen every time, which I think is consistent with it being SIMD-related, plus the only affected CI variants are what you'd expect for SIMD strangeness. I will look into that before declaring victory here.

kandersolar · 2022-04-28T23:22:13Z

Testing on the benchmark server (linux w/ new CPU, necessary for numpy SIMD to be active) confirms it's a small floating point rounding issue that goes away when SIMD is disabled. So similar to #1191, I just added a call to np.clip to keep the argument of arccos from going out of range.

For reference, here's how to run pytest with SIMD disabled:

$ NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512VL,AVX512BW,AVX512DQ,AVX512_SKX" pytest pvlib/tests/test_tracking.py

commit 5047b26 Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com> Date: Tue May 17 19:14:53 2022 +0530 Updated get_cams protocol to https pvlib#1457 (pvlib#1458) * Updated get_cams protocol to https pvlib#1457 * Updated instances of http to https. pvlib#1457 * Updated documentation links to https * Added Contributor commit a0812b1 Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com> Date: Wed May 4 20:01:42 2022 +0800 CI asv check (pvlib#1454) * CI asv check * added CI asv check * CI asv check * CI asv check * updated CI asv check * Update docs/sphinx/source/whatsnew/v0.9.2.rst updated v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 83e379a Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Apr 28 19:26:09 2022 -0400 Bump pandas to 0.25.0; test updates (pvlib#1448) * bump pandas min from 0.22.0 to 0.25.0 * fix buggy test__check_pandas_assert_kwargs don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289 * fix psm3 test (apparent_zenith -> solar_zenith) * whatsnew * better UTC conversion in sun_rise_set_transit_ephem * helpful comments * more whatsnew * '3.0' -> '3' in read_crn test? * apply dtypes during parsing in read_crn * move dropna() post-processing into read_fwf call * fix read_crn for pandas<1.2.0 * Update pvlib/solarposition.py Co-authored-by: Will Holmgren <william.holmgren@gmail.com> * nix pytz * UTC -> utc * address simd arccos issue in tracking.singleaxis Co-authored-by: Will Holmgren <william.holmgren@gmail.com> commit 8d0f863 Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Tue Apr 12 22:55:58 2022 +0530 Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (pvlib#1440) * Advance numba from 0.36.1 to 0.40.0 * Advance numba from 0.36.1 to 0.40.0 * Updated whatsnew.rst commit 5cb695d Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Wed Apr 6 23:58:03 2022 +0530 Remove unnecessary **kwargs from spa_python and get_total_irradiance (pvlib#1437) * Update Solarposition.py Removed **kwargs from pvlib.solarposition.spa_python * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235) * Update docs/sphinx/source/whatsnew/v0.9.2.rst * Update docs/sphinx/source/whatsnew/v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 8460b36 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 15:31:25 2022 -0600 Finalize 0.9.1 (pvlib#1431) * fix heading levels in user_guide/bifacial.rst * whatsnew cleanup * fix readme html missing tag, maybe unicode zero-width spaces? * readme: link to universal zenodo doi * readme: update installation link for pvlib#1173 * whatsnew date * additional contributors * delete errant space commit edbf2a6 Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com> Date: Wed Mar 30 01:58:18 2022 +0530 Updated plot_singlediode.py (pvlib#1434) * Update plot_singlediode.py Changed the unit from C to degree C (°C) * Update plot_singlediode.py Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters. * Update v0.9.1.rst Added name to the contributors' list * Update v0.9.1.rst commit cf4a8ad Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 14:04:40 2022 -0600 Update sphinx to 4.5.0 (pvlib#1435) * bump sphinx and pydata-sphinx-theme versions * clean up sphinx conf.py * fix distutils strangeness, maybe * use freshly-released sphinx==4.5.0 commit 884a153 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Wed Mar 23 13:41:35 2022 -0600 Clarify delta_t docstring descriptions (pvlib#1429) * clarify delta_t docstrings * whatsnew commit c243183 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 12:01:57 2022 -0600 Deprecate pvlib.forecast (pvlib#1426) * deprecate pvlib.forecast classes * catch warnings in tests * add warning admonition to forecasts.rst * whatsnew * stickler * pin pytest < 7.1.0 * pin pytest in the right place this time * more warning suppression in tests * unpin pytest * Update docs/sphinx/source/whatsnew/v0.9.1.rst * copy warning to reference/forecasting.rst commit e3baa12 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 11:28:56 2022 -0600 Fix conditional dependency on dataclasses (pvlib#1422) * better conditional dependency on dataclasses * whatsnew commit 27cba7a Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Thu Mar 17 22:48:08 2022 +0530 Added asv benchmarking badge to the table of badges in the main README. (pvlib#1427) * Update Readme.md Added benchmarks asv badge to the badge section * Updated v.0.9.1.rst Added my name to the list of Contributers. commit 1893b20 Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com> Date: Mon Mar 14 18:37:58 2022 +0100 Add variable mapping of psm3 (pvlib#1374) * Add variable mapping of psm3 * Add enhancement entry in whatsnew * Fix stickler * Map keys in metadata dict * Remove double spaces in docs * Fix stickler * Doc update Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Reformatting - changes by kanderso-nrel * Update docstring table with 2020 * Add deprecation warning test coverage * Rename to VARIABLE_MAP * Change apparent_zenith to solar_zenith Based on the decision in pvlib#1403 * Update attributes docstring * Change elevation to altitude when mapping variables * Update psm3 variable mapping test Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

* Add cams.get_cams_radiation function * Revert "Add cams.get_cams_radiation function" This reverts commit d7deb80. * Allow parsing of http files * Add test for https file * Squashed commit of the following: commit 5047b26 Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com> Date: Tue May 17 19:14:53 2022 +0530 Updated get_cams protocol to https #1457 (#1458) * Updated get_cams protocol to https #1457 * Updated instances of http to https. #1457 * Updated documentation links to https * Added Contributor commit a0812b1 Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com> Date: Wed May 4 20:01:42 2022 +0800 CI asv check (#1454) * CI asv check * added CI asv check * CI asv check * CI asv check * updated CI asv check * Update docs/sphinx/source/whatsnew/v0.9.2.rst updated v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 83e379a Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Apr 28 19:26:09 2022 -0400 Bump pandas to 0.25.0; test updates (#1448) * bump pandas min from 0.22.0 to 0.25.0 * fix buggy test__check_pandas_assert_kwargs don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289 * fix psm3 test (apparent_zenith -> solar_zenith) * whatsnew * better UTC conversion in sun_rise_set_transit_ephem * helpful comments * more whatsnew * '3.0' -> '3' in read_crn test? * apply dtypes during parsing in read_crn * move dropna() post-processing into read_fwf call * fix read_crn for pandas<1.2.0 * Update pvlib/solarposition.py Co-authored-by: Will Holmgren <william.holmgren@gmail.com> * nix pytz * UTC -> utc * address simd arccos issue in tracking.singleaxis Co-authored-by: Will Holmgren <william.holmgren@gmail.com> commit 8d0f863 Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Tue Apr 12 22:55:58 2022 +0530 Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (#1440) * Advance numba from 0.36.1 to 0.40.0 * Advance numba from 0.36.1 to 0.40.0 * Updated whatsnew.rst commit 5cb695d Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Wed Apr 6 23:58:03 2022 +0530 Remove unnecessary **kwargs from spa_python and get_total_irradiance (#1437) * Update Solarposition.py Removed **kwargs from pvlib.solarposition.spa_python * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235) * Update docs/sphinx/source/whatsnew/v0.9.2.rst * Update docs/sphinx/source/whatsnew/v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 8460b36 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 15:31:25 2022 -0600 Finalize 0.9.1 (#1431) * fix heading levels in user_guide/bifacial.rst * whatsnew cleanup * fix readme html missing tag, maybe unicode zero-width spaces? * readme: link to universal zenodo doi * readme: update installation link for #1173 * whatsnew date * additional contributors * delete errant space commit edbf2a6 Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com> Date: Wed Mar 30 01:58:18 2022 +0530 Updated plot_singlediode.py (#1434) * Update plot_singlediode.py Changed the unit from C to degree C (°C) * Update plot_singlediode.py Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters. * Update v0.9.1.rst Added name to the contributors' list * Update v0.9.1.rst commit cf4a8ad Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 14:04:40 2022 -0600 Update sphinx to 4.5.0 (#1435) * bump sphinx and pydata-sphinx-theme versions * clean up sphinx conf.py * fix distutils strangeness, maybe * use freshly-released sphinx==4.5.0 commit 884a153 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Wed Mar 23 13:41:35 2022 -0600 Clarify delta_t docstring descriptions (#1429) * clarify delta_t docstrings * whatsnew commit c243183 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 12:01:57 2022 -0600 Deprecate pvlib.forecast (#1426) * deprecate pvlib.forecast classes * catch warnings in tests * add warning admonition to forecasts.rst * whatsnew * stickler * pin pytest < 7.1.0 * pin pytest in the right place this time * more warning suppression in tests * unpin pytest * Update docs/sphinx/source/whatsnew/v0.9.1.rst * copy warning to reference/forecasting.rst commit e3baa12 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 11:28:56 2022 -0600 Fix conditional dependency on dataclasses (#1422) * better conditional dependency on dataclasses * whatsnew commit 27cba7a Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Thu Mar 17 22:48:08 2022 +0530 Added asv benchmarking badge to the table of badges in the main README. (#1427) * Update Readme.md Added benchmarks asv badge to the badge section * Updated v.0.9.1.rst Added my name to the list of Contributers. commit 1893b20 Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com> Date: Mon Mar 14 18:37:58 2022 +0100 Add variable mapping of psm3 (#1374) * Add variable mapping of psm3 * Add enhancement entry in whatsnew * Fix stickler * Map keys in metadata dict * Remove double spaces in docs * Fix stickler * Doc update Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Reformatting - changes by kanderso-nrel * Update docstring table with 2020 * Add deprecation warning test coverage * Rename to VARIABLE_MAP * Change apparent_zenith to solar_zenith Based on the decision in #1403 * Update attributes docstring * Change elevation to altitude when mapping variables * Update psm3 variable mapping test Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Revert "Squashed commit of the following:" This reverts commit b313c64. * Update whatsnew * Update read_surfrad documentation Co-authored-by: AdamRJensen <arajen@byg.dtu.dk>

kandersolar added 4 commits April 22, 2022 19:25

bump pandas min from 0.22.0 to 0.25.0

05e9d95

fix buggy test__check_pandas_assert_kwargs

0fbc534

don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289

fix psm3 test (apparent_zenith -> solar_zenith)

23b3b79

whatsnew

258a2aa

kandersolar added the testing label Apr 22, 2022

kandersolar added this to the 0.9.2 milestone Apr 22, 2022

kandersolar added 7 commits April 25, 2022 14:09

better UTC conversion in sun_rise_set_transit_ephem

1ae6dee

helpful comments

efc3873

more whatsnew

f1195a0

'3.0' -> '3' in read_crn test?

e02d9d3

apply dtypes during parsing in read_crn

cb486a3

move dropna() post-processing into read_fwf call

d97f067

fix read_crn for pandas<1.2.0

5d64cb8

kandersolar marked this pull request as ready for review April 26, 2022 02:06

kandersolar commented Apr 26, 2022

View reviewed changes

cwhanse approved these changes Apr 28, 2022

View reviewed changes

wholmgren approved these changes Apr 28, 2022

View reviewed changes

kandersolar and others added 3 commits April 28, 2022 16:00

Update pvlib/solarposition.py

12eb84a

Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

nix pytz

d000079

UTC -> utc

2fbccdd

address simd arccos issue in tracking.singleaxis

6494ea0

kandersolar merged commit 83e379a into pvlib:master Apr 28, 2022

kandersolar deleted the tests_and_pandas branch April 28, 2022 23:26

kandersolar mentioned this pull request Jan 17, 2023

Test pvsystem.singlediode against precisely generated IV curves #1573

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump pandas to 0.25.0; test updates #1448

Bump pandas to 0.25.0; test updates #1448

kandersolar commented Apr 22, 2022 •

edited

Loading

kandersolar left a comment

kandersolar Apr 26, 2022

cwhanse Apr 28, 2022

kandersolar Apr 26, 2022

wholmgren Apr 28, 2022

kandersolar Apr 26, 2022

cwhanse Apr 28, 2022

cwhanse Apr 28, 2022

cwhanse Apr 28, 2022

wholmgren Apr 28, 2022

kandersolar commented Apr 28, 2022

kandersolar commented Apr 28, 2022

Bump pandas to 0.25.0; test updates #1448

Bump pandas to 0.25.0; test updates #1448

Conversation

kandersolar commented Apr 22, 2022 • edited Loading

kandersolar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kandersolar commented Apr 28, 2022

kandersolar commented Apr 28, 2022

kandersolar commented Apr 22, 2022 •

edited

Loading