Allow read_surfrad to parse http files (#1459)

* Add cams.get_cams_radiation function * Revert "Add cams.get_cams_radiation function" This reverts commit d7deb80. * Allow parsing of http files * Add test for https file * Squashed commit of the following: commit 5047b26 Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com> Date: Tue May 17 19:14:53 2022 +0530 Updated get_cams protocol to https #1457 (#1458) * Updated get_cams protocol to https #1457 * Updated instances of http to https. #1457 * Updated documentation links to https * Added Contributor commit a0812b1 Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com> Date: Wed May 4 20:01:42 2022 +0800 CI asv check (#1454) * CI asv check * added CI asv check * CI asv check * CI asv check * updated CI asv check * Update docs/sphinx/source/whatsnew/v0.9.2.rst updated v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 83e379a Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Apr 28 19:26:09 2022 -0400 Bump pandas to 0.25.0; test updates (#1448) * bump pandas min from 0.22.0 to 0.25.0 * fix buggy test__check_pandas_assert_kwargs don't use monkeypatch and mocker in the same test function. pytest-dev/pytest-mock#289 * fix psm3 test (apparent_zenith -> solar_zenith) * whatsnew * better UTC conversion in sun_rise_set_transit_ephem * helpful comments * more whatsnew * '3.0' -> '3' in read_crn test? * apply dtypes during parsing in read_crn * move dropna() post-processing into read_fwf call * fix read_crn for pandas<1.2.0 * Update pvlib/solarposition.py Co-authored-by: Will Holmgren <william.holmgren@gmail.com> * nix pytz * UTC -> utc * address simd arccos issue in tracking.singleaxis Co-authored-by: Will Holmgren <william.holmgren@gmail.com> commit 8d0f863 Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Tue Apr 12 22:55:58 2022 +0530 Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (#1440) * Advance numba from 0.36.1 to 0.40.0 * Advance numba from 0.36.1 to 0.40.0 * Updated whatsnew.rst commit 5cb695d Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Wed Apr 6 23:58:03 2022 +0530 Remove unnecessary **kwargs from spa_python and get_total_irradiance (#1437) * Update Solarposition.py Removed **kwargs from pvlib.solarposition.spa_python * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235) * Update docs/sphinx/source/whatsnew/v0.9.2.rst * Update docs/sphinx/source/whatsnew/v0.9.2.rst Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> commit 8460b36 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 15:31:25 2022 -0600 Finalize 0.9.1 (#1431) * fix heading levels in user_guide/bifacial.rst * whatsnew cleanup * fix readme html missing tag, maybe unicode zero-width spaces? * readme: link to universal zenodo doi * readme: update installation link for #1173 * whatsnew date * additional contributors * delete errant space commit edbf2a6 Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com> Date: Wed Mar 30 01:58:18 2022 +0530 Updated plot_singlediode.py (#1434) * Update plot_singlediode.py Changed the unit from C to degree C (°C) * Update plot_singlediode.py Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters. * Update v0.9.1.rst Added name to the contributors' list * Update v0.9.1.rst commit cf4a8ad Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Tue Mar 29 14:04:40 2022 -0600 Update sphinx to 4.5.0 (#1435) * bump sphinx and pydata-sphinx-theme versions * clean up sphinx conf.py * fix distutils strangeness, maybe * use freshly-released sphinx==4.5.0 commit 884a153 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Wed Mar 23 13:41:35 2022 -0600 Clarify delta_t docstring descriptions (#1429) * clarify delta_t docstrings * whatsnew commit c243183 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 12:01:57 2022 -0600 Deprecate pvlib.forecast (#1426) * deprecate pvlib.forecast classes * catch warnings in tests * add warning admonition to forecasts.rst * whatsnew * stickler * pin pytest < 7.1.0 * pin pytest in the right place this time * more warning suppression in tests * unpin pytest * Update docs/sphinx/source/whatsnew/v0.9.1.rst * copy warning to reference/forecasting.rst commit e3baa12 Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> Date: Thu Mar 17 11:28:56 2022 -0600 Fix conditional dependency on dataclasses (#1422) * better conditional dependency on dataclasses * whatsnew commit 27cba7a Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com> Date: Thu Mar 17 22:48:08 2022 +0530 Added asv benchmarking badge to the table of badges in the main README. (#1427) * Update Readme.md Added benchmarks asv badge to the badge section * Updated v.0.9.1.rst Added my name to the list of Contributers. commit 1893b20 Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com> Date: Mon Mar 14 18:37:58 2022 +0100 Add variable mapping of psm3 (#1374) * Add variable mapping of psm3 * Add enhancement entry in whatsnew * Fix stickler * Map keys in metadata dict * Remove double spaces in docs * Fix stickler * Doc update Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Reformatting - changes by kanderso-nrel * Update docstring table with 2020 * Add deprecation warning test coverage * Rename to VARIABLE_MAP * Change apparent_zenith to solar_zenith Based on the decision in #1403 * Update attributes docstring * Change elevation to altitude when mapping variables * Update psm3 variable mapping test Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com> * Revert "Squashed commit of the following:" This reverts commit b313c64. * Update whatsnew * Update read_surfrad documentation Co-authored-by: AdamRJensen <arajen@byg.dtu.dk>
pvlib · Jul 19, 2022 · c3016f1 · c3016f1
1 parent f2d14ce
commit c3016f1
Show file tree

Hide file tree

Showing 3 changed files with 24 additions and 4 deletions.
diff --git a/docs/sphinx/source/whatsnew/v0.9.2.rst b/docs/sphinx/source/whatsnew/v0.9.2.rst
@@ -8,6 +8,9 @@ Deprecations
 
 Enhancements
 ~~~~~~~~~~~~
+* :py:func:`pvlib.iotools.read_surfrad` now also accepts remote files
+  with https links in addition to files on the SURFRAD FTP server
+  (:pull:`1459`)
 * Add :py:func:`pvlib.tracking.calc_surface_orientation` for calculating
   single-axis tracker ``surface_tilt`` and ``surface_azimuth`` from
   rotation angles. (:issue:`1471`, :pull:`1480`)
@@ -47,8 +50,9 @@ Requirements
 
 Contributors
 ~~~~~~~~~~~~
+* Adam R. Jensen (:ghuser:`AdamRJensen`)
 * Naman Priyadarshi (:ghuser:`Naman-Priyadarshi`)
 * Chencheng Luo (:ghuser:`roger-lcc`)
 * Prajwal Borkar (:ghuser:`PrajwalBorkar`) 
 * Kevin Anderson (:ghuser:`kanderso-nrel`)
-* Cliff Hansen (:ghuser:`cwhanse`)
+* Cliff Hansen (:ghuser:`cwhanse`)
diff --git a/pvlib/iotools/surfrad.py b/pvlib/iotools/surfrad.py
@@ -44,7 +44,7 @@ def read_surfrad(filename, map_variables=True):
     Parameters
     ----------
     filename: str
-        Filepath or url.
+        Filepath or URL. URL can be either FTP or HTTP.
     map_variables: bool
         When true, renames columns of the Dataframe to pvlib variable names
         where applicable. See variable :const:`VARIABLE_MAP`.
@@ -113,7 +113,8 @@ def read_surfrad(filename, map_variables=True):
     =======================  ======  ==========================================
 
     See README files located in the station directories in the SURFRAD
-    data archives [2]_ for details on SURFRAD daily data files.
+    data archives [2]_ for details on SURFRAD daily data files. In addition to
+    the FTP server, the SURFRAD files are also available via HTTP access [3]_.
 
     References
     ----------
@@ -122,8 +123,10 @@ def read_surfrad(filename, map_variables=True):
        `SURFRAD Homepage <https://www.esrl.noaa.gov/gmd/grad/surfrad/>`_
     .. [2] NOAA SURFRAD Data Archive
        `SURFRAD Archive <ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/>`_
+    .. [3] `NOAA SURFRAD HTTP Index
+       <https://gml.noaa.gov/aftp/data/radiation/surfrad/>`_
     """
-    if str(filename).startswith('ftp'):
+    if str(filename).startswith('ftp') or str(filename).startswith('http'):
         req = Request(filename)
         response = urlopen(req)
         file_buffer = io.StringIO(response.read().decode(errors='ignore'))

diff --git a/pvlib/tests/iotools/test_surfrad.py b/pvlib/tests/iotools/test_surfrad.py
@@ -7,6 +7,8 @@
 testfile = DATA_DIR / 'surfrad-slv16001.dat'
 network_testfile = ('ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/'
                     'Alamosa_CO/2016/slv16001.dat')
+https_testfile = ('https://gml.noaa.gov/aftp/data/radiation/surfrad/'
+                  'Alamosa_CO/2016/slv16001.dat')
 
 
 @pytest.mark.remote_data
@@ -19,6 +21,17 @@ def test_read_surfrad_network():
     assert local_data.equals(network_data)
 
 
+@pytest.mark.remote_data
+@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
+def test_read_surfrad_https():
+    # Test reading of https files.
+    # If this test begins failing, SURFRAD's data structure or data
+    # archive may have changed.
+    local_data, _ = surfrad.read_surfrad(testfile)
+    network_data, _ = surfrad.read_surfrad(https_testfile)
+    assert local_data.equals(network_data)
+
+
 def test_read_surfrad_columns_no_map():
     data, _ = surfrad.read_surfrad(testfile, map_variables=False)
     assert 'zen' in data.columns