Ideas for future development of pvlib iotools

[AdamRJensen]

I've attempted to provide a bit of an overview of the possible development of the iotools and gather some notes on the inconsistencies in how the functions are implemented. The items are primarily based on the existing open pull requests and issues with the io tag, which can be found here.

General

• iotools landing page / gallery. I think that it would be of great value to have some sort of landing page with an overview of supported file formats and data sources. There is some discussion here: How should pvlib.iotools be showcased in the documentation? #1304, Create weather data/iotools page in User's Guide #1754
• Table with data providers summarizing main differences (this is done in https://doi.org/10.1016/j.solener.2023.112092)

Reanalysis data

• ERA5 Add retrieval function for ERA5 reanalysis data #1264
• MERRA2 Add retrieval function for NASA MERRA2 reanalysis data #1274
  See also What to do about ERA5 and MERRA2? #1484
• NASA POWER

Commercial data
See #1310 for a general discussion.

• SolarAnywhere Add functions to read and retrieve SolarAnywhere irradiance data #1497
• Solcast Integrate Solcast API into iotools #1313 add functionality to load Solcast API data to iotools #1875
• Solargis Add get_solargis iotools function #1969
• Meteonorm

Other data sources

• PVDAQ
  • pvdaq io functions #664 (stale PR)
  • PVDAQ API IO tools #663
• Add iotools function to write SAM-formatted CSV files #1371
• Extend read_bsrn to also parse LR1000 etc.
• Horizon data
  • [ENH] download and use horizon data to affect sky diffuse and far shading #1290
  • Add retrieval function for horizon profile from MINES Paris Tech #1295
  • Adding PVGIS horizon data retrieval method #1395
• Rainfall data
  • NOAA NCDC global summary of the day GSOD for rainfall in iotools #668
  • get PRISM rainfall data #1293

Enhancements / inconsistencies

• Ensure map_variables returns altitude and not elevation #1420
• Add variable mapping of read_tmy3 #1517
• Add variable mapping to SRML iotools functions #1773
• Variable mapping of read_tmy2
• Variable mapping of read_epw
• Add coerce_year parameter to get_pvgis_tmy
• Make new function get_srml that is similar to read_srml_month_from_solardat but accepts start/end Add get_srml iotools function; deprecate read_srml_month_from_solardat  #1779
• Add a generic Latitude.from_meta() function (similar to from_epw and from_tmy)
• Address read_epw data shifted 1 hour back #1156
• Change server to url in pvlib.iotools.get_cams
• For all get functions, remove timeout and replace by kwargs that pass to requests.get
• Allow start/end to be string by wrapping in pd.to_datetime(start), see this comment
• Replacing read_srml_month_from_solardat with get_srml Add get_srml iotools function; deprecate read_srml_month_from_solardat  #1779
• Renaming of pvlib.iotools.read_midc_raw_data_from_nrel
• Add get_surfrad function
• Add get_solrad function Create pvlib.iotools.get_solrad #1967

Returned parameters
Another inconsistency is whether the iotools functions return a single data object or a tuple of (data, meta). A single data object is returned when the metadata is not being parsed or the file does not contain any metadata. It is likely that several of these functions might be improved in the future to also parse metadata; however, returning a tuple of (data, meta) would result in a breaking change which is undesirable. Therefore, I suggest that all new functions return a tuple of (data, meta); and in the case that the metadata is not parsed or nonexistent the meta object is simply an empty dictionary. Given the large number of existing functions, I do not suggest that we change the existing functions. The following functions only return a single DataFrame:

• read_midc
• read_midc_raw_data_from_nrel
• read_crn
• read_srml
• read_srml_month_from_solardat Add get_srml iotools function; deprecate read_srml_month_from_solardat  #1779
• read_solrad Have pvlib.iotools.read_solrad return metadata #1968
  Note the PVGIS functions return three or four objects.

[AdamRJensen]

Design spec (DRAFT)

This is a draft for an iotools design specification (this is already partly covered in https://doi.org/10.1016/j.solener.2023.112092)

Function pattern

Should return a tuple of two elements (data, metadata). The data object is often a pandas DataFrame and the metadata object is a dictionary. If the function does not parse any metadata, an empty dictionary should be returned to follow the standard format.

Function naming

The main iotools functions should follow the naming patterns read_x and get_x, where x is a short string describing the data source. For example, read_bsrn and get_bsrn.

Location

For measurement data, the location should be specified using the station parameters which is often an abbreviation of the station name. For gridded irradiance databases

Time

The timeframe of the requested data should be specified using the start and end parameters. It is highly beneficial that string arguments are also accepted, e.g., '2021-01-01' which can be achieved by using pd.to_datetime. Therefore, the first two lines of an iotools function are typically:

start = pd.to_datetime(start)
end = pd.to_datetime(end)

Credentials

The parameters used to specify credentials depend on the specific service. A suitable combination of the following parameters are recommended username, password, api_key.

Variable mapping

Functions should by default map variable names to the standard pvlib names listed in the variable and Symbols list. The parameter determining whether variables are mapped should be named map_variables

Other

Other things to consider: url, order of parameters, unit conversion, time_step

[mikofski]

Can we also distinguish between a “TMY” style file with 8760 hours of data versus other periods?

And I’d also like to see coerce_year be a common feature for TMY style files.

one issue I have with PVGIS is that it’s always UTC which means that it almost always starts 5 to 8 hours before midnight, so another feature I’d like to see is to rotate the data so it starts at midnight in the local timeline

[mikofski]

A TMY file, I believe, starts with the interval from midnight to 1am on Jan 1. Localizing the timezone is only a half solution. In North America, the pvgis file is still stuck starting at 4pm to 7pm on Dec 31 the previous day, and ends before the 11pm -12am interval on Dec 31 of the following year. We need to rotate the dataframe to make it comparable to other TMY files. All of the intervals should be in the coerced year.

[adriesse]

@mikofski I understand your point as I've dealt with this before. Have you raised the issue with PVGIS? Does other software do this "rotation" transparently? Anyway, np.roll() on the values should do the trick.

[mikofski]

Exactly, I’m suggesting roll_tz as part of the standard API if given would roll the pvgis file so that in the given local tz, it now matches a standard TMY

[AdamRJensen]

Exactly, I’m suggesting roll_tz as part of the standard API if given would roll the pvgis file so that in the given local tz, it now matches a standard TMY

If we have a roll_tz parameter, we probably first need to have an infer_tz parameter to set a meaningful time zone (can be inferred from the longitude). Agree?

[adriesse]

Exactly, I’m suggesting roll_tz as part of the standard API if given would roll the pvgis file so that in the given local tz, it now matches a standard TMY

I don't know, if it's just fixing a current deficiency of pvgis then it doesn't seem like a thing for a standard API.

[adriesse]

There are two general direction iotools could go:

1. Make all data sources look the same to the user, which is most convenient but may hide unique strengths and features of individual sources.
2. Just get the available data into a dataframe (or similar) and let the user take it from there.

I suppose the final product will be somewhere in between, but it may be helpful to keep those extremes in mind and to consider where iotools' aspirations lie in relation to them.