add a io module

[dacoex]

From:
#7 (comment)

Shall we add standard readers for typical data providers such as:

Input:

• Meteonorm
• SolarGIS, e.g. http://solargis.info/doc/_files/sample_data/SolarGIS-climData/SolarGIS-TMY-hourly-P50-PSA-ES.zip
• BSRN
• SoDa-IS

Output:

• epw: http://apps1.eere.energy.gov/buildings/energyplus/cfm/weather_data3.cfm
• SAM

Hints:
https://github.com/pydata/pandas/blob/master/pandas/io/date_converters.py

There was a similar effort at pandas some time ago which stalled:
pandas-dev/pandas#1180

This could be a meta issue for all input and output format existing in the (solar) energy meteo domain.

[wholmgren]

I would support renaming tmy.py to io.py and adding importers for the most useful data sets. We need to be careful about the scope, though. A comment on the pandas issue correctly points out the danger

It seems to me like this wiill either stagnate or grow into a melange of tailored
solutions to the 1001 weird data problems found in the wild that most pepole won't see.

It's also worth pointing out that pydata recently created a new repo pandas-datareader after having discussed this and related issues for a long time.

[bmu]

I would propose to call it data_io, because io is a module in the python standard library. Pandas uses io but if I remember correctly I had issues some time ago because of this name clash.

[wholmgren]

Do we want to go forward with this for 0.2? I'm still concerned about scope and future maintenance. I don't regularly use these kinds of datasets, so I'm not a good judge of the tradeoffs. Wondering what @Calama-Consulting thinks of this.

Does pvsystem.retrieve_sam belong here too?

Not sure about naming. Technically the module would be named pvlib.<<something>>, so shouldn't clashes should be fairly rare?

[jforbess]

Agreed that future maintenance of dataset importing is an unknowable
effort.

I think the scope to consider here would be how to frame it so that someone
could easily make a reader for a common dataset and add it to pvlib,
because I don't think it makes much sense to make the readers in advance of
need, but it would be great to get people to contribute a good reader if
they found themself using SolarGIS a lot. Personally, I anticipate using a
lot of data from individual plants, without a standard form, but a
well-defined reader template might help with that as well. Not sure.

Not sure about the EPW output. Isn't that an input? It's an input to
PVsyst.

On Wed, Apr 22, 2015 at 8:17 AM, Will Holmgren notifications@github.com
wrote:

Do we want to go forward with this for 0.2? I'm still concerned about
scope and future maintenance. I don't regularly use these kinds of
datasets, so I'm not a good judge of the tradeoffs. Wondering what
@Calama-Consulting https://github.com/Calama-Consulting thinks of this.

Does pvsystem.retrieve_sam belong here too?

Not sure about naming. Technically the module would be named
pvlib.<>, so shouldn't clashes should be fairly rare?

—
Reply to this email directly or view it on GitHub
#29 (comment).

[bmu]

One way for possible new readers would be to define a specific pvlib format for input and than every reader can read data and converts it to this format.

[wholmgren]

I'm still interested in this but will leave it the community to resolve. It gets a "Someday" milestone until somebody expresses interest in resolving it.

[dacoex]

@bmu Do you still think that this is needed?
Where do we get sample data for above data sources?
Which location is good for the test data?
Carpentras?
Or any where we get data for?

[bmu]

There is no need from my side at least ;-). So maybe Wills solution to let users add some readers is the best one.

Dont't think that it is a problem to geht some sample data. I could add Meteonorm data (and ask for permission) as well as SolarGis data.

Am 2. Dezember 2015 21:53:06 GMT-02:00, schrieb DaCoEx notifications@github.com:

@bmu Do you still think that this is needed?
Where do we get sample data for above data sources?
Which location is good?
Carpentras?
Or any where we get data for?

---

Reply to this email directly or view it on GitHub:
#29 (comment)

[dacoex]

There is no need from my side at least ;-). So maybe Wills solution to let users add some readers >is the best one.

OK, so we close the issue and then let users individually create the dateconverter functions etc.?

[wholmgren]

@dacoex I don't have anything new to add to the discussion and it's fine with me if we close it. You created the issue so you can close it if you think we've reach a resolution.

[dacoex]

Here's a repo for some sample data (still to be filled):
https://github.com/dacoex/pvlib_data

I plan to add some data for testing of the io functions (to be coded).

Additionally, the data can be used for tutorials / notebooks / cross-comparison with other tools / etc.

statsmodels do corss-comparison for

• Stata: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/tests/test_panel_robustcov.py
• R Project https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/tests/gen_contingency_table_tests.R

[cwhanse]

Will, others,

I'd like to re-open this issue and volunteer for the task of creating the .io module. I think we should focus on data import, providing tools to bring data into pvlib applications.

I'm going to start with importing data from the Surfrad archive, not that its the most needed converter, but rather because I already have some code to read these files. The first major task for pvlib is delivering the content in pvlib conventions.

Cliff

[adriesse]

Your update led me to the read the foregoing comments. I have spent a lot of time over the years importing, exporting, converting and even reverse-engineering data files. Any import routine that is published in pvlib is likely to find a thankful user somewhere down the line. Maybe even me.

I just wonder, would pecos be a better place for such tools?

[dacoex]

@cwhanse I could started this issue and prepared even a sample test data repo (pvlib_data) which could also be moved under the project umbrella (https://github.com/pvlib). Unfortunately, be daily work does not permit enough time for many coding adventures...

I have a reader function for PVSyst output & MACC-RAD.
We would have agree on the way we to reformat the output.
Or are you simply looking for a shortcut for importing the raw data into a dataframe?

@adriesse I would prefer to have meteo / solar / radiation specific formats in a module. As I understand pecos it's more of a processing tool using other support libs.

[dacoex]

See also further discussion on design in:
#261