You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One of the last steps in establishing interfacing between a module and the 'outside' is units metadata for .csv files.
This is the only file type we handle that has this issue, since netCDF files have units as standardized metadata.
Based on several tests, I believe the best approach is to add a second 'header' that states the unit of each column, using No Unit to specify columns with text values, ratios, etc.
Reasoning:
Removing a second header row is easy in pandas
Fits well with the 'tidy' dataframe approach.
It produces way less data overhead than adding an extra column in the case of timeseries.
calliope v0.7 is able to skip rows easily, so it avoids extra processing on the modelling side.
Allows module devs to easily use pint for unit conversion, if they choose to.
The problem is that pint libraries use No Unit (in pandas) and no unit (in xarray) as standard for unitless cases. I have not tested if capitalisation matters (hopefully not).
You can re-specify it, but using our own standard would add boilerplate, and introduce bugs if said boilerplate is not there.
So even if I dislike it, sticking to the pint naming would save us trouble, I think.
One of the last steps in establishing interfacing between a module and the 'outside' is units metadata for .csv files.
This is the only file type we handle that has this issue, since netCDF files have
units
as standardized metadata.Based on several tests, I believe the best approach is to add a second 'header' that states the unit of each column, using
No Unit
to specify columns with text values, ratios, etc.Reasoning:
calliope
v0.7 is able to skip rows easily, so it avoids extra processing on the modelling side.Examples
Picture this table:
Loading and removing the second header:
If you do not want to use any fancy libraries to handle units, cleaning the data is trivial:
Automatic unit conversion with
pint
If you want to be fancy (and lazy), you can just as easily use
pint
to do all the unit heavy lifting for you.The text was updated successfully, but these errors were encountered: