Release v0.74.0 · holukas/diive

v0.74.0 | 21 Apr 2024

Added: new function to remove rows that do not have timestamp
info (NaT) (diive.core.times.times.remove_rows_nat and diive.core.times.times.TimestampSanitizer)
Added: new settings VARNAMES_ROW and VARUNITS_ROW in filetypes YAML files, allows better and more specific
configuration when reading data files (diive/configs/filetypes)
Added: many (small) example data files for various filetypes, e.g. ETH-RECORD-TOA5-CSVGZ-20HZ
Added: new optional check in TimestampSanitizer that compares the detected time resolution of a time series with
the nominal (expected) time resolution. Runs automatically when reading files with ReadFileType, in which case
the FREQUENCY from the filetype configs is used as the nominal time
resolution. (diive.core.times.times.TimestampSanitizer, diive.core.io.filereader.ReadFileType)
Added: application of TimestampSanitizer after inserting a timestamp and setting it as index with
function insert_timestamp, this makes sure the freq/freqstr info is available for the new timestamp
index (diive.core.times.times.insert_timestamp)

General: Ran all notebook examples to make sure they work with this version of diive
Added: new notebook for reading EddyPro fluxnet output file with DataFileReader
parameters (notebooks/ReadFiles/Read_single_EddyPro_fluxnet_output_file_with_DataFileReader.ipynb)
Added: new notebook for reading EddyPro fluxnet output file with ReadFileType and pre-defined
filetype EDDYPRO-FLUXNET-CSV-30MIN (notebooks/ReadFiles/Read_single_EddyPro_fluxnet_output_file_with_ReadFileType.ipynb)
Added: new notebook for reading multiple EddyPro fluxnet output files with MultiDataFileReader and pre-defined
filetype EDDYPRO-FLUXNET-CSV-30MIN (notebooks/ReadFiles/Read_multiple_EddyPro_fluxnet_output_files_with_MultiDataFileReader.ipynb)

Renamed: function get_len_header to parse_header(diive.core.dfun.frames.parse_header)
Renamed: exampledata files (diive/configs/exampledata)
Renamed: filetypes YAML files to always include the file extension in the file name (diive/configs/filetypes)
Reduced: file size for most example data files

Added: various test cases for loading filetypes (tests/test_loaddata.py)
Added: test case for loading and merging multiple
files (tests.test_loaddata.TestLoadFiletypes.test_load_exampledata_multiple_EDDYPRO_FLUXNET_CSV_30MIN)
Added: test case for reading EddyPro fluxnet output file with DataFileReader
parameters (tests.test_loaddata.TestLoadFiletypes.test_load_exampledata_EDDYPRO_FLUXNET_CSV_30MIN_datafilereader_parameters)
Added: test case for resampling series to 30MIN time
resolution (tests.test_time.TestTime.test_resampling_to_30MIN)
Added: test case for inserting timestamp with a different convention (middle, start,
end) (tests.test_time.TestTime.test_insert_timestamp)
Added: test case for inserting timestamp as index (tests.test_time.TestTime.test_insert_timestamp_as_index)

Fixed: bug in class DetectFrequency when inferred frequency is None (diive.core.times.times.DetectFrequency)
Fixed: bug in class DetectFrequency where pd.Timedelta() would crash if the input frequency does not have a
number. Timedelta does not accept e.g. the frequency string min for minutely time resolution, even though
e.g. pd.infer_freq() outputs min for data in 1-minute time resolution. TimeDelta requires a number, in this
case 1min. Results from infer_freq() are now checked if they contain a number and if not, 1 is added at the
beginning of the frequency string. (diive.core.times.times.DetectFrequency)
Fixed: bug in notebook WindDirectionOffset, related to frequency detection during heatmap plotting
Fixed: bug in TimestampSanitizer where the script would crash if the timestamp contained an element that could
not be converted to datetime, e.g., when there is a string mixed in with the regular timestamps. Data rows with
invalid timestamps are now parsed as NaT by using errors='coerce'
in pd.to_datetime(data.index, errors='coerce'). (diive.core.times.times.convert_timestamp_to_datetime
and diive.core.times.times.TimestampSanitizer)
Fixed: bug when plotting heatmap (diive.core.plotting.heatmap_datetime.HeatmapDateTime)

Full Changelog: v0.73.0...v0.74.0