diive
is a Python library for time series processing, in particular ecosystem data. Originally developed
by the ETH Grassland Sciences group for Swiss FluxNet.
Recent updates: CHANGELOG
Recent releases: Releases
- For many examples see notebooks here: Notebook overview
- More notebooks are added constantly.
- Calculate z-aggregates in quantiles (classes) of x and y (notebook example)
- Daily correlation (notebook example)
- Decoupling: Sorting bins method (notebook example)
- Find data gaps (notebook example)
- Histogram (notebook example)
- Optimum range
- Percentiles (notebook example)
- Offset correction
- Set to threshold
- Wind direction offset detection and correction (notebook example)
- Calculate time since last occurrence, e.g. since last precipitation (notebook example)
- Calculate daytime flag, nighttime flag and potential radiation from latitude and longitude (notebook example)
- Day/night flag from sun angle
- VPD from air temperature and RH (notebook example)
- Flux detection limit from high-resolution data
- Find maximum covariance between turbulent wind and scalar
- Wind rotation to calculate turbulent departures of wind components and scalar (e.g. CO2)
- Detect expected and unexpected (irregular) files in a list of files
- Split multiple files into smaller parts and export them as (compressed) CSV files
- Read single data file with parameters (notebook example)
- Read single data file with pre-defined filetype (notebook example)
- Read multiple data files with pre-defined filetype (notebook example)
- Bin fitter
- Critical heat days for NEP, based on air temperature and VPD
- CO2 penalty
- USTAR threshold scenarios
For info about the Swiss FluxNet flux levels, see here.
- Flux processing chain (notebook example)
- The notebook example shows the application of:
- Level-2 quality flags
- Level-3.1 storage correction
- Level-3.2 outlier removal
- The notebook example shows the application of:
- Quick flux processing chain (notebook example)
Format data to specific formats
- Convert EddyPro fluxnet output files for upload to FLUXNET database (notebook example)
- Load and save parquet files (notebook example)
Fill gaps in time series with various methods
- XGBoostTS (notebook example (minimal), notebook example (more extensive))
- RandomForestTS (notebook example)
- Linear interpolation (notebook example)
- Quick random forest gap-filling (notebook example)
- Step-wise outlier detection
Single outlier tests create a flag where 0=OK
and 2=outlier
.
- Absolute limits (notebook example)
- Absolute limits, separately defined for daytime and nighttime data (notebook example)
- Hampel filter: based on Median Absolute Deviation (MAD) in a moving window (notebook example)
- Hampel filter, separately for daytime and nighttime data (notebook example)
- Incremental z-score: Identify outliers based on the z-score of double increments (notebook example)
- Local standard deviation: Identify outliers based on the local standard deviation from a running median (notebook example)
- Local outlier factor: Identify outliers based on local outlier factor, across all data
- Local outlier factor: Identify outliers based on local outlier factor, daytime nighttime separately
- Manual removal: Remove time periods (from-to) or single records from time series
- Missing values: Simply creates a flag that indicated available and missing data in a time series
- Trimming: Remove values below threshold and remove an equal amount of records from high end of data (notebook example)
- z-score: Identify outliers based on the z-score across all time series data (notebook example)
- z-score: Identify outliers based on the z-score, separately for daytime and nighttime (notebook example)
- z-score: Identify outliers based on the rolling z-score
- z-score: Identify outliers based on max z-scores in the interquartile range data
- Diel cycle per month (notebook example)
- Heatmap showing values (z) of time series as date (y) vs time (x) (notebook example)
- Heatmap showing values (z) of time series as year (y) vs month (x) (notebook example)
- Long-term anomalies per year (notebook example)
- Simple (interactive) time series plot (notebook example)
- ScatterXY plot (notebook example)
- Various classes to generate heatmaps, bar plots, time series plots and scatter plots, among others
- Stepwise MeteoScreening from database (notebook example)
- Calculate diel cycle per month (notebook example)
- Time series stats (notebook example)
- Create continuous timestamp based on number of records in the file and the file duration
- Detect time resolution from data (notebook example)
- Insert additional timestamps in various formats
diive
is currently developed under Python 3.9.7, but newer (and many older) versions should also work.
pip install diive
poetry add diive
conda intall -c conda-forge diive
Directly use .tar.gz file of the desired version.
pip install https://github.com/holukas/diive/archive/refs/tags/v0.76.2.tar.gz
One way to install and use diive
with a specific Python version on a local machine:
- Install miniconda
- Start
miniconda
prompt - Create a environment named
diive-env
that contains Python 3.9.7:conda create --name diive-env python=3.9.7
- Activate the new environment:
conda activate diive-env
- Install
diive
using pip:pip install diive
- If you want to use
diive
in Jupyter notebooks, you can install Jupyterlab. In this example Jupyterlab is installed from theconda
distribution channelconda-forge
:conda install -c conda-forge jupyterlab
- If used in Jupyter notebooks,
diive
can generate dynamic plots. This requires the installation of:conda install -c bokeh jupyter_bokeh