diive
is currently under active developement with frequent updates.
diive
is a Python library for time series processing, in particular ecosystem data. Originally developed
by the ETH Grassland Sciences group for Swiss FluxNet.
Recent updates: CHANGELOG
Recent releases: Releases
- For many examples see notebooks here: Notebook overview
- More notebooks are added constantly.
- Daily correlation: calculate daily correlation between two time series (notebook example)
- Decoupling: Investigate binned aggregates (median) of a variable z in binned classes of x and y (notebook example)
- Quantile aggregation: calculate z-aggregates in quantiles (classes) of x and y (notebook example)
- Data gaps identification: (notebook example)
- Histogram calculation: calculate histogram from Series (notebook example)
- Optimum range: find x range for optimum y
- Percentiles: Calculate percentiles 0-100 for series (notebook example)
- Offset correction radiation: correct nighttime offset of radiation data and set nighttime to zero
- Offset correction relative humidity: correct RH values > 100%
- Offset correction wind direction: correct wind directions by offset, calculated based on reference time period (notebook example)
- Set to threshold: set values above or below a threshold value to threshold value
Functions to create various variables.
- Time since: calculate time since last occurrence, e.g. since last precipitation (notebook example)
- Daytime/nighttime flag: calculate daytime flag, nighttime flag and potential radiation from latitude and longitude (notebook example)
- Vapor pressure deficit: calculate VPD from air temperature and RH (notebook example)
- Flux detection limit: calculate flux detection limit from high-resolution data (20 Hz)
- Maximum covariance: find maximum covariance between turbulent wind and scalar
- Turbulence: wind rotation to calculate turbulent departures of wind components and scalar (e.g. CO2)
Input/output functions.
- Detect files: detect expected and unexpected (irregular) files in a list of files
- Split files: split multiple files into smaller parts and export them as (compressed) CSV files
- Read single data files: read file using parameters (notebook example)
- Read single data files: read file using pre-defined filetypes (notebook example)
- Read multiple data files: read files using pre-defined filetype (notebook example)
- Bin fitter
Specific analyses of eddy covariance flux data.
- USTAR threshold scenarios: display data availability under different USTAR threshold scenarios
Post-processing of eddy covariance flux data. For info about the Swiss FluxNet flux levels, see here.
- Flux processing chain (notebook example)
- The notebook example shows the application of:
- Level-2 quality flags
- Level-3.1 storage correction
- Level-3.2 outlier removal
- The notebook example shows the application of:
- Quick flux processing chain (notebook example)
Format data to specific formats.
- Format: convert EddyPro fluxnet output files for upload to FLUXNET database (notebook example)
- Parquet files: load and save parquet files (notebook example)
Fill gaps in time series with various methods.
- XGBoostTS (notebook example (minimal), notebook example (more extensive))
- RandomForestTS (notebook example)
- Linear interpolation (notebook example)
- Quick random forest gap-filling (notebook example)
- Step-wise outlier detection: combine multiple outlier flags to one single overall flag
Create single outlier flags where 0=OK
and 2=outlier
.
- Absolute limits: define absolute limits (notebook example)
- Absolute limits daytime/nighttime: define absolute limits separately for daytime and nighttime data (notebook example)
- Hampel filter: based on Median Absolute Deviation (MAD) in a moving window (notebook example)
- Hampel filter daytime/nighttime, separately for daytime and nighttime data (notebook example)
- Local standard deviation: Identify outliers based on the local standard deviation from a running median (notebook example)
- Local outlier factor: Identify outliers based on local outlier factor, across all data (notebook example)
- Local outlier factor daytime/nighttime: Identify outliers based on local outlier factor, daytime nighttime separately (notebook example)
- Manual removal: Remove time periods (from-to) or single records from time series (notebook example)
- Missing values: Simply creates a flag that indicated available and missing data in a time series (notebook example)
- Trimming: Remove values below threshold and remove an equal amount of records from high end of data (notebook example)
- z-score: Identify outliers based on the z-score across all time series data (notebook example)
- z-score increments daytime/nighttime: Identify outliers based on the z-score of double increments (notebook example)
- z-score daytime/nighttime: Identify outliers based on the z-score, separately for daytime and nighttime (notebook example)
- z-score rolling: Identify outliers based on the rolling z-score (notebook example)
- Cumulatives per year (notebook example)
- Diel cycle per month (notebook example)
- Heatmap date/time: showing values (z) of time series as date (y) vs time (x) (notebook example)
- Heatmap year/month: showing values (z) of time series as year (y) vs month (x) (notebook example)
- Histogram: includes options to show z-score limits and to highlight the peak distribution bin (notebook example)
- Long-term anomalies: calculate and plot long-term anomaly for a variable, per year, compared to a reference period. (notebook example)
- Time series plot: Simple (interactive) time series plot (notebook example)
- ScatterXY plot (notebook example)
- Various classes to generate heatmaps, bar plots, time series plots and scatter plots, among others
- Stepwise MeteoScreening from database (notebook example)
- Diel cycle: calculate diel cycle per month (notebook example)
- Time series stats (notebook example)
- Continuous timestamp: create continuous timestamp based on number of records in the file and the file duration
- Time resolution: detect time resolution from data (notebook example)
- Timestamps: create and insert additional timestamps in various formats
diive
is currently under active developement using Python 3.9.7, but newer (and many older) versions should also work.
pip install diive
poetry add diive
conda intall -c conda-forge diive
Directly use .tar.gz file of the desired version.
pip install https://github.com/holukas/diive/archive/refs/tags/v0.76.2.tar.gz
One way to install and use diive
with a specific Python version on a local machine:
- Install miniconda
- Start
miniconda
prompt - Create a environment named
diive-env
that contains Python 3.9.7:conda create --name diive-env python=3.9.7
- Activate the new environment:
conda activate diive-env
- Install
diive
using pip:pip install diive
- If you want to use
diive
in Jupyter notebooks, you can install Jupyterlab. In this example Jupyterlab is installed from theconda
distribution channelconda-forge
:conda install -c conda-forge jupyterlab
- If used in Jupyter notebooks,
diive
can generate dynamic plots. This requires the installation of:conda install -c bokeh jupyter_bokeh