Ml-Impute is a library for generating synthetic data for null-value imputation, notably with the ability to handle mixed datatypes. This package is based off of the research of Audigier, Husson, and Josse and their method of iterative factor analysis for singular data imputation.
The goal of this package is to:
(a) provide an open source package for use of this method in Python for the first time, and;
(b) to provide an efficient parallelization of the algorithm when extending it to both single and multiple imputation.
Note: I am currently a university student and may not have the time to continue to release updates and changes as fast as some other packages might. In the spirit of open-source code, please feel free to add pull requests or open a new issue if you have bug fixes or improvements. Thank you for your understanding and for your contributions.
ML-Impute is currently available on PyPi.
Unix/Mac OS/Windows
pip install ml-impute
Currently, ML-Impute can handle both single and multiple imputation.
To follow a demonstration of both methods, proceed to the Example Section.
The following subsections provide an overview into each method along with their usage information.
To use the package post-installation via pip, instantiate the following object as follows:
from mpute import generator
gen = generator.Generator()
| Parameter | Description |
|---|---|
| dataframe | (required) Pandas dataframe object |
| encode_cols | (optional, default=[]) Categorical columns to be encoded. By default, ml-impute will encode all columns with object or category dtypes. However, many datasets contain numerical categorical data (ex/ Likert scales, classification types, etc.) that should be encoded. |
| exclude_cols | (optional, default=[]) Categorical columns to be excluded from encoding and/or imputation. On occastion, datasets will contain unique non-ordinal data (such as unique IDs) that, if encoded, will lead to large increases in memory usage and runtime. These columns should be excluded. |
| max_iter | (optional, default=1000) The maximum number of iterations of imputation before exit. |
| tol | (optional, default=1e-4) Tolerance bound for convergence. If Frobenius norm relative error is < tol before max_iter is reached, exit. |
| explained_var | (optional, default=0.95) Percentage of the total variance kept when reconstructing the dataframe after performing Singular Value Decomposition. |
| method | (optional, default="single") Specification for use of single or multiple imputation method. Possible values: ["single", "multiple"] |
| n_versions | (optional, default=20) If performing multiple imputation, the number of generated dataframes. If performing singular imputation, n_versions=1 |
| noise | (optional, default="gaussian") If performing multiple impuation, specify the type of noise added to each generated dataset to create variation. Gaussian noise is centered around 0 with a standard deviation of 0.1. If performing singular imputation, noise=None |
| engine | (optional, default="default") For either singular or multiple imputation, choose the engine through which the SVD is calculated. Possible values: ["default", "dask"] "default" utilizes the JAX numpy library for efficient SVD calculation and multiprocessing, and is recommended for speed. "dask" creates a dask distributed scheduler which is used to compute the SVD. Given that this is an iterative method, this is recommended only when working with very large datasets. |
| Method | Return Value |
|---|---|
| "single" | imputed_df: a copy of the dataframe argument with synthetic data imputed for all null values |
| "multiple" | df_dict: a dictionary containing each of the n_versions of generated datasets with variable synthetic data. keys: [0, n_versions) values: [dataframes] |
Single imputation works with the following line:
imputed_df = gen.generate(dataframe)
Multiple imputation is as simple as the following:
imputed_dfs = gen.generate(dataframe method="multiple")
For the following example, we will use the titanic example-dataset available in sklearn.datasets openml.
Build the titanic dataset and create a Generator object as follows:
import pandas as pd
from mpute import generator
from sklearn import datasets
titanic, target = datasets.fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
titanic['survived'] = target
gen = generator.Generator()
imputed_df = gen.generate(titanic, exclude_cols=['name', 'cabin', 'ticket'])
Note: 'name', 'cabin', and 'ticket' are excluded as they mainly contain unique identifiers, therefore unnecessary for imputation and if encoded, would result in a significant increase in memory usage.
It is possible to replace the cabin column with two columns such as 'deck' and 'position', as these may be a determinant of survival. However, this preprocessing would have to occur beforehand
Multiple imputation is as simple as the following:
imputed_dfs = gen.generate(titanic method="multiple")
That's all there is to it. Happy using!
ML-Impute is published under the MIT License. Please see the LICENSE file for more information.