How to Use

This guide provides detailed examples on how to use the MultivariateImputer and TimeSeriesImputer. DataFiller targets a pragmatic middle ground for imputation: it does not aim to match the absolute performance of large deep learning models on complex masking patterns, but it is simple to fit, easy to adapt, and flexible to integrate into existing workflows. It is also significantly faster than scikit-learn’s IterativeImputer, which makes it well-suited for fast iteration and production use.

Multivariate Imputer

The MultivariateImputer is the core of the library, designed to impute missing values in a 2D NumPy array or pandas DataFrame. It automatically handles mixed numerical, boolean, and categorical/string columns by one-hot encoding non-numerical features internally so they can help impute other columns, then returning the original schema.

Titanic Mixed-Feature Example

MultivariateImputer handles categorical, string, and boolean columns by one-hot encoding them internally and imputing missing labels with a classifier. The Titanic dataset provides a compact mixed-type example: categorical columns (such as sex or embarked) are used as predictors for other features while their own missing values are imputed with a classifier.

from datafiller.datasets import load_titanic
from datafiller import MultivariateImputer, ExtremeLearningMachine

df = load_titanic()

imputer = MultivariateImputer(regressor=ExtremeLearningMachine())
df_imputed = imputer(df)

The explorer below diffs the original dataset against the imputed result: every highlighted cell was a NaN and now carries a model prediction. Switch to the Original view to see the holes the imputer started from.

Parameters

The main initialization parameters are regressor and classifier to set the numeric/categorical models, plus scoring, rng, min_samples_train, and verbose to control feature selection, reproducibility, training thresholds, and logging. Call parameters include rows_to_impute and cols_to_impute to target subsets and n_nearest_features to limit the features used per imputation; setting n_nearest_features is recommended to reduce computation time. For a complete list and full descriptions, see the API Reference reference.

Time Series Imputer

The TimeSeriesImputer is a wrapper around the MultivariateImputer that is specifically designed for time series data. It can infer a regular DatetimeIndex frequency, reinsert missing timestamp rows inside the observed range, and add low-cost calendar features that remain observed through contiguous gaps.

PEMS-BAY Example

This example loads the PEMS-BAY traffic-speed dataset, punches a large contiguous hole in one sensor’s time series, adds 5% missing-at-random values to the other sensors, and reconstructs the hole using autoregressive lags and leads.

import numpy as np
from datafiller import TimeSeriesImputer
from datafiller.datasets import add_mar, load_pems_bay

df = load_pems_bay()
rng = np.random.default_rng(0)
target_col = rng.choice(df.columns)
ground_truth = df[target_col].copy()

df_missing = df.copy()
n_rows = len(df_missing)
hole_length = int(n_rows * 0.2)
start = n_rows // 2 - hole_length // 2
end = start + hole_length
df_missing.loc[df_missing.index[start:end], target_col] = np.nan

other_cols = df_missing.columns.drop(target_col)
np.random.seed(0)
df_missing.loc[:, other_cols] = add_mar(df_missing[other_cols], nan_ratio=0.05)

ts_imputer = TimeSeriesImputer(lags=[1, 2, 3, -1, -2, -3], rng=0)
df_imputed = ts_imputer(df_missing, cols_to_impute=[target_col], n_nearest_features=75)

The shaded band marks the withheld gap. Inside it, the gray trace is the ground truth the model never saw and the blue trace is the reconstruction; hover the chart to compare both at any timestamp.

Parameters

Initialization parameters include lags for autoregressive features (positive integers create lags like t-1, negative integers create leads like t+1), regressor for the numeric model, interpolate_gaps_less_than to pre-fill short gaps, add_time_features to include deterministic calendar/trend predictors, and the shared controls scoring, rng, min_samples_train, and verbose. Call parameters include rows_to_impute and cols_to_impute to target subsets, n_nearest_features to limit features used for imputation (recommended to reduce computation time), and before/after to restrict the time window. For a complete list and full descriptions, see the API Reference reference.