Dataset Builder¶

MLDatasetBuilder provides leakage-safe dataset preparation for ML training. It handles train/test splitting, automatic scaling (fit on train only), and cross-validation integration with proper fold-level preprocessing.

Use this page when you already have features and labels and want a reusable bridge from engineered data to train/test or cross-validation folds.

No checked book notebook at the pinned revision calls MLDatasetBuilder or create_dataset_builder. Preprocessing Pipeline calls Engineer's StandardScaler and teaches the split-aware preprocessing problem that the builder solves. Use the Book Guide for the exact relationship.

Basic Usage¶

from datetime import date, timedelta

import polars as pl
from ml4t.engineer import create_dataset_builder

features_df = pl.DataFrame({
    "momentum": [float(i) for i in range(20)],
    "volatility": [1.0 + (i % 4) * 0.1 for i in range(20)],
})
labels_series = pl.Series("label", [i % 2 for i in range(20)])
dates_series = pl.Series(
    "date",
    [date(2024, 1, 1) + timedelta(days=i) for i in range(20)],
)

builder = create_dataset_builder(
    features=features_df,
    labels=labels_series,
    dates=dates_series,
    scaler="standard",
)

X_train, X_test, y_train, y_test = builder.train_test_split(train_size=0.8)
assert len(X_train) == len(y_train) == 16
assert len(X_test) == len(y_test) == 4

Train/Test Split¶

X_train, X_test, y_train, y_test = builder.train_test_split(
    train_size=0.8,
    shuffle=False,              # Keep False for time series!
    random_state=None,
)

When a scaler is set, train_test_split automatically:

Fits the scaler on X_train only
Transforms both X_train and X_test using training statistics
Returns scaled DataFrames

This prevents information leakage by construction.

Dataset Info¶

info = builder.info
# DatasetInfo(
#     n_samples=2516,
#     n_features=45,
#     feature_names=["rsi", "macd", "atr", ...],
#     label_name="label",
#     has_dates=True,
# )

Scaler Configuration¶

from ml4t.engineer.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Via string shorthand
builder = create_dataset_builder(features, labels, scaler="standard")
builder = create_dataset_builder(features, labels, scaler="robust")
builder = create_dataset_builder(features, labels, scaler="minmax")
builder = create_dataset_builder(features, labels, scaler=None)  # No scaling

# Via scaler instance (custom parameters)
builder = create_dataset_builder(
    features, labels,
    scaler=RobustScaler(quantile_range=(10.0, 90.0)),
)

# Change scaler after construction
builder.set_scaler(MinMaxScaler(feature_range=(0, 1)))
builder.set_scaler(None)  # Disable scaling

Cross-Validation Integration¶

MLDatasetBuilder integrates with any splitter that follows the SplitterProtocol (compatible with ml4t-diagnostic's WalkForwardCV and CombinatorialCV).

for fold in builder.split(cv=splitter):
    # Each fold has properly scaled train/test data
    fold.X_train       # pl.DataFrame (scaled with train-only stats)
    fold.X_test        # pl.DataFrame (scaled with train stats)
    fold.y_train       # pl.Series
    fold.y_test        # pl.Series
    fold.fold_number   # int
    fold.scaler        # Fitted BaseScaler (or None)
    fold.train_indices # np.ndarray
    fold.test_indices  # np.ndarray

    # Convert to numpy for sklearn/lightgbm
    X_np, X_test_np, y_np, y_test_np = fold.to_numpy()

    # Train your model
    model.fit(X_np, y_np)
    preds = model.predict(X_test_np)

Each fold gets its own scaler instance, fitted independently on that fold's training data. This is the correct behavior for time-series cross-validation where the training window shifts.

FoldResult¶

The FoldResult dataclass returned by each iteration:

@dataclass
class FoldResult:
    X_train: pl.DataFrame
    X_test: pl.DataFrame
    y_train: pl.Series
    y_test: pl.Series
    train_indices: NDArray[np.intp]
    test_indices: NDArray[np.intp]
    fold_number: int
    scaler: BaseScaler | None = None

    def to_numpy(self) -> tuple[NDArray, NDArray, NDArray, NDArray]

Percentile Computation¶

For creating training-only thresholds (e.g., percentile-based labels):

# Feature percentiles (training data only)
cutoffs = builder.get_feature_percentiles(
    train_idx=train_indices,
    quantiles=[0.1, 0.25, 0.5, 0.75, 0.9],
)

# Label percentiles (for discretizing continuous targets)
label_cutoffs = builder.compute_label_percentiles(
    train_idx=train_indices,
    n_quantiles=5,
)

These methods ensure percentile thresholds are computed from training data only, preventing look-ahead bias.

Format Conversion¶

# To numpy (raw, no scaling applied)
X_np, y_np = builder.to_numpy()

# To pandas
X_pd, y_pd = builder.to_pandas()

Factory Function¶

The create_dataset_builder factory provides convenient scaler configuration:

from ml4t.engineer import create_dataset_builder

builder = create_dataset_builder(
    features=features_df,
    labels=labels_series,
    dates=dates_series,           # Optional timestamps
    scaler="standard",            # str, BaseScaler, PreprocessingConfig, or None
)

The scaler parameter accepts:

Value	Effect
`"standard"`	StandardScaler with defaults
`"minmax"`	MinMaxScaler with (0, 1) range
`"robust"`	RobustScaler with IQR
`None`	No scaling
`BaseScaler` instance	Custom scaler with your parameters
`PreprocessingConfig`	Config object that creates the scaler

End-to-End Example¶

import polars as pl
from ml4t.engineer import compute_features, create_dataset_builder
from ml4t.engineer.config import LabelingConfig
from ml4t.engineer.labeling import triple_barrier_labels

# 1. Compute features
df = pl.read_parquet("spy_daily.parquet")
features_df = compute_features(df, ["rsi", "macd", "atr", "bollinger_bands"])

# 2. Create labels
config = LabelingConfig.triple_barrier(
    upper_barrier=0.02, lower_barrier=0.01, max_holding_period=20,
)
labeled = triple_barrier_labels(features_df, config=config)

# 3. Build dataset
feature_cols = [c for c in features_df.columns if c not in df.columns]
builder = create_dataset_builder(
    features=labeled.select(feature_cols),
    labels=labeled["label"],
    dates=labeled["timestamp"],
    scaler="robust",
)

# 4. Train/test split with automatic scaling
X_train, X_test, y_train, y_test = builder.train_test_split(train_size=0.8)

See It In The Book¶

Preprocessing Pipeline calls StandardScaler, not the dataset builder.
Book Guide records that no checked notebook calls create_dataset_builder.
Run examples/complete_workflow_example.py for an end-to-end dataset-builder workflow.

Next Steps¶

Read Preprocessing for scaler behavior and transform choices.
Read Labeling if you still need to create supervised targets.
Use the API Reference when you need exact builder objects and module paths.