Learning Objectives
- Distinguish covariance-explaining attribution factors from priced factors, and explain why that distinction matters for prediction, risk decomposition, and trading applications
- Implement PCA on asset returns, interpret principal components as latent risk dimensions or eigenportfolios, and diagnose key practical issues including covariance noise, component selection, and loading instability
- Explain how IPCA and RP-PCA extend PCA by introducing time-varying characteristic-based betas and pricing-error penalties, and evaluate when these extensions are preferable to plain variance maximization
- Implement and evaluate Conditional Autoencoders using walk-forward validation, ensemble averaging, and interpretability diagnostics such as SHAP, while recognizing their main failure modes
- Explain how adversarial SDF estimation enforces no-arbitrage restrictions, how its objective differs from CAE reconstruction, and when direct pricing-error minimization is likely to add value
- Compare latent factor methods across datasets and modeling objectives, and choose among PCA, IPCA, RP-PCA, CAE, and SDF approaches based on dimensionality, economic goal, and evaluation design
Making the Case for Latent Factors
The section frames the factor zoo problem — over 400 published factors, most failing multiple-testing correction (Harvey, Liu, and Zhu 2016) or methodological scrutiny (Hou, Xue, and Zhang 2020's replication found 65% of anomalies disappeared with proper controls) — as the motivation for latent factor models that sidestep data mining by extracting systematic risk sources directly from return covariance structure. It identifies a resilient core of surviving factors (market, value, momentum, profitability, investment) supported by independent economic theories and ML evidence, and introduces the key distinction between attribution factors (explain co-movement, may carry zero premium) and priced factors (carry genuine risk premia), organizing the chapter's methods along three objectives: variance maximization, variance plus pricing errors, and direct pricing-error minimization.
Principal Component Analysis: The Mathematical Foundation
The section develops PCA's mechanics — eigendecomposition of the return covariance matrix to produce eigenvalues (variance explained) and eigenvectors (portfolio weights) — alongside three critical limitations for financial data: linearity, the disconnect between variance and pricing signal, and second-moment focus. It covers practical design choices (covariance vs correlation vs idiosyncratic-volatility-normalized PCA), covariance estimation quality (Marchenko-Pastur bounds for distinguishing signal from noise eigenvalues, Ledoit-Wolf shrinkage), and the BBP phase transition that determines when weak factors become irrecoverable given the N/T ratio of the dataset.
1 notebook
Eigenportfolios for Equity Strategies
The section shows how reading PCA eigenvectors as portfolio weights creates uncorrelated eigenportfolios that serve as building blocks for risk management and systematic trading. It demonstrates that the first eigenportfolio acts as a data-driven market proxy (>0.95 correlation with the equal-weighted index), while higher-order components capture interpretable sector rotations and style tilts that emerge purely from return covariance structure. The section addresses the critical practical challenges of eigenvector instability (near-degenerate eigenvalues causing phantom loadings flips) with the Procrustes rotation fix, introduces Hierarchical PCA for injecting known economic structure, and provides a two-stage production PCA recipe that separates fast-moving volatility dynamics from slow-moving correlation structure.
1 notebook
The Yield Curve Decoded
PCA's clearest success is in fixed income, where Litterman and Scheinkman (1991) showed that three factors — level (>90% of variance, parallel shift reflecting inflation and growth expectations), slope (5-8%, steepening/flattening tied to monetary policy), and curvature (1-2%, butterfly twist linked to rate volatility) — explain 95-99% of Treasury yield curve variation, a result that has replicated across decades and markets. The section explains why PCA works so well for yield curves but not equities: no-arbitrage constraints tightly couple prices across maturities, producing a clean low-rank structure, whereas equities' thousands of idiosyncratic drivers produce complex, time-varying, regime-dependent covariance that defies simple decomposition.
1 notebook
Advanced Statistical Models: Bridging Economics and Data
The section introduces IPCA and RP-PCA as solutions to PCA's two main limitations for equities. IPCA (Kelly, Pruitt, and Su 2019) makes factor loadings dynamic through the insight that "characteristics are covariances" — firm characteristics predict returns because they proxy for time-varying risk exposures, not because they represent standalone anomalies — while RP-PCA (Lettau and Pelger 2020) explicitly targets pricing-relevant factors by minimizing a weighted combination of unexplained variance and cross-sectional pricing errors, detecting "weak but priced" factors that standard PCA misses entirely. The section also covers identification caveats (K selection sensitivity, characteristic set constraints, linear mapping limitations), the double-selection LASSO for robust factor validation, and test-asset design as a choice that determines which factors can be detected.
2 notebooks
Conditional Autoencoders and Deep Learning for Asset Pricing
The section introduces deep learning architectures that learn nonlinear mappings from characteristics to factor exposures, carefully distinguishing two approaches: the Conditional Autoencoder (GKX, trained on reconstruction error, generalizing IPCA's linear beta mapping to a neural network) and the adversarial SDF (CPZ, directly minimizing pricing errors through conditional Euler-equation restrictions with a minimax adversarial mechanism that constructs worst-case test portfolios). Empirical replication on the Chen-Pelger-Zhu dataset achieves lower ICs than the original papers due to shorter test periods, smaller universes, and fewer characteristics, with the SDF model outperforming on raw returns while label winsorization helps supervised models more than latent models. The section also covers ensemble averaging (40% improvement over single models), SHAP interpretability for the beta network, and extensions including cross-asset attention and text-derived characteristics.
2 notebooks
Building the Conditional Autoencoder
This section provides the practical implementation guide for the CAE, covering the full pipeline from data engineering (universe filtering with a microcap toggle, cross-sectional rank normalization to [-1,1]) through model architecture (dual beta-and-factor network structure, K selection via IC cross-validation, Optuna-based hyperparameter search) to training and validation (walk-forward cross-validation, ensemble averaging across random seeds). It catalogues three common failure modes — training instability, initialization sensitivity, and mode collapse in the beta network — with diagnostic criteria and fixes, and provides a six-point robustness checklist (universe dependence, feature leakage, regime dependence, hyperparameter sensitivity, test-asset span, evaluation metric consistency) that applies to all latent factor models.
1 notebook
Case Study Results
Latent factor models were evaluated on five case studies with sufficient cross-sectional breadth, revealing that the objective ladder is only partially ordered in practice: PCA succeeds where low-dimensional structure is genuine (ETFs, US Equities) but fails on narrow cross-sections (CME Futures), where only the no-arbitrage SDF model recovers signal. The strongest results appear on S&P 500 Equity+Options (CAE IC +0.073) and US Equities with IPCA (IC +0.074 at 21 days), while the showcase US Firm Characteristics dataset shows only the SDF (IC +0.014) achieving meaningful positive signal on raw returns — with label winsorization dramatically helping supervised models but not latent models. Fold-level distributions reveal that no primary-label combination achieves a t-statistic above about 1.4, far below the Harvey-Liu-Zhu t>3.0 standard, meaning the point estimates reveal meaningful relative ordering but none would survive multiple testing correction.
1 notebook
Related Case Studies
See where these chapter concepts get applied in end-to-end trading workflows.
ETF Cross-Asset Exposures
All six model families compared across 100 ETFs spanning 9 asset classes
Crypto Perpetuals Funding
Alternative data and non-standard frequencies in 24/7 crypto markets
US Firm Characteristics
Classic factor investing with ML on monthly fundamental data
CME Futures
Carry signals across 30 products — data quality as the critical variable
US Equities Panel
Large-scale cross-sectional prediction across 3,200 stocks with 16 walk-forward folds