Learning Objectives
- Distinguish structural breaks, regimes, data drift, concept drift, and online detection, and explain why static trading models degrade in changing markets
- Explain the ML4T Workflow as a research-to-production system, including its data infrastructure foundation, scoping invariants, iterative research modules, and feedback loops from live trading back to research
- Define the evidence boundary between exploration and confirmation, and explain how trial logging, sealed holdouts, and selection-aware evaluation preserve research integrity
- Describe how causal inference and generative AI fit within a disciplined trading workflow, including the main benefits they provide and the new failure modes they introduce
- Apply regime thinking, implementability checks, and monitoring logic to diagnose strategy vulnerabilities and to adapt workflow discipline across independent and institutional settings
Why process discipline matters
Durable trading performance depends more on a disciplined, adaptive workflow than on model sophistication. The section traces four market shockwaves (2020-2025) that broke assumptions calibrated to the prior decade, introduces a precise vocabulary for non-stationarity (structural breaks, regimes, drift, online detection), and grounds the claim in Andrew Lo's Adaptive Markets Hypothesis. The takeaway is that process failures dominate algorithmic failures, and that a research-to-production loop capable of detecting decay and distinguishing noise from regime change is the true source of edge.
Introducing the ML4T workflow
The complete ML4T lifecycle is presented as a two-layer system: a data infrastructure foundation (Chapters 2-5) that enforces point-in-time correctness and reproducibility, and an iterative strategy research loop (Chapters 6-21) that moves from scoping through feature engineering, modeling, backtesting, and deployment. The section details what must be fixed before research begins — decision-time correctness, universe rules, label definitions, cost-model class, evaluation protocol — and what remains iterative. The central discipline mechanism is the evidence boundary separating exploration from confirmation, where credibility comes from counting the search and reserving untouched holdout data for final evaluation.
Causal inference and generative AI in the workflow
Causal inference and generative AI are located within the ML4T workflow, arguing that both amplify the upside of good process and the downside of bad process rather than replacing discipline. The section introduces two practical research entry points (prediction-first vs. mechanism-first) and two deliverable types (tradable signals vs. measurement deliverables like premia estimates or risk attributions). Causal inference is positioned as a discipline-enforcing diagnostic lens, while generative AI is shown to scale leakage, hallucination, and complexity inflation unless outputs are treated as candidates subject to the same evaluation safeguards as any other research artifact.
Keeping up with changing market regimes
Regime detection is demonstrated as a practical risk lens rather than a return-timing tool, using two worked examples: style regimes from AQR's century-long factor premia dataset and macro regimes from FRED indicators. A two-state GMM on factor returns separates Risk-On from Risk-Off environments with a 1.3x volatility ratio, while a four-component macro GMM links economic configurations to regime-conditional volatility and drawdown. The reader learns to build regime overlays that map to concrete risk actions (position sizing, exposure caps) while avoiding common pitfalls such as look-ahead bias from ex-post labels and over-interpretation of unsupervised clusters.
Independent versus institutional workflows in the real world
The section contrasts the failure modes that institutional infrastructure partially mitigates with those that independent researchers must address deliberately: goalpost drift, assumption stacking, flexibility without accounting, and late-discovered untradability. It prescribes a series of documented checkpoints (scoping, data integrity, signal robustness, tradability fragility, monitoring) and identifies where independents hold asymmetric advantages — particularly in capacity-constrained opportunities and tighter iteration loops. The key takeaway is that the highest-leverage investment for a solo practitioner is reusable infrastructure that makes disciplined experimentation cheap and repeatable.
Summary
Factor Regimes
Macro Regimes
142 references cited in this chapter.
C. W. J. Granger (1980) — Journal of Economic Dynamics and Control
Granger proposes an operational, forecasting-based definition of causality for time series and argues that causal claims should be validated using out-of-sample forecasting improvements rather than in-sample model fit.
International Asset Allocation With Regime Shifts
Andrew Ang and Geert Bekaert (2002) — Review of Financial Studies · 1567 citations
Despite correlations rising in bear markets, international diversification remains economically valuable, particularly when investors can switch into cash (risk-free assets) during high-volatility regimes.
Ana L. C. Bazzan et al. (2004) — Springer Berlin Heidelberg
This paper introduces a method for detecting concept drift in online learning by monitoring the error rate of a learning algorithm and adapting the model when significant increases in error suggest a change in the underlying data distribution.
Andrew W. Lo (2004) · 46 citations
This paper introduces the Adaptive Markets Hypothesis (AMH), a new framework that reconciles the Efficient Markets Hypothesis (EMH) with behavioral finance by applying evolutionary principles to financial markets, suggesting that market efficiency is dynamic and context-dependent.
David Easley et al. (2012)
High Frequency Trading (HFT) is defined not by speed, but by operating in 'event-based time' (volume clocks), which allows for statistical normality and the exploitation of Low Frequency Traders (LFTs) who operate in chronological time.
David H. Bailey et al. (2014) — Notices of the American Mathematical Society · 150 citations
The paper shows mathematically and via simulations how trying many strategy variations can manufacture high in-sample Sharpe ratios that collapse (or even turn negative) out-of-sample, and it proposes a “minimum backtest length” requirement tied to the number of trials.
David H. Bailey et al. (2015) · 119 citations
This paper proposes a practical, model-free way to estimate how likely your “best” backtest is a false positive caused by trying many strategy variations on the same data.
J.D. Hamilton (2016) — Elsevier
A comprehensive survey of Markov-switching models for identifying economic regimes (recessions/expansions), detailing estimation via EM algorithms and Gibbs sampling.
...and the Cross-Section of Expected Returns
Campbell R. Harvey et al. (2016) — Review of Financial Studies · 1838 citations
Due to extensive data mining in asset pricing ('the factor zoo'), the standard t-statistic threshold of 2.0 is insufficient; this paper mathematically demonstrates that a t-statistic > 3.0 is required to establish true significance.
Jie Lu et al. (2018) — IEEE Transactions on Knowledge and Data Engineering · 1684 citations
This paper reviews methodologies and techniques for concept drift detection, understanding, and adaptation in machine learning, providing a framework and discussing datasets and research directions.
Robert D. Arnott et al. (2018)
This paper proposes a research protocol for backtesting investment strategies, particularly in the context of machine learning, to mitigate the risk of overfitting and false positives due to limited financial data.
Judea Pearl (2019) — Communications of the ACM · 620 citations
Pearl argues that moving from pattern recognition to causal models (especially interventions and counterfactuals) is necessary to make ML systems robust, explainable, and capable of “what if?” reasoning.
Shihao Gu et al. (2020) — The Review of Financial Studies · 1802 citations
This paper benchmarks major machine-learning methods for predicting U.S. equity risk premia and shows that nonlinear models (especially shallow neural nets and trees) materially improve out-of-sample forecast accuracy and investable Sharpe ratios versus classic linear/panel regressions.
Kewei Hou et al. (2020) — The Review of Financial Studies · 759 citations
This paper replicates 452 published financial anomalies and finds that the majority fail to hold up to current empirical finance standards, particularly when controlling for microcaps and multiple testing, suggesting capital markets are more efficient than previously recognized.
Darrell Duffie (2020)
The paper analyzes the March 2020 Treasury market liquidity collapse, attributing it to regulatory constraints on dealer balance sheets, and proposes mandatory central clearing to enable netting and restore market elasticity.
Bernhard Schölkopf et al. (2021) · 351 citations
A foundational framework proposing the integration of Causal Inference with Deep Learning to solve the 'out-of-distribution' generalization problem common in non-stationary environments like financial markets.
A. Sinem Uysal and John M. Mulvey (2021) — The Journal of Financial Data Science · 20 citations
The paper uses supervised ML (especially random forests) to predict recessions and equity “crash” regimes from macro data and then uses these probabilities to improve risk parity portfolios via regime-aware covariance estimation and overlay trades.
Blanka Horvath et al. (2021) · 12 citations
This paper introduces the Wasserstein k-means (WK-means) algorithm, a robust, non-parametric method for clustering financial time series into distinct market regimes by treating segments as probability distributions and using the p-Wasserstein distance, outperforming traditional moment-based k-means and HMMs, especially for non-Gaussian data.
Stefan Studer et al. (2021) — Machine Learning and Knowledge Extraction · 235 citations
A comprehensive process model extending CRISP-DM to include Quality Assurance (QA) and a specific 'Monitoring and Maintenance' phase to address the 75-85% failure rate of ML projects.
Antti Ilmanen et al. (2021)
A century-long analysis across six asset classes confirms that while factor premia are robust and not degraded by arbitrage, factor timing strategies generally fail to overcome transaction costs.
Stefano Giglio et al. (2022) — Annual Review of Financial Economics
This paper surveys how machine learning methods are revolutionizing asset pricing by improving the estimation of expected returns, factors, risk exposures, and risk premia, especially in high-dimensional settings with many assets and characteristics.
Martin Luk (2023) · 11 citations
A comprehensive survey detailing the technical architecture of Generative AI (Transformers, Diffusion) and its specific applications in quantitative finance, including superior sentiment analysis and synthetic data generation.
Marcos López de Prado (2023) — The Journal of Portfolio Management
The paper argues that standard factor investing is flawed because it treats associational regression betas as causal effects without establishing a causal graph, leading to spurious anomalies and portfolio risks.
Robin Marshall (2023)
Marcos Lopez de Prado et al. (2024)
This paper argues that mainstream (associational) factor model specification practices systematically select misspecified models—especially by mis-handling confounders and colliders—causing disappointing live factor performance and even systematic losses, and it calls for rebuilding factor research around causal graphs and do-calculus.
James Ryseff et al. (2024) · 14 citations
This report identifies the primary reasons why AI projects fail, based on interviews with experienced AI practitioners, highlighting leadership miscommunication, data quality issues, and a focus on technology over problem-solving.
Frank J. Fabozzi et al. (2024) — The Journal of Portfolio Management
The paper argues that causal modeling for investing should combine (1) empirical data, (2) theory-implied causal structure, and (3) real-time qualitative “priors” into one structural causal modeling workflow to better navigate non-mechanistic financial systems.
What AI Can (and Can't Yet) Do for Alpha
Ziang Fang and Jason Moore (2025)
Man Numeric details 'AlphaGPT', a proprietary multi-agent AI workflow that automates the quantitative research cycle (ideation, coding, evaluation) to overcome human bandwidth constraints.
Justina Lee (2025) — Bloomberg.com
Marcos López de Prado and Vincent Zoonekynd (2025) — The Journal of Portfolio Management · 1 citations
The paper argues that standard “kitchen-sink” econometric factor-model practices systematically select misspecified models (especially by controlling for colliders), producing biased betas that can make factor strategies lose money even when true risk premia are constant—so factor research should adopt a causal-graph-based protocol.
Frank J. Fabozzi and Caleb C. Stenholm (2025) — The Journal of Portfolio Management
This paper establishes a comprehensive operational framework for asset managers by mapping military doctrines—specifically the OODA loop, mission command, and after-action reviews—to investment decision-making processes.
Alex Botte and Doris Bao
Man Group
A roundtable of leading academics and Man Group practitioners discussing the specific challenges of applying Generative AI to quantitative finance, identifying look-ahead bias in backtesting as the primary obstacle.
Halbert White (2000) — Econometrica · 1769 citations
This paper introduces a 'Reality Check' procedure to test whether the best model found during a specification search has genuine predictive power over a benchmark, accounting for data snooping biases.
Statistical Modeling: The Two Cultures
Leo Breiman (2001)
This paper argues that the statistical community's over-reliance on data models has led to irrelevant theory and missed opportunities, advocating for a shift towards algorithmic modeling which focuses on predictive accuracy and can handle complex, unknown data mechanisms.
Andrew Ang and Allan Timmermann (2011) · 454 citations
This paper reviews how regime-switching models (HMMs) capture the abrupt, persistent changes in financial data (volatility clustering, skewness) that linear models miss, demonstrating that optimal portfolios must dynamically adjust to 'bull' and 'bear' states.
David H. Bailey and Marcos Lopez de Prado (2012) · 147 citations
The paper replaces “point-estimate Sharpe ratio” with a probability-of-skill metric (Probabilistic Sharpe Ratio, PSR) that adjusts for non-normal returns and track-record length, and extends this idea to a portfolio-selection concept called the Sharpe Ratio Efficient Frontier (SEF).
David H. Bailey et al. (2014)
The paper demonstrates—via an online simulator—that even very simple trading strategies can look great in-sample purely by trying enough parameter combinations, yet fail out-of-sample because the apparent edge is just backtest overfitting.
David H. Bailey and Marcos Lopez de Prado (2014) · 110 citations
The paper introduces the Deflated Sharpe Ratio (DSR), a statistic that adjusts performance metrics for the probability of backtest overfitting caused by multiple testing and non-normal returns.
David Colquhoun (2014) — Royal Society Open Science
The paper shows (with simple probability arguments and t-test simulations) that treating p = 0.05 as “a discovery” typically implies a false discovery rate around 30% or higher—often much higher in low-power studies—so p-values are routinely over-interpreted.
Kay H. Brodersen et al. (2015) — The Annals of Applied Statistics · 854 citations
The paper introduces a fully Bayesian state-space (structural time-series) approach to estimate the causal effect of an intervention by forecasting a counterfactual time series (synthetic control) and comparing it to observed outcomes, implemented in the CausalImpact R package.
Marcos Lopez de Prado (2015)
This paper argues that most published backtests in empirical finance are statistically unreliable because they ignore multiple testing and other sources of performance inflation, and it proposes practical corrections (Deflated Sharpe Ratio and Probability of Backtest Overfitting).
D. Sculley et al. (2015) — Curran Associates, Inc. · 1336 citations
This paper discusses the concept of technical debt in machine learning systems, highlighting the often-overlooked maintenance costs and ML-specific risk factors that can lead to long-term problems.
Online tools for demonstration of backtest overfitting
David H Bailey et al. (2015) — Available at SSRN 2597421
This paper introduces two free online simulators (BODT and TMST) that show how optimizing many strategy variants on the same backtest can manufacture high in-sample Sharpe ratios that collapse out-of-sample—and why metrics like the Deflated Sharpe Ratio are needed to correct for multiple testing.
John M. Mulvey and Han Liu (2016) — The Journal of Portfolio Management
The paper shows how a simple, data-driven two-regime (growth vs. contraction) model—detected via trend filtering on the S&P 500—materially increases estimated left-tail risk for endowments, and how contingent (drawdown-triggered) spending cuts can significantly reduce long-horizon downside risk.
Campbell R. Harvey (2017) — The Journal of Finance · 286 citations
This paper discusses the overuse and misuse of p-values in financial economics research, highlighting issues like p-hacking and publication bias, and suggests adopting a Bayesian perspective for more robust and transparent research practices.
Victor Chernozhukov et al. (2018) — The Econometrics Journal · 3298 citations
This paper shows how to get valid (root-N, asymptotically normal) inference for causal/structural parameters when nuisance components are learned with high-dimensional ML, by combining Neyman-orthogonal scores with cross-fitting.
Regina Hammerschmid and Harald Lohre (2018) — International Review of Economics & Finance
By applying Markov regime switching models to a PCA-derived macroeconomic factor, this paper demonstrates that regime-based signals provide complementary predictive power to fundamental and technical indicators, yielding a 3.77% annual utility gain for S&P 500 investors.
Marcos Lopez de Prado (2019)
This paper describes Numerai, a crowdsourced machine learning tournament that democratizes quantitative finance by providing anonymized data and a structured problem, incentivizing participants to build robust forecasting models while mitigating common research handicaps like data access, domain expertise, and selection bias.
Niklas Pfister et al. (2019) — Journal of the American Statistical Association
The paper extends Invariant Causal Prediction to sequential (time-ordered) data without knowing regime labels, enabling statistically valid discovery of causal predictors (including instantaneous effects in time series) by testing invariance of a linear-Gaussian conditional model over time.
Campbell R. Harvey and Yan Liu (2019) · 70 citations
A comprehensive meta-analysis of over 400 published investment factors, arguing that most are false positives resulting from p-hacking and multiple testing failures.
Clark Glymour et al. (2019) — Frontiers in Genetics
This review explains how modern causal discovery algorithms use graphical models, conditional independence, and functional causal assumptions (e.g., non-Gaussianity/nonlinearity) to infer causal structure from observational data when experiments are infeasible.
Alice's Adventures in Factorland: Three Blunders That Plague Factor Investing
Robert D. Arnott et al. (2019) — Journal of Portfolio Management
Factor investing returns are consistently overstated due to data mining and crowding, while risks are understated due to non-normal tail behavior and vanishing diversification during crises.
Jakob Runge et al. (2019) — Science Advances
The paper introduces PCMCI, a two-stage causal discovery method for high-dimensional, autocorrelated, nonlinear time series that improves true-link detection while controlling false positives compared with Granger/FullCI, PC, and Lasso.
Sören R. Künzel et al. (2019) — Proceedings of the National Academy of Sciences
The paper systematizes “meta-learners” (S-, T-, X-learners) that wrap standard ML regressors to estimate conditional average treatment effects (CATE), and introduces the X-learner which is especially effective with unbalanced treatment/control sample sizes and/or structurally simple treatment effects.
Jakob Runge et al. (2019) — Nature Communications · 904 citations
This Perspective surveys modern methods for causal discovery from time series (beyond correlation/forecasting) and maps them to recurring real-world problems—arguing for benchmarks and clearer assumptions to make causal claims credible in complex dynamical systems.
Guido W. Imbens (2020) — Journal of Economic Literature
Imbens compares Pearl-style DAG/do-calculus and Rubin-style potential outcomes, arguing that while DAGs are great for clarifying assumptions and complex identification puzzles, the potential-outcomes framework better matches the identification strategies, estimands (e.g., LATE), and design/inference concerns that dominate applied economics.
Alex Lipton and Marcos Lopez de Prado (2020) · 5 citations
This note argues that COVID-19 exposed structural weaknesses in many quant workflows—overreliance on long-horizon forecasting, backtest-driven “discoveries,” and belief in all-weather alphas—and proposes nowcasting, theory-first research, and regime-adaptive ensembles as remedies.
Ronen Israel et al. (2020)
This paper argues that machine learning in finance, particularly for return prediction, faces unique challenges like small data and low signal-to-noise ratios, suggesting an evolutionary rather than revolutionary impact, with greater potential in areas like risk management and transaction costs.
Andreas G. F. Hoepner et al. (2021) — The European Journal of Finance
Establishes a framework for evaluating financial ML models by distinguishing between statistical/economic significance and relevance, while arguing that 'White Box' explainability is superior to 'Black Box' neural networks for finance.
Campbell R. Harvey (2021) — SSRN Electronic Journal · 2 citations
A foundational overview distinguishing systematic from discretionary investing, outlining the historical evolution of algos, and warning against the dangers of overfitting in the ML era.
Andrew Y. Chen and Tom Zimmermann (2021) · 185 citations
This paper provides an open-source dataset and code demonstrating that nearly 100% of published cross-sectional stock return predictability results can be successfully reproduced, challenging claims of widespread non-replicability and p-hacking.
Isaac Gibbs and Emmanuel Candès (2021)
This paper introduces Adaptive Conformal Inference (ACI), a method for constructing prediction sets that maintain target coverage even when the underlying data distribution changes over time, which is crucial for real-world applications like finance where market behavior is non-stationary.
Stefano Giglio et al. (2021) — The Review of Financial Studies
This paper develops a robust framework for multiple hypothesis testing of alphas in linear asset pricing models, controlling for data snooping, omitted factors, and missing data, and demonstrates its superior performance in hedge fund evaluation.
Bernhard Schölkopf (2022)
Schölkopf argues that many key failures of modern machine learning—poor robustness under distribution shift, weak transfer, and adversarial fragility—are fundamentally problems of causality, and that structural causal models plus the “independent causal mechanisms” principle provide a route to more invariant, transferable learning.
Marcos Lopez de Prado (2022)
The paper argues that current factor investing relies on spurious associational correlations rather than causal mechanisms, and proposes adopting 'Causal Factor Investing' using causal graphs and do-calculus to distinguish true drivers of returns from statistical artifacts.
Charles K. Assaad et al. (2022) — Journal of Artificial Intelligence Research
A comprehensive survey of causal discovery methods for multivariate time series, plus an empirical comparison showing that performance is highly assumption- and dataset-dependent—no single method dominates.
Kevin Khang (2022) — The Journal of Portfolio Management · 1 citations
Risk-model volatility forecasts depend heavily on the estimation lookback window, and a practical way to become “regime-aware” is to monitor real-time dispersion across forecasts from different calibrations and switch between slow- and fast-moving models when misspecification risk is high.
Pratiksha Sharma et al. (2022) — The Journal of Portfolio Management
A sector rotation strategy using Hidden Markov Models to detect volatility regimes outperforms the S&P 500 by switching to T-bills during high-variance states.
Jakob Runge et al. (2023) — Nature Reviews Earth & Environment
This technical review explains how to do causal discovery and causal effect estimation with time series, emphasizing practical pitfalls like contemporaneous links, latent confounding, and non-stationarity, and illustrating solutions via two Earth-science case studies with concrete workflows and numbers.
Marcos López de Prado (2023) — The Journal of Portfolio Management
The paper argues that financial research relies too heavily on backtesting (associational evidence) and proposes a new hierarchy where strategies must be validated through causal graphs and natural experiments to be considered scientific.
Fabian Hinder et al. (2023) · 17 citations
This paper surveys concept drift detection methods in unsupervised data streams, focusing on monitoring and anomaly detection, providing a taxonomy, localization techniques, and standardized experiments.
Chang Gong et al. (2023)
A comprehensive survey that unifies causal discovery methods for both regularly-sampled multivariate time series and irregular event sequences, and lays out practical evaluation resources plus future directions (amortized/supervised causal discovery and causal representation learning).
Alik Sokolov et al. (2023)
The paper proposes an end-to-end, LLM-assisted pipeline that generates causal DAGs from feature definitions (not just correlations) and then empirically prunes/validates edges with do-calculus—demonstrated on equity factor investing and credit-spread drivers.
Satyam Kumar et al. (2023)
A survey of 37 papers (1992–2023) showing how causal inference (e.g., DAGs/do-calculus, Bayesian networks, Granger causality, counterfactuals) is being used across banking, finance, and insurance—and arguing that banking and insurance applications are still early-stage, leaving room for impactful research and practice.
David Blitz et al. (2023) — The Journal of Portfolio Management
A practitioner-focused review arguing that ML's primary value lies in detecting nonlinearities and interactions, provided rigorous protocols are used to manage low signal-to-noise ratios and transaction costs.
Andrew Ang (2023) — The Journal of Portfolio Management
A regime-based analysis of five major equity style factors showing that Value and Momentum have structurally deteriorated in the 21st century while Quality and Size have improved, decomposing returns into long-term trends and short-term cycles.
Yongjae Lee et al. (2023) — The Journal of Portfolio Management
A comprehensive survey categorizing machine learning models by their specific applications in asset management, ranging from tree-based models for tabular data to reinforcement learning for hedging.
Bryan T. Kelly and Dacheng Xiu (2023)
A comprehensive survey arguing that highly complex, over-parameterized machine learning models—when properly regularized—outperform parsimonious econometric models in return prediction, factor modeling, and portfolio choice.
Isaac Gibbs and Emmanuel Candès (2023) · 145 citations
This paper introduces a new method called Dynamically-Tuned Adaptive Conformal Inference (DtACI) for creating prediction sets that adapt to changing data distributions over time, improving upon existing methods by dynamically adjusting the step-size parameter based on historical performance.
Emre Kıcıman et al. (2023)
This paper evaluates whether GPT-3.5/4 can do useful causal reasoning (discovery, counterfactuals, actual causality) and finds they set new benchmark highs—while still failing unpredictably—suggesting they’re best used as domain-knowledge assistants alongside formal causal tools.
Andrei Paleyes et al. (2023) — ACM Computing Surveys · 546 citations
This survey paper identifies and categorizes the challenges faced when deploying machine learning models into production environments across various industries, highlighting issues at each stage of the ML deployment workflow.
Ruijie Tang (2024)
The paper tests whether time-series causal discovery (tsFCI, VarLiNGAM, TiMINo) can be turned into a daily long-short equity strategy, finding it can help returns but is often computationally infeasible at realistic universe sizes—except for VarLiNGAM.
Yingguang Zhang et al. (2024)
This paper shows that the very large alpha reported by Binsbergen et al. (2023) from trading against machine-learning-predicted analyst forecast errors is largely a look-ahead bias artifact; once corrected, the alpha disappears and simple linear models perform as well as (or better than) the ML approach.
Agathe Sadeghi et al. (2024)
The paper proposes CD-NOTS, a constraint-based causal discovery algorithm tailored to nonstationary time series that explicitly models lagged dependencies, and shows in simulations and finance case studies that it can recover more reliable causal graphs than common baselines when stationarity/linearity assumptions fail.
Ronald Hua and Clare Sun (2024)
The paper proposes a practical framework to quantify how much historical factor information should be trusted out-of-sample under market nonstationarity, showing that optimal “shrinkage toward 1/N” depends strongly on market efficiency.
Yangyang Yu et al. (2024) · 112 citations
FINCON is a GPT-4-based manager–analyst multi-agent trading system that uses CVaR-based real-time risk alerts plus “conceptual verbal reinforcement” across training episodes to improve sequential trading decisions for both single stocks and small portfolios.
Joseph Simonian (2024) — The Journal of Portfolio Management · 1 citations
A methodological framework proposing 'Constructive Empiricism' to integrate econometrics (for bias reduction and explanation) and machine learning (for variance reduction and prediction) within a single investment process.
Angelos Kanas (2025) — The Journal of Portfolio Management
The paper replaces “which factors are significant in regressions?” with “which factors are direct causes?” by estimating Bayesian-network/SEM causal graphs linking eight common factors (FF5 + MOM + BAB + QMJ) to 10 US industry portfolios, and shows that the causal factor set is industry-specific and time-varying, with additional causal spillovers from EU and Japan after 1990.
Miquel Noguer i Alonso (2025)
This paper introduces a comprehensive framework for Financial Information Theory, applying information-theoretic concepts to financial time series and demonstrating their empirical usefulness for regime detection, market efficiency testing, and portfolio construction using S&P 500 data from 2000-2025.
Alejandro Lopez-Lira et al. (2025)
LLMs can appear to “forecast” economics/markets pre-cutoff because they often memorize realized historical outcomes, making pre-cutoff forecast evaluation fundamentally non-identifiable and prone to lookahead bias.
Marcos Lopez de Prado and Vincent Zoonekynd (2025)
This paper argues that many failed factor strategies are not just p-hacked but causally misspecified ('Factor Mirages'), and demonstrates how standard regression techniques involving colliders can flip the sign of risk premia.
Francesco A. Fabozzi and Marcos López Prado (2025) — The Journal of Portfolio Management
A practitioner-oriented guide to deploying LLMs in asset management that emphasizes governance, training, workflow integration, and the operational risk of non-reproducible (time-fragile) outputs.
Anton Korinek (2025) · 5 citations
This paper explains what LLM-based AI agents are and gives economists a practical, code-backed playbook for using and building them to automate literature review, coding, and data analysis—while emphasizing key failure modes that require human oversight.
Marcos López de Prado and Frank J. Fabozzi (2025) — The Journal of Portfolio Management
A comprehensive survey detailing ten specific areas where machine learning outperforms traditional econometrics in finance, ranging from hierarchical portfolio construction to meta-labeling for bet sizing.
Alexander Rudin et al. (2025) — The Journal of Portfolio Management
A strategic framework for modern asset management that advocates combining economic theory (priors) with machine learning (adaptation) and human discretion to navigate market complexity.
Frank J. Fabozzi (2025) — The Journal of Portfolio Management
López de Prado argues that finance requires specialized 'Alpha Assembly Lines' and distinct ML algorithms (like HRP and DSR) to mine 'microscopic alpha' and avoid the overfitting traps of standard econometrics and backtesting.
Andrew Chin (2025) — The Journal of Portfolio Management
Large Language Models (LLMs) are converging investment styles by giving discretionary managers scalable 'breadth' and systematic managers qualitative 'depth', creating a hybrid 'Iron-Person' investor model.
Hoyoung Lee et al. (2025) · 9 citations
This paper proposes a way to measure and benchmark latent investment preferences (biases) in LLMs, showing that many models systematically favor certain equity styles (e.g., tech/large-cap/contrarian) and often display confirmation bias under “knowledge conflict” scenarios.
Rohan Alur et al. (2025) · 1 citations
A multi-agent LLM system achieves human superforecaster performance by combining agentic search, supervisor-based reconciliation, and statistical calibration.
Assumption violations in causal discovery and the robustness of score matching
Francesco Montagna and Atalanti A Mastakouri
A large-scale benchmark shows that score-matching-based causal discovery methods (SCORE/NoGAM/DAS) recover causal order more robustly than popular alternatives when key assumptions (iid, faithfulness, no confounding, no measurement error, correct functional form) are violated.
Ryan Sullivan et al. — The Journal of Finance
This paper applies White's Reality Check bootstrap methodology to rigorously evaluate thousands of technical trading rules, finding that while they show significant in-sample performance even after data-snooping adjustment, this performance largely disappears out-of-sample, particularly in liquid futures markets.
Causal Discovery from Heterogeneous/Nonstationary Data
Biwei Huang et al.
CD-NOD turns distribution shift (across domains or time) from a nuisance into a signal: it uses a domain/time index to recover causal graphs, orient edges via independently changing mechanisms, and visualize what drives nonstationarity—useful when markets/relationships drift.
Robert F. Engle et al. (1983) — Econometrica
This paper provides rigorous definitions for weak, strong, and super exogeneity, clarifying their distinct roles in econometric inference, forecasting, and policy analysis, especially in the context of parameter changes.
Guido W. Imbens and Joshua D. Angrist (1994) — Econometrica · 5082 citations
This paper introduces the concept of Local Average Treatment Effect (LATE) as an alternative to Average Treatment Effect (ATE) when evaluating social programs, especially in cases where full random assignment is not feasible, providing a method to identify treatment effects for subpopulations influenced by an instrumental variable.
{Board of Governors of the Federal Reserve System (2011) · 51 citations
The foundational regulatory framework establishing the industry standard for Model Risk Management (MRM), mandating independent validation, effective challenge, and governance for all financial models.
Xun Zheng et al. (2018) · 1250 citations
This paper introduces a novel method called NOTEARS for learning directed acyclic graphs (DAGs) by reformulating the combinatorial optimization problem into a continuous one, enabling the use of standard numerical algorithms.
Andrew Tiffin (2019)
The paper shows how “causal machine learning,” especially causal forests, can estimate the causal growth cost of a banking crisis and how that cost differs by country, while also providing interpretable drivers via Shapley-style explanations.
Ryan J. Tibshirani et al. (2020) · 617 citations
This paper extends conformal prediction to settings where training and test data distributions differ (covariate shift) by introducing a weighted approach that maintains valid prediction intervals, even when weights are estimated.
Kristian Bondo Hansen (2020) — Big Data & Society
This paper explores how quantitative analysts in algorithmic trading manage the complexity of machine learning models by applying Ockham's razor as a heuristic to ensure interpretability, control, and prevent overfitting.
Alexander Reisach et al. (2021) — Curran Associates, Inc. · 182 citations
This paper demonstrates that the performance of continuous causal structure learning algorithms on simulated data can be attributed to a property called 'varsortability,' where marginal variance increases along the causal order, and that this advantage disappears when data is standardized.
Atalanti A. Mastakouri et al. (2021) — PMLR
SyPI is a constraint-based causal feature selection method for time series with latent confounders that identifies which candidate series cause a target using only two conditional-independence tests per candidate, achieving <1% false positives in simulations.
Is There a Replication Crisis in Finance?
Theis Ingerslev Jensen et al. (2022) · 285 citations
This paper argues that the perceived replication crisis in financial economics is overstated, demonstrating through a novel hierarchical Bayesian model and global data that most asset pricing factors are replicable, cluster into 13 themes, and show external validity.
Joseph Simonian (2022) — The Journal of Portfolio Management
The paper proposes a way to make portfolio decisions when you can’t confidently identify the true economic causal model, by selecting the “most central” causal graph using a numeric distance between candidate causal schemes and then optimizing trades to minimize that distance.
Jean Kaddour et al. (2022)
This survey paper provides a comprehensive overview of Causal Machine Learning (CausalML), categorizing methods into five groups: causal supervised learning, causal generative modeling, causal explanations, causal fairness, and causal reinforcement learning, and discusses open problems and future research directions.
Klaus Grobys et al. (2022) — Applied Economics
This paper demonstrates that hedge funds leveraging Artificial Intelligence and Machine Learning (AIML) significantly outperform traditional human-managed funds, yielding a monthly alpha of 50-56 basis points for a long-short strategy against human-driven funds.
Aaron Brixton et al. (2022) — AQR Alternative Thinking · 8 citations
This paper demonstrates that the stock-bond correlation is driven by the relative volatility of inflation versus growth, warning that a return to high inflation uncertainty could flip the correlation positive and increase 60/40 portfolio risk by 20%.
Ali Shojaie and Emily B. Fox (2022) — Annual Review of Statistics and Its Application · 458 citations
This review explains what Granger causality really measures (predictive, time-ordered dependence), why naive/bivariate VAR tests can mislead, and how modern high-dimensional, nonlinear, discrete-valued, and mixed-frequency methods extend the framework.
Bernhard Schölkopf and Julius von Kügelgen (2022)
A high-level tutorial/position paper arguing that many core ML failures under distribution shift are best addressed by moving from purely statistical learning to causal (interventional) modeling, especially via structural causal models and independent causal mechanisms.
Rina Foygel Barber et al. (2023) — The Annals of Statistics · 385 citations
This paper extends conformal prediction to provide valid prediction intervals even when data is not exchangeable (e.g., due to distribution drift or non-symmetric algorithms), using weighted quantiles and a novel randomization technique.
Ashwin Alankar et al. (2023) — The Journal of Portfolio Management · 1 citations
Using 1871–2022 US monthly data, the paper identifies 23 equity drawdowns ≥15%, clusters them into 5 pre-crisis “environments,” and shows a simple 2-variable model can usually tell whether the next drawdown will be a fast “gamma” crash or a slow “delta” grind—guiding tail-hedge choice.
Jason Wei et al. (2023) · 16721 citations
The paper shows that giving large language models a few examples that include intermediate reasoning steps (“chain-of-thought”) can dramatically improve accuracy on multi-step reasoning tasks without finetuning.
Hanchen Wang et al. (2023) — Nature
A comprehensive review of how AI—specifically geometric deep learning, self-supervised learning, and generative models—is transforming the scientific process from hypothesis generation to autonomous experimentation.
Andrew Lampinen et al. (2023) — Advances in Neural Information Processing Systems
The paper shows that agents—and even pretrained language models—can learn generalizable “do experiments, infer causal structure, then exploit” strategies from purely passive data, as long as they can intervene at test time, especially when training includes natural-language explanations.
Kieran Wood et al. (2024) — The Journal of Financial Data Science
The X-Trend model uses cross-attention over a context set of historical market regimes to adapt trend-following strategies rapidly to new conditions (like COVID-19) and trade unseen assets.
Alejandro Rodriguez Dominguez (2024)
The paper derives a “conditional risk-neutral” PDE system for a portfolio by conditioning constituent returns on a small set of common causal drivers (modelled via Gaussian copulas), and proposes monitoring PDE deviations as a dynamic risk-management signal.
Clifford S. Asness (2024)
Asness argues that despite faster information flow, stock markets have become less informationally efficient over the last 30 years due to social media and gamified trading, resulting in wider and longer-lasting valuation spreads.
Andrea L. Eisfeldt and Gregor Schubert (2024)
This paper reviews the significant impact of Generative AI as a technology shock on firm values and corporate policies, and as a transformative methodological tool for financial research, offering practical guidance and future research directions.
Tianjiao Zhao et al. (2025) · 6 citations
This paper introduces AlphaAgents, a multi-agent LLM framework that uses specialized agents and a debate mechanism to enhance equity stock selection and mitigate human cognitive biases, outperforming single-agent and benchmark strategies in risk-neutral settings.
Parshin Shojaee et al. (2025)
The paper shows that “thinking” (chain-of-thought) reasoning models improve on medium-complexity puzzles but still hit sharp complexity thresholds where accuracy collapses—and their “thinking effort” paradoxically decreases right near failure.
Christian Mueller-Glissmann (2025) — The Journal of Portfolio Management
The traditional 60/40 portfolio is failing due to positive equity-bond correlations driven by inflation; practitioners must shift to dynamic asset allocation using machine learning and broader diversification into real assets and FX overlays.
Marcos López de Prado et al. (2025) — The Journal of Portfolio Management
The paper argues that you cannot compute a truly efficient portfolio frontier with a purely correlational (associational) factor model—efficient portfolio construction requires a causally specified factor model, otherwise optimization can systematically produce the wrong trades.
Alejandro Michel et al. (2025)
This paper introduces FinCARE, a hybrid framework that significantly improves causal discovery in finance by combining statistical algorithms with domain knowledge from SEC 10-K knowledge graphs and large language model reasoning, leading to more accurate causal graphs and reliable counterfactual predictions.
Agostino Capponi et al. (2025)
This paper introduces the nonstationarity-complexity tradeoff in financial return prediction, proposing an adaptive model selection method (ATOMS) that jointly optimizes model complexity and training window size to significantly improve out-of-sample R^2 and trading returns, especially during economic recessions.
Irene Aldridge et al. (2025)
This survey comprehensively reviews agentic AI in finance, highlighting its autonomous reasoning and adaptive decision-making capabilities, while also analyzing its profound impact on market structure, efficiency, and systemic risks, alongside the significant regulatory challenges it introduces.
Philip Treleaven et al. — Communications of the ACM
This paper provides a comprehensive review of Algorithmic Trading (AT) systems, defining key concepts, outlining system components, and illustrating a market-neutral momentum strategy with a backtested case study to encourage computer scientists in the field.
Stefan Wager and Susan Athey — Journal of the American Statistical Association
This paper introduces “causal forests,” a random-forest-based estimator for heterogeneous treatment effects that comes with provably valid (asymptotically centered, normal) confidence intervals under unconfoundedness.
Causal AI Scientist: Facilitating Causal Data Science with Large Language Models
Vishal Verma et al.
This paper introduces Causal AI Scientist (CAIS), an LLM-augmented tool that autonomously performs causal inference from natural language queries and datasets, improving method selection and estimation accuracy through a decision tree and self-correction.
Matthew J. Vowels et al. — ACM Computing Surveys
A broad survey of causal structure learning methods—especially modern continuous-optimization approaches like NOTEARS—and a cautionary discussion about when learned graphs should (and should not) be interpreted as causal.
Susan Athey et al. — Journal of Econometrics
The paper proposes using Wasserstein GANs to generate realistic, data-anchored Monte Carlo simulations—reducing researcher discretion in simulation design and enabling finite-sample evaluation of causal estimators on distributions that mimic a target dataset.
Pairwise Likelihood Ratios for Estimation of Non-Gaussian Structural Equation Models
Aapo Hyvarinen et al.
This paper introduces new measures based on likelihood ratios to determine causal directions between non-Gaussian random variables, applicable for estimating structural equation models, even with limited data or in cyclic/nonlinear scenarios.