Distinguish among market, fundamental, and alternative data, and explain how dataset definitions shape what each source means in research and trading applications
Compare the observability, conventions, and engineering constraints of major asset classes, and identify how market structure changes what can be measured and modeled
Apply a financial data quality framework to diagnose common failure modes, especially point-in-time violations, survivorship bias, corporate action errors, and identifier mismatches
Conduct vendor due diligence across data quality, legal and compliance, and technical and commercial dimensions
Choose storage and query architectures that fit research and production needs, including when to use partitioned files, embedded analytical databases, or server-based systems
2.1
A Modern taxonomy of financial data
Financial data is classified into market data, fundamental data, and alternative data, with the emphasis that every dataset embeds hidden definitions — timestamp conventions, adjustment policies, identifier schemes, revision rules — that must be understood before modeling. Market data is presented as a hierarchy from raw event streams to aggregated bars, fundamental data is characterized by release lags and institutional rules that vary by asset class, and alternative data carries a high validation burden around coverage stability, reproducibility, and usage rights. The reader learns to lock down four items before any research: timestamps, corporate action adjustments, identifiers, and revision policy. **Asset-class EDA notebooks** (Sections 2.1-2.2):
2.2
The asset-class market data landscape
Eight asset classes are surveyed (equities, ETPs, futures, options, digital assets, FX, fixed income, and commodities), documenting observability levels, key failure modes, and the engineering decisions required for each. The section provides concrete data on market sizes, explains how market structure determines what data is available and how informative it is, and identifies class-specific pitfalls: corporate action discontinuities for equities, roll rules for futures, surface construction for options, venue integrity screening for crypto, and close-convention ambiguity for FX. **Asset-class engineering notebooks**:
2.3
A due diligence framework for data sourcing
A systematic framework for data sourcing addresses four finance-specific failure modes: point-in-time violations, survivorship bias, corporate action errors, and identifier integrity breakdowns. The section details bitemporal storage and as-of query patterns for PIT correctness, quantifies survivorship bias using Monte Carlo simulation on 3,199 US equities (showing 63-109 percentage point distortions from missing delisted stocks), and provides vendor due diligence checklists spanning data quality, legal compliance, and technical reliability. The takeaway: data sourcing is risk management.
2.4
Storing data
File formats (Parquet, HDF5, CSV) and database engines (DuckDB, kdb+, ClickHouse, QuestDB, TimescaleDB, InfluxDB) are benchmarked for financial data workflows, measuring file size, write speed, read speed, and ASOF join performance. Parquet emerges as the recommended default for research (3.4x compression vs CSV with fast columnar reads), DuckDB provides SQL analytics over Parquet files without server overhead, and Polars delivers the fastest in-memory ASOF joins (3.8x faster than pandas). The reader gets a decision matrix mapping objectives (research velocity, production reliability, extreme throughput) to recommended storage stacks.
2.5
Summary
01 Us Equities Eda
02 Corporate Actions
03 Etfs Eda
04 Cme Futures Eda
05 Futures Session Aggregation
06 Futures Continuous
07 Sp500 Options Eda
08 Options Greeks Computation
09 Options Continuous
10 Crypto Perps Eda
11 Crypto Premium Analysis
12 Fx Pairs Eda
13 Data Quality Framework
14 Point In Time Validation
15 Survivorship Bias Detection
16 Provider Comparison
17 Complete Pipeline
18 Data Management
19 Incremental Updates
20 Storage Benchmark File
21 Storage Benchmark Database
22 Pandas Polars Benchmark
2 primer topics providing foundational concepts for this chapter.
Edwin J. Elton et al. (1996) — The Review of Financial Studies · 563 citations
By tracking every mutual fund from 1976 through 1993, this paper quantifies survivorship bias at approximately 90 basis points per year, revealing that ignoring failed funds significantly inflates historical performance estimates.
Tyler Shumway (1997) — The Journal of Finance · 1105 citations
This paper identifies a significant delisting bias in the CRSP database due to missing delisting returns for stocks delisted for negative reasons, leading to overstated portfolio returns.
Mark M. Carhart et al. (2002) — The Review of Financial Studies · 166 citations
Survivorship bias is not constant but increases with sample length, inflating annual returns by up to 1% in samples longer than 15 years and distorting factor analysis.
This paper examines the historical performance of commodity futures as an asset class, finding that they offer similar returns and Sharpe ratios to equities but with negative correlation to stocks and bonds, making them a valuable diversification tool and inflation hedge.
William Beaver et al. (2007) — Journal of Accounting and Economics · 233 citations
This paper examines how the inclusion or exclusion of delisting returns affects the performance of trading strategies based on accounting anomalies, finding that the impact varies depending on the specific anomaly.
Seven Sins of Quantitative Investing
Yin Luo et al. (2014)
A comprehensive empirical audit of seven common backtesting biases, demonstrating how errors in data handling (survivorship, look-ahead) and modeling (outliers, signal decay) can invert strategy performance from profitable to disastrous.
Validates that daily low-frequency data can accurately proxy FX transaction costs and demonstrates that liquidity evaporates globally when funding constraints (TED spread) and volatility (VIX) rise.
A seminal overview of how high-frequency trading has rendered traditional microstructure metrics (like trade direction and realized spreads) obsolete, requiring new approaches to data analysis and execution.
Kingsley Y L Fong et al. (2017) — Review of Finance
Using 19 years of global intraday trades/quotes as ground truth, the paper identifies which daily-data liquidity proxies best approximate spreads and price impact—finding Closing Percent Quoted Spread (CPQS) is best for percent-cost, and Amihud/impact-style proxies are best for Kyle’s lambda correlations but not its level.
Advances in Financial Machine Learning
Marcos Lopez de Prado (2018) — John Wiley & Sons · 106 citations
John Lehoczky and Mark Schervish (2018) — Annual Review of Statistics and Its Application
A historically organized survey of how equity-market data, statistical models, and trading strategies evolved from pre-CRSP fundamentals/technical analysis to statistical arbitrage, machine learning, and today’s electronic limit-order-book/HFT markets.
Igor Makarov and Antoinette Schoar (2020) — Journal of Financial Economics
This paper documents massive, persistent arbitrage spreads in cryptocurrency markets driven by capital controls and identifies that a single common order flow factor explains 80% of Bitcoin returns.
A comprehensive regulatory report detailing the dominance of algorithmic trading in U.S. markets, concluding that while it improves liquidity and efficiency in normal conditions, it introduces operational risks and potential instability during stress events.
Alex Lipton and Marcos Lopez de Prado (2020) · 5 citations
This note argues that COVID-19 exposed structural weaknesses in many quant workflows—overreliance on long-horizon forecasting, backtest-driven “discoveries,” and belief in all-weather alphas—and proposes nowcasting, theory-first research, and regime-adaptive ensembles as remedies.
Gene Ekster and Petter N. Kolm (2020) · 5 citations
A comprehensive guide to the alternative data ecosystem, detailing a specific preprocessing pipeline (entity tagging, stabilization, debiasing) that reduces revenue prediction error from 88% to 2.6%.
Tim Loughran and Bill McDonald (2020) — Annual Review of Financial Economics · 169 citations
A practitioner-oriented review of how finance uses text (social media, politics, fraud) that argues “readability” metrics like the Fog Index are mis-specified for 10-Ks and should be replaced by text-based measures of firm complexity.
David Easley et al. (2021) — The Review of Financial Studies · 68 citations
Machine learning demonstrates that traditional microstructure measures (specifically VPIN and Roll) retain significant out-of-sample predictive power for volatility and liquidity in modern high-frequency markets, especially when using dollar-volume bars and cross-asset features.
Florian Berg et al. (2022) — Review of Finance · 1489 citations
ESG ratings from major providers exhibit low correlation (average 0.54), driven primarily by disagreements on how to measure specific attributes (56% of divergence) rather than what attributes to include or how to weight them.
David Vidal-Tomás (2022) — International Review of Financial Analysis · 93 citations
This paper finds that major cryptocurrency coin-ranking sites like Coinmarketcap and Coingecko provide data with the same underlying statistical properties as direct exchange data for liquid cryptocurrencies, making them suitable for research despite concerns about data aggregation.
Lin William Cong et al. (2023) — Management Science
A systematic forensic analysis of cryptocurrency exchanges revealing that over 70% of reported volume on unregulated platforms is wash trading, detected via statistical anomalies in trade size distributions.
This paper derives the theoretical no-arbitrage price for cryptocurrency perpetual futures and demonstrates that a threshold-based trading strategy exploiting deviations from this price yields Sharpe ratios between 1.8 and 3.5.
A practitioner-focused guide to three backtest frameworks (walk-forward, resampling, Monte Carlo) and to avoiding common biases—especially selection bias from multiple testing that inflates Sharpe ratios.
T. Clifton Green and Shaojun Zhang (2024) — The Journal of Portfolio Management
A comprehensive survey of the alternative data landscape, categorizing sources into firm-released, government, attention, and third-party data, while analyzing the regulatory drivers and implementation challenges like alpha decay.
This paper provides the first empirical evidence of cross-market price discovery in modern prediction markets, finding significant arbitrage opportunities and demonstrating that Polymarket leads Kalshi in information aggregation, driven by liquidity and informed 'whale' trades.
B. Espen Eckbo and Markus Lithell (2025) — Journal of Financial and Quantitative Analysis · 3 citations
This paper argues that the decline in U.S. stock market listings after 1996 is overstated because it doesn't account for firms that remain under public ownership after being acquired by publicly listed companies, and that once merger activity is considered, the U.S. exhibits a listing advantage due to its efficient market for mergers.
Gueorgui S. Konstantinov (2025) — The Journal of Portfolio Management
A framework for transforming currency from a passive hedging byproduct into an active alpha source by integrating carry, value, trend, and volatility styles via risk budgeting.
Understanding the Market for U.S. Equity Market Data
Charles M Jones
An economic analysis arguing that U.S. equity market data fees are stable, competitively priced, and constitute a negligible fraction of total industry costs compared to commissions and third-party vendor fees.
This paper examines the historical returns of commodity futures, finding that while individual commodity futures have near-zero average returns, a rebalanced portfolio can achieve equity-like returns due to diversification, and that term structure and momentum may offer opportunities for above-average returns, though their dependability is uncertain.
This seminal paper demonstrates that increases in Google search volume for financially negative terms like 'debt' precede stock market falls, enabling a strategy that returned 326% over 7 years.
A comprehensive analysis of crypto asset characteristics demonstrating that while fundamental valuation models are flawed, active strategies like volatility targeting and trend following significantly improve risk-adjusted returns.
Dirk G. Baur and Lee A. Smales (2022) — Journal of Futures Markets
Leveraged money traders (hedge funds) in Bitcoin futures act as 'smart money' by successfully timing the market through adjustments in their short positions, offering a profitable signal for other investors.
Arash Aloosh and Jiasun Li (2024) — Management Science
Using leaked Mt. Gox data with user IDs, this paper provides ground-truth validation for statistical methods used to detect wash trading, finding that Benford's Law and size clustering are effective while Power Law tail analysis is unreliable.
Santiago Guzman et al. (2025) — The Journal of Portfolio Management
The proliferation of ETFs has significantly increased the 'macroefficiency' (price synchronicity with aggregate information) of developed equity markets, but this effect is largely absent in developing markets.
This paper analyzes historical and forward-looking expected returns for major asset classes (equities, bonds, credit, real estate, commodities, hedge funds, and private equity), advocating for diversification beyond equities and a balanced approach incorporating historical data, financial theories, and market indicators.
This paper identifies four well-known characteristics (carry, defensive, momentum, and value) that significantly explain cross-sectional corporate bond excess returns, demonstrating their economic significance even after transaction costs and distinctness from traditional market risks and equity anomalies.
These lecture notes survey the main empirical liquidity (and transaction cost) estimators—spreads, price-impact measures, non-trading-based measures, and estimators from autocovariances and high/low ranges—highlighting what each captures and the data requirements.
The paper analyzes the March 2020 Treasury market liquidity collapse, attributing it to regulatory constraints on dealer balance sheets, and proposes mandatory central clearing to enable netting and restore market elasticity.
Athanasios P. Fassas and Costas Siriopoulos (2021) — The Quarterly Review of Economics and Finance
This paper reviews and tests the information content of 47 publicly available implied volatility indices across various asset classes, finding that implied volatility contains information about future volatility beyond past volatility and that different asset classes react differently to underlying price changes.
Xiao-Yang Liu et al. (2022) — Advances in Neural Information Processing Systems
FinRL-Meta is an open-source, DataOps-style pipeline + library that turns multiple real-market data sources into gym-style trading environments and reproducible DRL benchmarks (with cloud-based visualization/competitions) to reduce irreproducibility and the sim-to-real gap in financial RL.
This paper derives no-arbitrage pricing formulas for various perpetual futures contracts (linear, inverse, quanto, and everlasting options) in discrete and continuous time, showing that their prices can be represented as the risk-neutral expectation of the spot price at a random future time, and identifies funding specifications that ensure perfect spot-price anchoring and dynamic replicability.
Unknown (2025) — Blockchain: Research and Applications
This paper analyzes the risk and return profiles of funding rate arbitrage strategies in cryptocurrency perpetual futures markets on both centralized and decentralized exchanges, finding that it offers diversification benefits and can generate substantial returns with limited losses compared to HODL strategies.
This paper reveals that the NYSE Trade and Quote (TAQ) data, widely used in finance research, suffers from a systematic look-ahead bias due to variable exchange-to-SIP latency and event clustering, leading to incorrect trade signing and understated spreads, and proposes a new methodology to correct these errors.
Systematic Fixed- Income Investing Comes of Age
Scott DiMaggio and Bernd Wuebben
A framework for applying systematic multifactor investing to fixed income markets, emphasizing the necessity of proprietary liquidity modeling and specific factor definitions (Value, Carry, Quality) distinct from equities.
Alya Al Nasseri et al. — Expert Systems with Applications
This paper applies Decision Trees to StockTwits data to identify specific semantic terms (and term combinations) whose volume fluctuations predict DJIA price movements, finding that 'sell' terms are more predictive than 'buy' terms.
