Learning Objectives
- Distinguish lexical features, static embeddings, sequential models, and Transformers in terms of the information each representation preserves and loses
- Explain how Transformer self-attention produces contextual embeddings and why this resolves key limitations of earlier NLP methods, including polysemy and long-range dependence
- Apply a practical financial NLP workflow that combines pre-trained checkpoints, domain adaptation when needed, and task fine-tuning for classification or extraction tasks
- Design text-derived features such as sentiment, narrative surprise, or structured event signals using point-in-time-safe timestamps, model cutoffs, and aggregation rules
- Evaluate text-derived signals using horizon-aware diagnostics, coverage-aware analysis, and event-time alignment rather than benchmark accuracy alone
- Use token-level attribution and related diagnostics to audit, debug, and stress-test NLP features before deployment
Lexical and statistical models
Static embeddings
Sequential models
Transformers
The modern feature extraction workflow
Summary
01 Word2Vec Training
02 Asset Embeddings
03 Sentiment Evolution
04 Bert Finetuning
05 Financial Ner Finetuning
06 Finbert Cross Dataset
07 News Return Signals
08 Text Feature Evaluation
09 Filing Text Signals
5 primer topics providing foundational concepts for this chapter.
Coverage-Aware Evaluation and Event-Time Alignment for Text Signals
A text model is not useful because it predicts labels accurately. It is useful only if its signal is available when you trade, on enough names, at the horizon that matters.
Domain Adaptation vs. Task Fine-Tuning in Financial NLP
In financial NLP, the failure mode is often not the architecture but starting from the wrong checkpoint for the wrong stage.
Self-Attention and Contextual Embeddings
Self-attention lets the same token mean different things in different sentences because its representation is rebuilt from context instead of looked up once from a static table.
Training Neural Networks
How forward passes, losses, backpropagation, optimization, and regularization work together so the architecture
When Long-Context Encoders Earn Their Keep
Use full-document encoding only when the label depends on interactions between distant parts of the document; otherwise chunking is usually the better default.
52 references cited in this chapter.
LSTM can solve hard long time lag problems
Sepp Hochreiter and Jürgen Schmidhuber (1996) — MIT Press · 1063 citations
This paper introduces Long Short-Term Memory (LSTM) networks and demonstrates their ability to solve complex sequence learning tasks with long time lags, outperforming traditional recurrent neural networks and other methods.
Paul C. Tetlock (2005) · 4065 citations
Stephen Robertson and Hugo Zaragoza (2009) — Found. Trends Inf. Retr. · 4941 citations
Tim Loughran and Bill Mcdonald (2011) — The Journal of Finance · 711 citations
Generic sentiment dictionaries (e.g., Harvard-IV negative words) badly misclassify tone in 10-Ks, so the paper builds finance-specific dictionaries and shows they better link 10-K language to market reactions and firm outcomes.
Tomas Mikolov et al. (2013) — arXiv preprint arXiv:1301.3781 · 33959 citations
The paper introduces CBOW and Skip-gram—two simple, fast neural architectures that learn high-quality word embeddings from billions of tokens and exhibit strong linear “analogy” structure (e.g., king − man + woman ≈ queen).
Jeffrey Pennington et al. (2014) — Association for Computational Linguistics · 34211 citations
Ashish Vaswani et al. (2017) — arXiv:1706.03762 [cs] · 171159 citations
This paper introduces the Transformer, a sequence-to-sequence model that replaces recurrence and convolutions with multi-head self-attention, achieving state-of-the-art translation quality with much faster, highly parallel training.
Natural Language Processing - Part I: Primer
Frank Zhao (2017)
This primer explains core NLP concepts and shows (with code) how to turn earnings-call transcripts into investable signals such as negativity-based sentiment changes and language-complexity measures linked to future returns.
Scott M Lundberg et al. (2017) — Curran Associates, Inc.
This paper proposes SHAP, a unified framework showing that many popular explanation methods are all approximations to a single, uniquely justified set of feature attributions (Shapley values) with desirable properties like local accuracy and consistency.
Dogu Araci (2019) · 895 citations
This paper introduces FinBERT, a BERT-based language model further pre-trained on financial text, and demonstrates its state-of-the-art performance on financial sentiment analysis tasks, outperforming existing methods and highlighting the benefits of transfer learning in finance.
Nils Reimers and Iryna Gurevych (2019) — arXiv:1908.10084 [cs] · 16856 citations
SBERT modifies BERT into a siamese/triplet architecture to produce cosine-comparable sentence embeddings that make semantic search and clustering practical (seconds instead of hours) while retaining strong accuracy on STS-style tasks.
Jacob Devlin et al. (2019) — Association for Computational Linguistics · 112230 citations
BERT introduces a Transformer encoder pre-trained with masked-token prediction (and next-sentence prediction) to learn deep bidirectional language representations that can be fine-tuned with minimal task-specific changes to achieve state-of-the-art NLP results.
Tim Loughran and Bill McDonald (2020) — Annual Review of Financial Economics · 169 citations
A practitioner-oriented review of how finance uses text (social media, politics, fraud) that argues “readability” metrics like the Fog Index are mis-specified for 10-Ks and should be replaced by text-based measures of firm complexity.
Allen Huang et al. (2020) — SSRN Electronic Journal · 13 citations
The paper builds FinBERT, a finance-domain BERT model, and shows it measures sentiment in financial text far more accurately than common dictionary and bag-of-words methods—materially improving event-study and earnings-prediction inferences.
Edward J. Hu et al. (2021)
Rajeev Bhargava et al. (2023) — The Journal of Portfolio Management · 4 citations
The paper builds daily, media-based “narrative” indicators (coverage intensity + sentiment) and shows they explain and sometimes predict market moves, and can be used to improve asset allocation and to build portfolios with explicit exposure to a narrative (e.g., COVID-19 recovery).
Shijie Wu et al. (2023) · 1242 citations
Andrew W. Lo et al. (2023) — The Journal of Portfolio Management
This article surveys how NLP evolved from rule-based systems to transformer LLMs and explains what that evolution means for practical financial workflows like trading signals, risk management, and ESG/impact investing.
Qianqian Xie et al. (2024) — Advances in Neural Information Processing Systems · 123 citations
FinBen is a large open-source benchmark (42 datasets, 24 tasks) designed to measure what today’s LLMs can and cannot do in real financial NLP, forecasting, risk, and trading/agent settings.
Benjamin Warner et al. (2024) · 506 citations
Leland Bybee et al. (2024) — The Journal of Finance · 84 citations
This paper introduces a novel approach to measuring the state of the economy by analyzing the full text of business news articles from the *Wall Street Journal* using topic modeling, demonstrating its ability to track economic activities and forecast stock market returns and macroeconomic dynamics.
Yaxuan Kong et al. (2024) — The Journal of Portfolio Management · 8 citations
This survey maps the emerging landscape of large language models (LLMs) in finance—what specialized models exist, where they can improve investment workflows, and which technical, evaluation, and ethical risks must be managed for responsible deployment.
Xavier Gabaix et al. (2025) · 4 citations
This paper introduces "asset embeddings" derived from institutional portfolio holdings using AI/ML techniques, demonstrating their superior ability to predict relative valuations, explain return comovement, and forecast portfolio decisions compared to traditional firm characteristics or generic text-based embeddings.
Guido Baltussen et al. (2025) — The Journal of Portfolio Management
This practitioner-oriented paper explains how asset managers can convert large-scale text (filings, earnings calls, news) into investable signals using NLP—from bag-of-words to LLM embeddings—while managing leakage, bias, hallucinations, translation, and cost.
Tomas Mikolov et al. (2013)
This paper introduces extensions to the Skip-gram model for learning word embeddings, including negative sampling and subsampling of frequent words, and demonstrates how to learn vector representations for phrases and combine word vectors through addition.
Paul C. Tetlock (2014) — Annual Review of Financial Economics
A comprehensive review of how media content, linguistic tone, and investor attention influence asset prices, establishing the foundation for Natural Language Processing (NLP) in finance.
Natural Language Processing - Part II: Stock Selection
Frank Zhao and Quantamental Research (2018)
Using earnings-call transcripts, simple NLP features (sentiment and “transparency” proxies like language simplicity and numeric intensity) predict cross-sectional U.S. stock returns, with long-short spreads of ~2–4% per year that remain significant after controlling for common factors.
Yinhan Liu et al. (2019) — arXiv:1907.11692 [cs]
RoBERTa shows that many post-BERT gains come from a better training recipe (more data, longer training, larger batches, no NSP, dynamic masking) rather than fundamentally new pretraining objectives, yielding state-of-the-art results on GLUE, SQuAD, and RACE.
Lingyun Zhao et al. (2020) — arXiv:2001.05326 [cs]
The paper fine-tunes RoBERTa to (1) classify sentiment in online financial text with emphasis on negative sentiment and (2) extract the key entity/entities driving that negative news using sentence matching and MRC-style QA instead of standard NER.
Victor Sanh et al. (2020) — arXiv:1910.01108 [cs]
The paper shows how to pre-train a smaller BERT-like model using knowledge distillation so it keeps almost all of BERT’s accuracy while being much smaller and faster—making on-device NLP practical.
Leland Bybee et al. (2020)
This paper introduces a novel approach to gauging the state of the economy by analyzing the full text of Wall Street Journal articles using topic modeling, demonstrating its accuracy in tracking economic activity and forecasting macroeconomic outcomes.
Shuohang Wang et al. (2021)
The paper evaluates GPT-3 as a cheap “pseudo-labeler” to annotate training data for smaller deployable NLP models, showing large labeling cost savings (50%–96%) and further gains when mixing GPT-3 and human labels under a fixed budget.
Jules H. van Binsbergen et al. (2022)
The paper shows that markets underreact to fraud-related cash-flow news in short-seller reports: prices drop about −4.9% on release day but continue drifting down to roughly −15% over the next year, especially when the report’s text emphasizes fraud.
Leland Bybee et al. (2023) — The Review of Financial Studies
This paper introduces a narrative factor pricing model using news text from *The Wall Street Journal*, demonstrating that these narrative factors achieve higher out-of-sample Sharpe ratios and smaller pricing errors compared to traditional characteristic-based factor models, while also predicting future investment opportunities consistent with the ICAPM.
Alejandro Lopez-Lira (2023) · 34 citations
The paper uses topic modeling on 10-K “Item 1A Risk Factors” text to infer firm-specific risk exposures, builds tradable risk-mimicking portfolios and a parsimonious factor model from them, and shows these “firm-identified risk factors” add pricing information beyond standard factor models.
Qianqian Xie et al. (2023) — arXiv preprint arXiv:2306.05443 · 259 citations
PIXIU open-sources an instruction-tuned finance LLM (FinMA), a 128,640-sample finance instruction dataset (FIT), and a benchmark (FLARE) spanning NLP and prediction tasks to systematically train and evaluate LLMs for finance.
Yaxuan Kong et al. (2024) — The Journal of Portfolio Management
Qianqian Xie et al. (2024)
Peter Andre et al. (2024)
This paper examines the narratives people use to explain macroeconomic phenomena, specifically the surge in US inflation in 2021-2022, and how these narratives shape inflation expectations and aggregate economic outcomes.
Ming Deng et al. (2025) — The Journal of Portfolio Management
Four vendor news-sentiment feeds are only weakly correlated, and averaging them into a single monthly firm-level signal produces a stronger, robust cross-sectional return premium that looks like slow-moving fundamental information rather than compensation for risk.
Alejandro Lopez-Lira and Yuehua Tang (2025)
This paper demonstrates that large language models (LLMs) like ChatGPT can predict stock market reactions to news headlines, achieving high portfolio-day hit rates and predicting subsequent price drift, especially for small stocks and negative news. The forecasting ability increases with model size, but strategy returns decline as LLM adoption rises, suggesting improved price efficiency.
Yixuan Tang and Yi Yang (2025) · 13 citations
This paper introduces FinMTEB, a new benchmark for evaluating text embedding models in the financial domain, and Fin-E5, a finance-adapted embedding model that achieves state-of-the-art performance on this benchmark, highlighting the importance of domain-specific adaptation.
Takaya Sekine et al. (2025) — The Journal of Portfolio Management
A practitioner tutorial showing how modern NLP (QA pipelines, news narrative construction, and ESG-claim classification) can turn unstructured text into scalable decision-support inputs for asset management.
Paul Glasserman et al. (2025)
Using 2.4M Reuters articles (1996–2022), the paper shows that persistent firm-level news-topic exposures and opposite intraday vs overnight price responses to the same topics explain a large share of the long-run U.S. “overnight drift” (overnight gains vs flat/negative intraday returns) and help forecast which stocks will do best overnight and worst intraday.
Ali Jaffri et al. (2025) — The Journal of Portfolio Management
This article is a step-by-step, practitioner-oriented tutorial showing how FinBERT can convert firm-specific financial news headlines into an interpretable sentiment index that can be overlaid on intraday price charts to contextualize short-term market moves.
Alya Al Nasseri et al. — Expert Systems with Applications
This paper applies Decision Trees to StockTwits data to identify specific semantic terms (and term combinations) whose volume fluctuations predict DJIA price movements, finding that 'sell' terms are more predictive than 'buy' terms.
Zhuang Liu et al. — International Joint Conferences on Artificial Intelligence Organization
This paper introduces FinBERT, a domain-specific language model pre-trained on a large financial corpus using multi-task learning, which outperforms state-of-the-art models in financial text mining tasks like sentiment analysis and question answering.
Alexis Conneau and Douwe Kiela (2018) — European Language Resources Association (ELRA)
SentEval is a standardized, easy-to-use toolkit that evaluates universal sentence embeddings across a community-agreed suite of transfer tasks, reducing pipeline variability and making results more comparable.
Dawei Cheng et al. (2020) — Association for Computing Machinery · 89 citations
This paper introduces a knowledge graph-based event embedding framework (KGEEF) to improve quantitative investment strategies by incorporating lead-lag relationships between entities extracted from financial news.
Chanyeol Choi et al. (2025) · 16 citations
FINDER is a 5,703-example expert-annotated dataset of real, ambiguous finance search queries grounded in S&P 500 10‑K evidence, designed to realistically benchmark retrieval-augmented generation (RAG) for financial question answering.
Orion Weller et al. (2025) · 68 citations
Single-vector (dense) retrieval has a hard, dimension-driven ceiling on the number of distinct top‑k document sets it can represent, and this limit shows up even for extremely simple “who likes X?” queries in the LIMIT benchmark.
Wim De Mulder et al. — Computer Speech & Language
This survey reviews how recurrent neural networks (RNNs) are used for statistical language modeling, why vanilla RNN LMs work well but train slowly and struggle with long context, and which architectural/training/decoding extensions (classes, LSTM, bidirectionality, etc.) appear most promising based on reported perplexity/WER/BLEU results.