Phase 18 - Lesson 18.8
Capstone 8 - Final Quant Research Project
The full arc: a research question turned into a reproducible, bias-controlled, cost-aware, risk-analyzed study with an honest executive summary.
Builds on: Integrates every phase: foundations (1–6), probability/stochastics (7–9), optimization/learning (10–13), finance (14–16), and engineering (17).
Leads to: This is the terminal deliverable of the academy - the artifact you would show to demonstrate quant competence.
Leads to: This is the terminal deliverable of the academy - the artifact you would show to demonstrate quant competence.
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Formulate a precise research question with a mathematical rationale and a source map.
- Assemble a reproducible pipeline: data description, assumptions, baseline, model, and validation.
- Control bias (survivorship, look-ahead, multiple testing) and incorporate realistic transaction costs.
- Conduct a risk analysis and state limitations without overclaiming.
- Write an executive summary that an informed reader can act on and a peer can reproduce.
Key Vocabulary
- Research question
- A precise, falsifiable statement of what you are testing and under what conditions.
- Source map
- An explicit mapping from each method used to the phase/book that justifies it.
- Baseline
- A simple benchmark the model must beat to be worth its complexity (e.g. buy-and-hold, zero-signal).
- Bias controls
- Procedures that remove survivorship, look-ahead, and multiple-testing bias from the study.
- Risk analysis
- Quantifying drawdown, tail risk, exposures, and scenario behavior of the result.
- Executive summary
- A concise, honest statement of question, method, result, uncertainty, and limitations.
Intuition & Motivation
Intuition
This capstone is not a new technique - it is the assembly of everything into one honest artifact. A quant research project is judged less by whether the answer is exciting than by whether it is trustworthy: could a skeptical peer reproduce it, are the biases controlled, are the costs real, is the risk understood, and does the summary claim exactly what the evidence supports - no more.
The single most valuable habit you can demonstrate is calibrated honesty. Most research questions, tested properly, yield a null or marginal result. Reporting that clearly - with reproducible code, out-of-sample net-of-cost numbers, and stated uncertainty - is worth more than a spectacular backtest that quietly leaked the future. Never promise profitability.
The brief: a complete research report
Choose a concrete, testable question (examples below) and produce a report plus reproducible code covering the fourteen required components. The report must let an informed reader judge the claim and a peer regenerate every number.
Required components (all must appear)
- Research question - precise and falsifiable.
- Mathematical rationale - why the question is well-posed and what theory motivates it.
- Source map - each method mapped to its phase/book (e.g. ‘walk-forward CV: Phase 12; ridge: Phase 11/ESL’).
- Data description - source, period, frequency, universe, and how it was snapshotted (point-in-time).
- Assumptions - stated explicitly (costs, execution, stationarity, no dividends, etc.).
- Reproducible code - seeded, version-controlled, one command regenerates results (17.3).
- Baseline - the simple benchmark the model must beat.
- Model - the method, its hyperparameters, and how they were chosen (on training folds only).
- Validation - out-of-sample, time-aware, with standard errors (Capstone 5).
- Bias controls - survivorship, look-ahead, and multiple-testing all addressed.
- Transaction costs - realistic, with a sensitivity curve (Capstones 5–6).
- Risk analysis - drawdown, tail behavior, exposures, scenario stress.
- Results - net, out-of-sample, with uncertainty; honest about significance.
- Limitations & executive summary - what the study does not show, then a concise action-oriented summary.
From question to rationale to source map
A good question is narrow and falsifiable: not ‘does momentum work?’ but ‘does a 12-1 month cross-sectional momentum signal, on the liquid US large-cap universe, 2005–2020, deliver positive net-of-cost out-of-sample returns after controlling for market exposure?’ The mathematical rationale states why you’d expect signal (e.g. autocorrelation structure, risk-based or behavioral hypotheses) and what would falsify it. The source map makes the intellectual lineage explicit and auditable.
| Component | Draws on | What it delivers |
|---|---|---|
| Estimation / least squares | Capstone 1, Phase 4–5 | stable, validated fits |
| Probability & simulation | Capstones 2–3, Phase 7–13 | null distributions, MC error bars |
| Validation & costs | Capstone 5, Phase 11–12 | OOS, leakage-free, net-of-cost |
| Execution & microstructure | Capstones 6–7, Phase 16 | realistic cost of trading the signal |
| Engineering | Phase 17 | reproducible, tested, modular code |
The honesty scaffold: baseline, bias, cost, risk
Four checks separate a research project from a story:
- Baseline: if the model doesn’t beat buy-and-hold (or a zero-signal null) net of cost, its complexity is unjustified.
- Bias controls: survivorship-free point-in-time universe (17.6), walk-forward validation (Capstone 5), and an accounting of how many variants were tried.
- Transaction costs: net \(=\) gross \(-\,c\cdot\) turnover (18.12), shown across a cost range (18.6).
- Risk analysis: maximum drawdown, tail (e.g. 5% CVaR), factor/market exposure, and scenario stress - a high Sharpe with a 60% drawdown is not ‘good’.
Worked Example - A reference report skeleton and grading rubric
1
§1 Question & rationale: one paragraph, falsifiable; state the null.
2
§2 Data & assumptions: source, period, universe (point-in-time), costs, what’s excluded.
3
§3 Method & source map: baseline, model, hyperparameter selection on training folds; each choice cited to a phase.
4
§4 Validation & bias controls: walk-forward design (18.11), leakage audit, multiple-testing note.
5
§5 Results: OOS net returns with SE, cost-sensitivity curve, comparison to baseline.
6
§6 Risk: drawdown, CVaR, exposures, scenario stress.
7
§7 Limitations & executive summary: what it does not show; then the 5-sentence summary.
8
Rubric (weightings): reproducibility 20%, bias control 20%, cost realism 15%, validation rigor 15%, risk analysis 15%, clarity/honesty 15%. A brilliant result with leakage scores near zero.
▶ Executive-summary template (fill honestly)
Question: <the falsifiable question>
Method: <baseline vs model, validation scheme, cost model>
Result: <OOS net metric> +/- <SE> vs baseline <metric>
Evidence: <is it distinguishable from the null? how many variants tried?>
Risk: <max drawdown, tail, key exposure>
Limits: <what this does NOT establish; regime/period caveats>
Claim: <exactly what the evidence supports -- no profit promise>
If the honest result is ‘no reliable edge net of cost out-of-sample’, that is a complete and valuable answer. Fabricating a positive claim is the one unforgivable error.
Common Mistakes to Avoid
- Picking a vague, unfalsifiable question so any result can be spun as success.
- Building the universe from today’s survivors (survivorship bias) - inflates every historical return.
- Leaking the future via full-sample scaling or test-set tuning (look-ahead bias).
- Reporting gross, in-sample, or single-variant-best numbers as if they were out-of-sample evidence.
- Omitting transaction costs or presenting a single optimistic cost point.
- Reporting return without risk (drawdown, tail) or without a standard error.
- Promising profitability - overclaiming beyond what leakage-free, net, out-of-sample evidence supports.
Quant Practitioner Tips
- Write the executive summary and limitations first, in pencil; they force honesty about what you can actually show.
- Make one command regenerate every figure from a fixed seed and a snapshotted dataset (17.3).
- Include a leakage audit as an explicit section - a reviewer will look for it.
- Always compare to a baseline and report OOS net-of-cost results with uncertainty.
- Report the number of strategies/variants tried; discount the headline accordingly.
- State the risk (drawdown, CVaR, exposures) alongside every return number.
- If the answer is null, say so clearly - a well-executed null is a strong deliverable.
Interactive: assemble and validate the report’s integrity checks
Rather than a single model, this check verifies the discipline: a leakage-free split, a net-of-cost result vs. baseline, and a completeness check that all fourteen required components are present.
Knowledge Check
Q1 Medium
Which deliverable would a skeptical reviewer weight most heavily when judging a quant research project?
Q2 Hard
If your properly-validated study finds no reliable edge net of costs out-of-sample, the correct action is:
Q3 Medium
A baseline (e.g. buy-and-hold) is required in the report because:
Practical Exercise
Draft the full skeleton of your final project for a specific question of your choice (e.g. ‘Does a low-volatility factor earn positive net-of-cost out-of-sample returns on US large caps, 2005–2020?’). Provide one to two sentences for each of the fourteen required components, and make the executive summary honest about uncertainty. You may assume a null or marginal result.
▶ Show full solution
A defensible skeleton (illustrative; a null result is fully acceptable):
- Question: Does a low-volatility factor deliver positive net-of-cost OOS returns on liquid US large caps, 2005–2020, after controlling for market exposure?
- Rationale: The low-volatility anomaly is documented; hypothesized from leverage constraints/behavioral biases. Null: no excess return net of costs.
- Source map: factor construction (Phase 11/ESL), walk-forward validation (Phase 12/Capstone 5), costs (Capstone 6), risk (Phase 14).
- Data: daily prices, 2005–2020, point-in-time large-cap universe reconstructed monthly, snapshotted with a content hash.
- Assumptions: 10 bps round-trip cost, monthly rebalance, no leverage, dividends reinvested, no shorting constraints beyond borrow.
- Code: seeded, version-controlled;
make reportregenerates all figures. - Baseline: cap-weighted buy-and-hold of the same universe.
- Model: rank stocks by trailing 1-yr volatility, long low-vol quintile, market-neutralized; no test-set tuning.
- Validation: walk-forward, expanding window; report OOS net returns with SE.
- Bias controls: survivorship-free universe, PIT features, leakage audit, note that 6 factor variants were examined.
- Transaction costs: net returns across 5–25 bps; sensitivity curve included.
- Risk: max drawdown, 5% CVaR, market beta, sector exposures, 2008/2020 stress.
- Results: e.g. OOS net excess return \(0.8\%\pm1.5\%\)/yr - not distinguishable from zero after costs.
- Limitations & executive summary: single market/period, cost model simplified, factor definition sensitive; Summary: ‘On this universe and period, the low-vol factor’s net-of-cost OOS excess return is statistically indistinguishable from zero; we do not find evidence of a tradeable edge after realistic costs, and make no profitability claim.’
Note how the summary states exactly what the evidence supports and no more - the hallmark of trustworthy quant research.
After the reveal, answer for yourself: Which two components, if omitted, would most damage a reviewer’s trust in your conclusion, and why?
Lesson Summary
The final capstone assembles the whole academy into one trustworthy artifact: a falsifiable research question with mathematical rationale and a source map, a point-in-time data description and explicit assumptions, reproducible code, a baseline, a model, time-aware validation, bias controls, realistic transaction costs, a risk analysis, honest results with uncertainty, and limitations plus an executive summary. Trust, not excitement, is the deliverable - and a well-executed null result is a complete answer. Never promise profitability.
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
▶ Show retrieval prompts & answers
Q: List at least six of the fourteen required components of the final research project.
A: Any six of: research question, mathematical rationale, source map, data description, assumptions, reproducible code, baseline, model, validation, bias controls, transaction costs, risk analysis, results, limitations/executive summary.
A: Any six of: research question, mathematical rationale, source map, data description, assumptions, reproducible code, baseline, model, validation, bias controls, transaction costs, risk analysis, results, limitations/executive summary.
Q: Why is a well-executed null result an acceptable, even valuable, deliverable?
A: Because most properly-tested research questions yield null or marginal results; honestly reporting a leakage-free, net-of-cost, out-of-sample null - with reproducible code and stated uncertainty - is trustworthy and informative, whereas a spectacular backtest that leaked the future is worthless. Calibrated honesty is the graded skill.
A: Because most properly-tested research questions yield null or marginal results; honestly reporting a leakage-free, net-of-cost, out-of-sample null - with reproducible code and stated uncertainty - is trustworthy and informative, whereas a spectacular backtest that leaked the future is worthless. Calibrated honesty is the graded skill.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check
Confidence / mastery rating
Personal notes