Phase 17 - Lesson 17.3

Reproducibility, Version Control, and Experiment Tracking

A result you cannot reproduce is not a result. Seeds, environments, version control, and logged experiments turn lucky runs into science.

⏱ 50 min● Intermediate🔗 Prereqs: 17.1, 17.2

↖ Phase 17 hub

Builds on: 17.2 made single runs correct; here you make whole experiments repeatable.
Leads to: The capstones in Phase 18 are graded partly on reproducibility - a fresh clone must regenerate every number.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Explain the components of a reproducible experiment: code, data, environment, and seed.
Implement a deterministic-seed check that two runs produce identical output.
Describe how version control (commits, branches, tags) records the exact state that produced a result.
Design an experiment-tracking record that maps a result back to its inputs and config.
Distinguish reproducibility (same inputs → same output) from robustness (result survives reasonable changes).

Key Vocabulary

Reproducibility: Given the same code, data, environment, and seed, you get bit-for-bit (or within-tolerance) the same result.

Seed: The initial state of a pseudo-random generator; fixing it makes ‘random’ draws deterministic and repeatable.

Version control: A system (e.g. git) that records every change as a commit, so any past state can be recovered exactly.

Environment pin: An exact record of package versions (e.g. a lockfile) so the same code runs the same everywhere.

Experiment tracking: Logging each run’s config, code version, data hash, and metrics so a result maps back to its inputs.

Global mutable state: Hidden state (a module-level RNG, a cached file) that makes runs depend on order - the enemy of reproducibility.

Intuition & Motivation

Intuition

You run a backtest, see a Sharpe of 1.4, close the laptop. Next week the same script prints 0.9. Which number is real? If you can’t answer, you don’t have a result - you have an anecdote. Reproducibility is the discipline that lets you (and a reviewer, and future-you) regenerate any number from its inputs, on demand.

Four things must be pinned: the code (a git commit), the data (a snapshot or hash), the environment (package versions), and the seed (RNG state). Miss any one and ‘random’ results wander. The good news: pinning them is cheap, and it is the difference between research and gambling.

The four pillars

Pillar	What varies if unpinned	How to pin it
Code	A refactor changes results silently	Commit; tag the exact SHA that produced the figure
Data	Vendor revises history; you re-download	Snapshot to disk; record a content hash
Environment	numpy/scipy version changes an algorithm	Lockfile of exact versions
Seed	Every run draws different randoms	`rng = np.random.default_rng(seed)` and pass it around

Seeds: making randomness repeatable

A pseudo-random generator is a deterministic function of its state. Fix the initial state (the seed) and the entire stream is reproducible. Modern NumPy prefers an explicit generator object over the legacy global functions:

\[\texttt{rng = np.random.default\_rng(seed)}\ \Rightarrow\ \texttt{rng.normal(size=n)}\ \text{is a pure function of (seed, n)}.\] (17.4)

Key Idea

Prefer a passed-in rng object to the global np.random.seed. Global state is order-dependent: any other code that draws a random number shifts your stream. A local generator is a clean, testable input.

Worked Example - A deterministic-seed reproducibility check

Wrap the experiment: def run(seed): rng = np.random.default_rng(seed); return estimate(rng).

Run twice with the SAME seed: a = run(42); b = run(42). They must be identical: assert a == b.

Run with a DIFFERENT seed: c = run(43). It should differ (else the seed isn’t actually driving the draws).

This two-line test catches hidden global state, un-seeded library calls, and accidental time/PID-based randomness.

Version control in one paragraph

Git records your project as a chain of commits, each a full snapshot with a unique hash. A result figure should carry the commit SHA that produced it; a reviewer can then git checkout that SHA and reproduce it. Branches isolate experiments; tags mark releases (‘the numbers in the report’). Commit small, commit often, and never commit large data or secrets - snapshot data separately and record its hash.

Experiment tracking: mapping results to inputs

For every run, log a record: the config (parameters), the code version (SHA), a data hash, the seed, and the metrics. Then any row in your results table points back to exactly the inputs that produced it. This is the audit trail that turns ‘I think I used \(\lambda=0.1\)’ into a fact.

Common Mistakes to Avoid

Using the global np.random.seed and assuming it’s reproducible - any other draw in the process shifts your stream.
Reporting a number without recording the code commit, so it can never be regenerated.
Re-downloading data each run; the vendor revises history and your ‘same’ backtest changes.
Pinning the seed but not the environment; a numpy upgrade changes an algorithm and your result drifts.
Seeding once at import time so results depend on execution order across notebooks/cells.
Confusing reproducibility with robustness: a result can be perfectly reproducible and still fragile to a tiny data change.

Quant Practitioner Tips

Make the top-level function take seed as an argument and pass the rng down; never draw from globals.
Commit before every experiment; put the SHA in the output filename or figure caption.
Snapshot data to a read-only file and store its SHA-256; treat raw data as immutable.
Log config + metrics to a small CSV/JSON per run; future-you will thank present-you.
Separate reproducibility (same in → same out) from robustness checks (vary seed/window/costs - does the conclusion survive?).

Interactive: prove your experiment is reproducible

Implement run(seed) so that same-seed runs are identical and different-seed runs differ. The tests are exactly the deterministic-seed check from the worked example.

Knowledge Check

Q1 Easy

Which set of things must be pinned for a Monte Carlo result to be reproducible?

Only the seed

Code, data, environment, and seed

Only the code

The seed and the operating system font

Q2 Medium

Why prefer a local rng = np.random.default_rng(seed) over the global np.random.seed?

It is faster

The global RNG is shared mutable state, so any other draw shifts your stream; a local generator is order-independent

The global one is deprecated and removed

Local generators use less memory

Q3 Medium

Reproducibility and robustness differ because:

They are the same thing

Reproducibility is same-inputs→same-output; robustness is whether the conclusion survives reasonable changes to inputs

Robustness means the code never crashes

Reproducibility requires the cloud

Practical Exercise

Describe, as a checklist, everything you would record so that a colleague with only your git repository could regenerate Figure 3 of your research note (a backtest equity curve) exactly. Then state the one additional experiment you would run to check the result is robust, not merely reproducible.

▶ Show full solution

Reproducibility checklist:

The git commit SHA that produced the figure (in the caption).
The exact data snapshot file and its SHA-256 hash (raw prices, read-only).
The environment lockfile (exact numpy/pandas/scipy versions).
The random seed and confirmation the code uses a local default_rng(seed).
The full config: universe, date range, parameters, transaction-cost assumptions.
A single command that runs end-to-end and writes the figure.

Robustness check: re-run the backtest across a grid of seeds and a few shifted train/test windows (and with higher transaction costs). If the equity curve’s qualitative conclusion collapses under these reasonable perturbations, the result is reproducible but not robust - and should not be trusted.

After the reveal, answer for yourself: Which item on the checklist is most often skipped in practice, and what failure does skipping it cause?

Lesson Summary

A result you cannot regenerate is an anecdote. Pin the four pillars - code (commit), data (snapshot/hash), environment (versions), and seed (local generator) - and log each run’s config and metrics so every number maps back to its inputs. Reproducibility (same in → same out) is necessary but not sufficient; also test robustness by varying seed, window, and costs.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: What four things must be pinned for a random experiment to be reproducible?
A: Code (git commit), data (snapshot/content hash), environment (exact package versions), and the RNG seed (via a local generator). Missing any one lets the result drift.

Q: Why is a local default_rng(seed) preferred over np.random.seed for reproducibility?
A: The global RNG is shared mutable state, so any unrelated draw in the process shifts your stream and breaks order-independence. A passed-in local generator makes the output a pure function of the seed.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

The Pragmatic Programmer (Thomas & Hunt, 2nd/20th Anniv. ed., 2019) current - Ch. 3, 7 - Version control, the ‘source of truth’, and avoiding global mutable state for repeatable builds.
Monte Carlo Methods in Financial Engineering (Paul Glasserman, 2004) foundational - Ch. 4 - Common random numbers and seeded streams for reproducible and variance-reduced simulation.

← 17.2 Next: 17.4 →